NFS dies under load

[freefan]

Also just for the record to rule out any other non-network issue, Cifs capability was installed on the Freenas box and tried mounting the Freenas server with a Cifs share instead of a NFS share - Still the same problem:

Sep 23 04:44:40 myshare01 kernel: INFO: task php-fpm:2084 blocked for more than 120 seconds.
Sep 23 04:44:40 myshare01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 23 04:44:40 myshare01 kernel: php-fpm D 0000000000000002 0 2084 2080 0x00000080
Sep 23 04:44:40 myshare01 kernel: ffff88020cb95ba8 0000000000000086 0000000000000000 ffffffff81fcf870
Sep 23 04:44:40 myshare01 kernel: 01ceb5ac78eb05bb 01ceaaa85fdefd00 01ceb12bd5e1242c 00000000000003e9
Sep 23 04:44:40 myshare01 kernel: ffff88020e0f7098 ffff88020cb95fd8 000000000000fb88 ffff88020e0f7098
Sep 23 04:44:40 myshare01 kernel: Call Trace:
Sep 23 04:44:40 myshare01 kernel: [<ffffffff812231f1>] ? avc_has_perm+0x71/0x90
Sep 23 04:44:40 myshare01 kernel: [<ffffffff8150ed3e>] __mutex_lock_slowpath+0x13e/0x180
Sep 23 04:44:40 myshare01 kernel: [<ffffffff81224f94>] ? inode_has_perm+0x54/0xa0
Sep 23 04:44:40 myshare01 kernel: [<ffffffff8150ebdb>] mutex_lock+0x2b/0x50
Sep 23 04:44:40 myshare01 kernel: [<ffffffff8119037b>] do_lookup+0x11b/0x230
Sep 23 04:44:40 myshare01 kernel: [<ffffffff8119069d>] __link_path_walk+0x20d/0x1030
Sep 23 04:44:40 myshare01 kernel: [<ffffffff8119174a>] path_walk+0x6a/0xe0
Sep 23 04:44:40 myshare01 kernel: [<ffffffff8119191b>] do_path_lookup+0x5b/0xa0
Sep 23 04:44:40 myshare01 kernel: [<ffffffff81182460>] ? get_empty_filp+0xa0/0x180
Sep 23 04:44:40 myshare01 kernel: [<ffffffff8119285b>] do_filp_open+0xfb/0xdd0
Sep 23 04:44:40 myshare01 kernel: [<ffffffff8109be2f>] ? hrtimer_try_to_cancel+0x3f/0xd0
Sep 23 04:44:40 myshare01 kernel: [<ffffffff8103c7b8>] ? pvclock_clocksource_read+0x58/0xd0
Sep 23 04:44:40 myshare01 kernel: [<ffffffff8119f562>] ? alloc_fd+0x92/0x160
Sep 23 04:44:40 myshare01 kernel: [<ffffffff8117de79>] do_sys_open+0x69/0x140
Sep 23 04:44:40 myshare01 kernel: [<ffffffff8117df90>] sys_open+0x20/0x30
Sep 23 04:44:40 myshare01 kernel: [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
Sep 23 04:45:42 myshare01 kernel: CIFS VFS: Server 10.95.1.40 has not responded in 300 seconds. Reconnecting...
Sep 23 04:45:42 myshare01 kernel: CIFS VFS: Send error in SETFSUnixInfo = -11
Sep 23 04:50:43 myshare01 kernel: CIFS VFS: Server 10.95.1.40 has not responded in 300 seconds. Reconnecting...
Sep 23 04:55:44 myshare01 kernel: CIFS VFS: Server 10.95.1.40 has not responded in 300 seconds. Reconnecting...
Sep 23 04:55:44 myshare01 kernel: CIFS VFS: Send error in SETFSUnixInfo = -11
Sep 23 05:00:45 myshare01 kernel: CIFS VFS: Server 10.95.1.40 has not responded in 300 seconds. Reconnecting...
Sep 23 05:00:45 myshare01 kernel: CIFS VFS: Send error in SETFSUnixInfo = -11
Sep 23 05:00:45 myshare01 kernel: CIFS VFS: Unexpected lookup error -5
Sep 23 05:05:47 myshare01 kernel: CIFS VFS: Server 10.95.1.40 has not responded in 300 seconds. Reconnecting...
Sep 23 05:05:47 myshare01 kernel: CIFS VFS: Send error in SETFSUnixInfo = -11
Sep 23 05:10:48 myshare01 kernel: CIFS VFS: Server 10.95.1.40 has not responded in 300 seconds. Reconnecting...
Sep 23 05:10:48 myshare01 kernel: CIFS VFS: Send error in SETFSUnixInfo = -11
Sep 23 05:10:48 myshare01 kernel: CIFS VFS: Unexpected lookup error -5
Sep 23 05:15:49 myshare01 kernel: CIFS VFS: Server 10.95.1.40 has not responded in 300 seconds. Reconnecting...
Sep 23 05:15:49 myshare01 kernel: CIFS VFS: Send error in SETFSUnixInfo = -11
Sep 23 05:20:50 myshare01 kernel: CIFS VFS: Server 10.95.1.40 has not responded in 300 seconds. Reconnecting...
Sep 23 05:20:50 myshare01 kernel: CIFS VFS: Send error in SETFSUnixInfo = -11
Sep 23 05:20:50 myshare01 kernel: CIFS VFS: Unexpected lookup error -5
Sep 23 05:25:51 myshare01 kernel: CIFS VFS: Server 10.95.1.40 has not responded in 300 seconds. Reconnecting...


Let me know if you have any success on the alpha if you install it (I might try it in the next couple days if time is willing) .



Also here is what mine network driver looks like:

[root@Storage00] /mnt/myvol00/mydir# sysctl dev | grep dev.em.0
dev.em.0.%desc: Intel(R) PRO/1000 Network Connection 7.3.7
dev.em.0.%driver: em
dev.em.0.%location: slot=0 function=0
dev.em.0.%pnpinfo: vendor=0x8086 device=0x1096 subvendor=0x8086 subdevice=0x3476 class=0x020000
dev.em.0.%parent: pci4
dev.em.0.nvm: -1
dev.em.0.debug: -1
dev.em.0.fc: 3
dev.em.0.rx_int_delay: 0
dev.em.0.tx_int_delay: 66
dev.em.0.rx_abs_int_delay: 66
dev.em.0.tx_abs_int_delay: 66
dev.em.0.itr: 488
dev.em.0.rx_processing_limit: 100
dev.em.0.eee_control: 1
dev.em.0.link_irq: 0
dev.em.0.mbuf_alloc_fail: 0
dev.em.0.cluster_alloc_fail: 0
dev.em.0.dropped: 0
dev.em.0.tx_dma_fail: 22118
dev.em.0.rx_overruns: 3
dev.em.0.watchdog_timeouts: 0
dev.em.0.device_control: 1075593793
dev.em.0.rx_control: 67141634
dev.em.0.fc_high_water: 30720
dev.em.0.fc_low_water: 29220
dev.em.0.queue0.txd_head: 213
dev.em.0.queue0.txd_tail: 215
dev.em.0.queue0.tx_irq: 0
dev.em.0.queue0.no_desc_avail: 0
dev.em.0.queue0.rxd_head: 955
dev.em.0.queue0.rxd_tail: 953
dev.em.0.queue0.rx_irq: 0
dev.em.0.mac_stats.excess_coll: 0
dev.em.0.mac_stats.single_coll: 0
dev.em.0.mac_stats.multiple_coll: 0
dev.em.0.mac_stats.late_coll: 0
dev.em.0.mac_stats.collision_count: 0
dev.em.0.mac_stats.symbol_errors: 0
dev.em.0.mac_stats.sequence_errors: 0
dev.em.0.mac_stats.defer_count: 0
dev.em.0.mac_stats.missed_packets: 361
dev.em.0.mac_stats.recv_no_buff: 128
dev.em.0.mac_stats.recv_undersize: 0
dev.em.0.mac_stats.recv_fragmented: 0
dev.em.0.mac_stats.recv_oversize: 0
dev.em.0.mac_stats.recv_jabber: 0
dev.em.0.mac_stats.recv_errs: 0
dev.em.0.mac_stats.crc_errs: 0
dev.em.0.mac_stats.alignment_errs: 0
dev.em.0.mac_stats.coll_ext_errs: 0
dev.em.0.mac_stats.xon_recvd: 0
dev.em.0.mac_stats.xon_txd: 0
dev.em.0.mac_stats.xoff_recvd: 0
dev.em.0.mac_stats.xoff_txd: 0
dev.em.0.mac_stats.total_pkts_recvd: 100075792
dev.em.0.mac_stats.good_pkts_recvd: 100075431
dev.em.0.mac_stats.bcast_pkts_recvd: 85022
dev.em.0.mac_stats.mcast_pkts_recvd: 553
dev.em.0.mac_stats.rx_frames_64: 131797
dev.em.0.mac_stats.rx_frames_65_127: 14411147
dev.em.0.mac_stats.rx_frames_128_255: 5080953
dev.em.0.mac_stats.rx_frames_256_511: 217218
dev.em.0.mac_stats.rx_frames_512_1023: 225632
dev.em.0.mac_stats.rx_frames_1024_1522: 80008684
dev.em.0.mac_stats.good_octets_recvd: 123487191509
dev.em.0.mac_stats.good_octets_txd: 60078821701
dev.em.0.mac_stats.total_pkts_txd: 128318863
dev.em.0.mac_stats.good_pkts_txd: 128318863
dev.em.0.mac_stats.bcast_pkts_txd: 3107
dev.em.0.mac_stats.mcast_pkts_txd: 14
dev.em.0.mac_stats.tx_frames_64: 37805
dev.em.0.mac_stats.tx_frames_65_127: 87150724
dev.em.0.mac_stats.tx_frames_128_255: 5437932
dev.em.0.mac_stats.tx_frames_256_511: 824081
dev.em.0.mac_stats.tx_frames_512_1023: 389396
dev.em.0.mac_stats.tx_frames_1024_1522: 34478925
dev.em.0.mac_stats.tso_txd: 10363597
dev.em.0.mac_stats.tso_ctx_fail: 0
dev.em.0.interrupts.asserts: 21175683
dev.em.0.interrupts.rx_pkt_timer: 4571
dev.em.0.interrupts.rx_abs_timer: 0
dev.em.0.interrupts.tx_pkt_timer: 269
dev.em.0.interrupts.tx_abs_timer: 519
dev.em.0.interrupts.tx_queue_empty: 0
dev.em.0.interrupts.tx_queue_min_thresh: 0
dev.em.0.interrupts.rx_desc_min_thresh: 0
dev.em.0.interrupts.rx_overrun: 0


I don't mind running slower than before as long as it runs :) is my driver out of date too? It looks a bit different from yours. Thanks!

 

[freefan]

Well I upgraded to the latest alpha released last night 9.2 in hopes of it going away, but it seems to have gotten worse :) This time the NFS network will die right in the middle of a file copy, or download, and my VM get's hung, and I can't kill libvirtd or the processes associated with it on the host box. Only way to bring everything back to normal is reboot the server. Seems much more unstable - but then again it is alpha.

I'm considering dropping back down to the 8* branch. after a data backup as it's an older version of ZFS there.

 

[aufalien]

Curious, before throwing in the towel, perhaps check this link and see what works for you. After all, tweaking is half the game, the other half is figuring out what it all means :)

https://calomel.org/freebsd_network_tuning.html

 

[freefan]

Hey thanks! I'm going to give those tuning settings a try.

 

[freefan]

I thought i was home free, as the network was running A LOT better after using those config files you posted, but after about 10 minutes of writing over the network (file by file), eventually I got:


Sep 24 20:47:37 myshare01 kernel: FS-Cache: Netfs 'nfs' registered for caching
Sep 24 20:47:37 myshare01 acpid: starting up
Sep 24 20:47:37 myshare01 acpid: 1 rule loaded
Sep 24 20:47:37 myshare01 acpid: waiting for events: event logging is off
Sep 24 20:47:37 myshare01 acpid: client connected from 1383[68:68]
Sep 24 20:47:37 myshare01 acpid: 1 client rule loaded
Sep 24 20:47:38 myshare01 automount[1403]: lookup_read_master: lookup(nisplus): couldn't locate nis+ table auto.master
Sep 24 20:47:42 myshare01 abrtd: Init complete, entering main loop
Sep 24 21:05:22 myshare01 kernel: INFO: task flush-0:19:2081 blocked for more than 120 seconds.
Sep 24 21:05:22 myshare01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 24 21:05:22 myshare01 kernel: flush-0:19 D 0000000000000000 0 2081 2 0x00000080
Sep 24 21:05:22 myshare01 kernel: ffff88020b875c00 0000000000000046 0000000000000000 0000000000000000
Sep 24 21:05:22 myshare01 kernel: ffff88020b875ce0 ffffffff81059784 ffff88020b875c80 ffff88020b875bf0
Sep 24 21:05:22 myshare01 kernel: ffff880210ce65f8 ffff88020b875fd8 000000000000fb88 ffff880210ce65f8
Sep 24 21:05:22 myshare01 kernel: Call Trace:
Sep 24 21:05:22 myshare01 kernel: [<ffffffff81059784>] ? find_busiest_group+0x244/0x9f0
Sep 24 21:05:22 myshare01 kernel: [<ffffffff8119c240>] ? inode_wait+0x0/0x20
Sep 24 21:05:22 myshare01 kernel: [<ffffffff8119c24e>] inode_wait+0xe/0x20
Sep 24 21:05:22 myshare01 kernel: [<ffffffff8150e82f>] __wait_on_bit+0x5f/0x90
Sep 24 21:05:22 myshare01 kernel: [<ffffffff811abb28>] inode_wait_for_writeback+0x98/0xc0
Sep 24 21:05:22 myshare01 kernel: [<ffffffff81096cc0>] ? wake_bit_function+0x0/0x50
Sep 24 21:05:22 myshare01 kernel: [<ffffffff811acf53>] wb_writeback+0x1f3/0x3f0
Sep 24 21:05:22 myshare01 kernel: [<ffffffff8150d6e0>] ? thread_return+0x4e/0x76e
Sep 24 21:05:22 myshare01 kernel: [<ffffffff81081ac2>] ? del_timer_sync+0x22/0x30
Sep 24 21:05:22 myshare01 kernel: [<ffffffff811ad2e9>] wb_do_writeback+0x199/0x240
Sep 24 21:05:22 myshare01 kernel: [<ffffffff811ad3f3>] bdi_writeback_task+0x63/0x1b0
Sep 24 21:05:22 myshare01 kernel: [<ffffffff81096b47>] ? bit_waitqueue+0x17/0xd0
Sep 24 21:05:22 myshare01 kernel: [<ffffffff8113ca40>] ? bdi_start_fn+0x0/0x100
Sep 24 21:05:22 myshare01 kernel: [<ffffffff8113cac6>] bdi_start_fn+0x86/0x100
Sep 24 21:05:22 myshare01 kernel: [<ffffffff8113ca40>] ? bdi_start_fn+0x0/0x100
Sep 24 21:05:22 myshare01 kernel: [<ffffffff81096916>] kthread+0x96/0xa0
Sep 24 21:05:22 myshare01 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
Sep 24 21:05:22 myshare01 kernel: [<ffffffff81096880>] ? kthread+0x0/0xa0
Sep 24 21:05:22 myshare01 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Sep 24 21:06:22 myshare01 kernel: nfs: server 10.95.1.40 not responding, still trying


So another "flush-0:19:2081 blocked for more than 120 seconds"

If you have any other ideas please let me know. I think the NFS server is still "going away" due to that 'not responding still trying" message.

I'm gonna not write any files over night and see if it survives with just reads.

 

[dvc9]

Shit, yea... this looks like a problem

You run a 1Gb Intel card, im on a 10GbE, it´s not to do with the drivers of the card then, but rather the NFS server.
I have gone all over the tweeks, and the only thing that "works" is by teaming up the NIC, and as soon as one nic gets a "hang" message, then it restarts the NIC
and we are back in buissenis, The user wont even know what happend.

But it´s not a good solution at all.

 

[dvc9]

Just for FYI

We are running a server with over 60 TB of storage, soon to be 100.
the "dd command" speed tests initiates a Write speed over 800 MB /s and 1 GB /s of reads

So it´s not the disks,
my Iperf testings gives 9.97 Gbit between two Linux machines, but to the FreeNAS then i have to do a -P4 run 4 Paralell threds to get that speed.

So there is something in between here the error is.

 

[cyberjock]

You're not recognizing that dd tests become very sequential and random seeks may be the cause for the problems. As the NFS and iSCSI using guys will tell you, performance can tank badly without adequate hardware(and rarely do people actually have the money and experience to get the necessary hardware right).

Just search the forum for NFS and ESXi and you'll probably find 50 threads in the last 2 months of people that have had their server performance tank to the point of being less than 1MB/sec and losing the connection from their FreeNAS machine and the ESXi box.

 

[dvc9]

I guess so, but a Random seek preformance is quite well, (The storages is shared up with 9 disks vdevs)
and the preformance i do get with all the machines now is around 250 - 300 MB/s pr machine.
Thats 4 ProTools suites, one Autodesk Flame stone, One Assimilate Scratch, a Davinchi Resolve suite, and 2 FinalCut suites.

So I will not complain at all of the server =P

But the Linux machines, the Flame.
keeps loosing NFS and NFS starts up again when under heavy load (Sequential DPX files )

This where not a problem in 8.3.

So my bad, i shuld not have updated.

 

[aufalien]

Interesting. Well I'm planning a serious load test using a diff card (SolarFlare duel 10Gb) and will report back. Could be a driver issue or a network stack issue perhaps?

My load test will comprise of many render nodes hitting the server so if any load were to kill it, this def would fo sho.

However I'm tending to think that this is a non issue as it would have come up in the devs system tests before going rel for sure.

 

[freefan]

Yeah mine is hanging up with no where near the amount of data dvc9 is throwing at his :) On that last hang coredump when i got "INFO: task flush-0:19:2081 blocked for more than 120 seconds." it died on a wget like this:

99% [===============================================================================================================================> ] 33,402,341 751K/s eta 0s

So A few things at this point on my end:

1. the freenas box never crashes - it just 'goes away' over the network when a random program trys interacting with the share. I never ever have to reboot freenas, but the client actually connecting and doing work on freenas over the network hangs up and I will have to reboot the client box.

2. 1 more cavet is I am connecting to freenas from a KVM vm client - not sure if that has anything to do with it - but I was connecting with a VM on 8.3 doing the same operations described in this thread just fine previously and no hang ups.

3. "keeps loosing NFS and NFS starts up again when under heavy load (Sequential DPX files )" - this right here - except its not really 'heavy' load on my end, but more like 'some load' on this end. Just trying to download with wget and save files - nothing really hammering the server too much.

Just some notes if it sparks any ideas . Maybe it it is a non-issue - but it's quite an anomaly. Also It might not be detected in 1 fail swoop (ie just write for 5 minutes as much as you can and if it works we're good). In the 9.1 Freenas I upgraded to I noticed the system might run for 6 hours just fine, only to come back and see it core dumped with that message. So sometimes it takes time to happen, other times it happens like 15 minutes into boot. Just my experience so far, and im not ruling out it's me :D Thanks for the insights so far.

 

[freefan]

1 last thing, I read recently that Freenas 9.* does not support NFS 4 yet fully - However when the client box that connects to freenas (cent OS 6.4) when it boots it says:

"Sep 24 21:16:58 mybox01 kernel: RPC: Registered tcp NFSv4.1 backchannel transport module."

So my system is registering the NFSc4.1 backchannel transport module - Does that mean the system should be able to connect to freenas still even though freenas is still using NFSv3 ? And if so would that pose a problem? Or is it backwards compatable. I just remember reading the other day that NFSv4 is not supported yet, and if the backchannel module is trying to use V4 - know what I mean? :) is this a possibility?

 

[dvc9]

Hey!
After some more testing, i did add my clients adresses to the FreeNAS´s hostfile

10.0.0.1 client1
10.0.0.2 client2

its here :
/Network/GlobalConfiguration/ Host name data base

It looks promising, and have not crashed in 24 houres.

 

[freenas-supero]

Posting here in the hope this will help solving this issue, I am NOT trying to hijack this thread (or at least this is not my intention)

Last reply to this thread dates from Sep 30 2013, almost a year ago, and I have been running into a very similar issue in the last month. To summarize, over the last few weeks, I have seen two occurences of what appears to be kernel oops on a Centos virtual machine running on a Proxmox VE virtualization server. The Centos guys have indicated that it could very well be related to backend storage, and knowing that my Centos VM uses NFS shares from a Freenas storage server, the first thing that came to my mind was to look on this forum.

The first time I experienced a similar kernel oops, I couldnt save the output but I clearly remember lots of references to nfs. This morning, I saved the dmesg output from the Centos VM:

INFO: task flush-0:19:22529 blocked for more than 120 seconds.
Not tainted 2.6.32-431.29.2.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
flush-0:19 D 0000000000000009 0 22529 2 0x00000080
ffff880037935c00 0000000000000046 0000000000000000 0000000000000000
ffff880037935ce0 ffffffff810598e4 ffff880037935c80 ffff880037935bf0
ffff880e05d5daf8 ffff880037935fd8 000000000000fbc8 ffff880e05d5daf8
Call Trace:
[<ffffffff810598e4>] ? find_busiest_group+0x244/0x9e0
[<ffffffff811a5860>] ? inode_wait+0x0/0x20
[<ffffffff811a586e>] inode_wait+0xe/0x20
[<ffffffff8152a0af>] __wait_on_bit+0x5f/0x90
[<ffffffff811b4c88>] inode_wait_for_writeback+0x98/0xc0
[<ffffffff8109b020>] ? wake_bit_function+0x0/0x50
[<ffffffff811b61d5>] wb_writeback+0x205/0x410
[<ffffffff81528e4e>] ? thread_return+0x4e/0x770
[<ffffffff81084aa2>] ? del_timer_sync+0x22/0x30
[<ffffffff811b6585>] wb_do_writeback+0x1a5/0x240
[<ffffffff811b6683>] bdi_writeback_task+0x63/0x1b0
[<ffffffff8109ae27>] ? bit_waitqueue+0x17/0xd0
[<ffffffff81143700>] ? bdi_start_fn+0x0/0x100
[<ffffffff81143786>] bdi_start_fn+0x86/0x100
[<ffffffff81143700>] ? bdi_start_fn+0x0/0x100
[<ffffffff8109abf6>] kthread+0x96/0xa0
[<ffffffff8100c20a>] child_rip+0xa/0x20
[<ffffffff8109ab60>] ? kthread+0x0/0xa0
[<ffffffff8100c200>] ? child_rip+0x0/0x20

Also yesterday I intensively used the freenas NFS shares, and noticed the tasks were hanging. For example copying large files to the Freenas server using scp, it took several seconds after I entered the password to start the operation. Usually right after the password is entered, the operation begins..

Directory listing (ls) also took several seconds to initiate.

Some background info:

Freenas server:
Old generation Supermicro X7DBE+
48GB RAM ECC DDR2-667 tested and certified
Mixture of 8x 2TB SATA2/3 drives all assembled as a big raidz3 array, 4 drives connected to a M1015, the other 4 connected to a different M1015
2X Xeon L5420 (4 cores each)
Using motherboard's NICs ( Intel ESB2/Gilgal 82563EB Dual-port)
Single gigabit connection between the virtualization server and the freenas server

Hostname freenas
Build FreeNAS-9.2.1.6-RELEASE-x64 (ddd1e39)
Platform Intel(R) Xeon(R) CPU L5420 @ 2.50GHz
Memory 49130MB
System Time Tue Oct 14 07:55:01 EDT 2014
Uptime 7:55AM up 16 days, 4:28, 0 users
Load Average 0.13, 0.18, 0.18

Virtualization server:
Supermicro H8DCL-iF
64GB RAM ECC DDR3-1800 tested & certified
2X Hitachi Ultrastar SAS2 300GB in RAID1 connected to a IBM M5016 with 1GB cache & BBU
2X Opteron 4334 (6 cores each)
Using motherboard's NICs ( Dual Intel 82574 Gigabit Ethernet Controller)
Runs Proxmox VE (latest release and updated)

Main switch:
HP Procurve 1800-24G (no modifications, factory settings)

I checked Proxmox's syslog and dmesg to make sure the issue was not on PVE's side. All logs clean. RAID controller is also configured to fire up an email if a problem arise. Nothing was issued and the array is in Optimal state.

There are no abnormal messages or errors in Freenas's dmesg, and NFS daemon is not reporting unusual stuff.

I can & will gladly provide any additional info to help troubleshoot this.

thanks!

 

[cyberjock]

Do you have a ZIL and L2ARC?

I won't lie, and we just told someone a few minutes ago that if you are wanting to go with virtualization, that means 64GB of RAM minimum, plus an L2ARC, plus a ZIL if you are using NFS (required if going ESXi, may not be necessary for others). 96GB of RAM is a major improvement and strongly recommended though.

If you don't have the hardware I mentioned above, I'd strongly recommend you try that first.

You're definitely in the minority running Proxmox though, so ymmv.

 

[HoneyBadger]

I wondered why this thread from a year ago had arisen from the dead.

Here's what jumped out immediately at me: "8x 2TB SATA2/3 drives all assembled as a big raidz3 array"

That's 5 data drives (an odd number) and a whole lot of parity that needs to be calculated (not that you don't have the CPU for it, but still) and neither of those makes for good random I/O, which VMs generate almost exclusively. I'm not sure what the default behavior for NFS under Proxmox is, but if you're doing sync writes performance will definitely suffer without SLOG.

 

[freenas-supero]

Hey cyberjock

Your recommendation for 64GB RAM for virtuialization, is it on the Freenas server? I am not using the freenas server for virtualization, only as a NFS share server for my VM's and other physical clients (my desktop computer, my htpc, etc)... No iSCSI is being used. Otherwise I would have used something snappier and more stable than gigabit to connect the iSCSI backend with the Proxmox host...

I am however curious about the L2ARC and ZIL. While I understand what they are for, I am pretty sure that I read somewhere on this forum that these were not essential for my usage, but I may have decided not to use them for costs reasons. I am being stupid here...
Let me clarify why I opted for no l2arc or zil...

This server supports only 64GB ram.. I could max it out but right now I have a ratio of 4.8GB RAM per TB of usable storage (or 3GB per TB or RAW hard drive capacity) which I believe is the reason why I did not opt to install a L2ARC at first..

Obviously with growing storage capacity (lets say I someday have 16x 4TB drives which would max out this server), then I will have to add another 16GB RAM, and my ratio will fall to 1GB per TB od raw hard drive capacity which at that point I will definitely install L2ARC.

If you think I should use L2ARC now, then I will document myself and proceed with it.

As for the ZIL, my basic understanding is that its useful for recovery (lets say the server is powered off and ZFS hasnt time to write to the pool). I should install this one no matter how much RAM I have....

 

[freenas-supero]

"Here's what jumped out immediately at me: "8x 2TB SATA2/3 drives all assembled as a big raidz3 array"

Another topic here that I would be VERY glad to start under a new thread but Im sure this has been covered 1E100 times already so... But I FULLY agree with you. At the time I built this Freenas server, I had 0 experience with ZFS, freebsd, freenas or such large qty of hard drives (not that 8 is a LOT but for me it was..) and I didnt know better than to assemble everything as a single large array with optimal protection. The only parameter I had at the time was data integrity and protection from data loss resulting from hardware failure...
this server has 16 physical hotswappable caddies. My plan was to use the first 8 drives (all 2TB) as I did, then gradually add 4TB drives in groups of 4 to expand the initial pool.

8x 2TB in RZ3
8x 2TB in RZ3 + 4x 4TB in ???
8x 2TB in RZ3 + 4x 4TB in ??? + 4x 4TB in ???
Then upgrade the 2TBs to 4TBs and later grow the initial pool...

In reality, I have no strategy because I have very little experience. for now the pool is 75% full and growing but I HIGHLY doubt I will ever need 30-40-50-60TBs.....

Id be very willing to restart fresh (nuke everything) provided I could backup 7-8TB of data... I will try to get 2X 4TB drives perhaps..

 

[cyberjock]

Everything I said was for using FreeNAS to store VMs, not to be virtualized itself.

You're taking blanket statements. You can't do that. You need to understand the technology and how to apply those technologies when they are appropriate, and to the extent appropriate. You're trying to store VMs, which is the worst thing imaginable for ZFS. So you're jumping into the fire, and the only way out is to read the living hell out of the forum and documentation until you understand it all enough to apply it. VMs are random read and random write workloads. You can't ask for a crappier set of conditions for ZFS.

 

[freenas-supero]

cyberjock,
I may have not been clear enough, but I am NOT using freenas to store virtual machines.

I am using Freenas to serve NFS shares TO my vm's.

I am not using it as a VM storage backend (like iScSI) or anything else. As a matter of fact, I use my freenas server for the VM's the same way I use it to mount NFS shares on my desktop machine or on my HTPC in my living room to play music, movies, etc...

Sorry if I wasnt clear enough!