FreeNAS Crashing on heavy data transfers

[rruss]

So, the basic details of my system are in my signature.

I've have a chronic problem with FreeNAS crashing on heavy file transfers over the network, and I'm not sure where to look to debug what could be going on. I've suspected various things over time, and after reconfiguring, rebooting, etc - things seem stable, and then out of the blue, FreeNAS completely locks up during file transfers over the network and requires a reboot to recover.

FreeNAS provides many NFS shares to my physical linux hosts, as well as CIFS shares to windows, and iSCSI for an ESXi host as the vm storage.

My initial build had the 2 Intel I210 NICs teamed together to my Cisco SG200-26 switch. Things worked fine, but data transfers weren't as fast as we expected, and we enabled jumbo frames on the switch and set MTU=9000 on FreeNAS. An ESXi 6 host was added to the network, also with dual NICs teamed together to the switch.

On a big data write to an NFS mount on FreeNAS, everything locked up on FreeNAS. Things had been stable for several months, but usage was very light. This was the first really heavy use. I rebooted, and it was good for a month, and then did it again. This time, I suspected the LACP NIC teaming and went down to 1 NIC, things seemed stable, but a week later, it crashed FreeNAS again. Now I suspected the MTU=9000, set it back to default, again, things seem stable (oddly, the ifconfig below shows "JUMBO_MTU" in options, but I'm not sure where it might be getting that.

Most recently, we reconfigured things to use the second NIC in both FreeNAS and the ESXi host to have a dedicated iSCSI bridge to get the vm disk traffic off the primary NIC to avoid interfering with NFS traffic to the linux hosts. This all seemed to be working well... but the iSCSI writes were slow, so yesterday I was experimenting and built another zvol for a second iSCSI target, and while using a VM that used this new iSCSI target to test a data transfer, FreeNAS locked up again.

Complete reboots are always required to get it back. I haven't lost any data in the process, yet, but I have to get to the bottom of this.

I believe my hardware is a well supported configuration. The I210 NICs should be stable.

I'm not terribly familiar with FreeBSD, and I'm not sure where to look on FreeNAS after a reboot to figure out what it was struggling with before crashing. If you have any ideas after reading this and can help steer me to figure out what could be going wrong, I'd really appreciate it!

Here is an ifconfig on freenas:

igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>
ether d0:50:99:c0:11:30
inet 192.168.100.99 netmask 0xffffff00 broadcast 192.168.100.255
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
igb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>
ether d0:50:99:c0:11:31
inet 10.10.10.1 netmask 0xffffff00 broadcast 10.10.10.255
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active

 

[Mirfster]

Posting your specs here; since those on mobile devices don't see signature info:

FreeNAS-9.3-STABLE-201602031011
ASRock C2750D4I | Intel(R) Atom(TM) CPU C2750 | 8 cores @ 2.40GHz
Dual Intel I210 NICs - on motherboard
Crucial 16GB (2 x 8GB) DDR3 SDRAM ECC | CT2KIT102472BD160B
4 x 4TB HDD | WD red 5400 RPM
2 x 256GB SSD | SAMSUNG 850 PRO
Rack mounted | Redundant Power Supplies | UPS w/ NUT control​

Most recently, we reconfigured things to use the second NIC in both FreeNAS and the ESXi host to have a dedicated iSCSI bridge to get the vm disk traffic off the primary NIC to avoid interfering with NFS traffic to the linux hosts. This all seemed to be working well... but the iSCSI writes were slow, so yesterday I was experimenting and built another zvol for a second iSCSI target, and while using a VM that used this new iSCSI target to test a data transfer, FreeNAS locked up again.

Going to need more RAM for iSCSI: Why iSCSI often requires more resources for the same result

 

[rruss]

Thanks for that Mirfster - didn't know my signature wasn't always visible.

Thanks for the thoughts on RAM and iSCSI as well. I could easily bump to 32GB or even 48G on my configuration. I just need to understand what is causing the crashes first - I don't think it has anything to do with non-optimal ARC caching for iSCSI. I had these crashes before the iSCSI target was even added.

I'm pretty certain I can cause a crash at will, and can bring the system to a state to tolerate the crash... I just need to know what to watch to see gain insight while it happens. Is there a place I can see a more real-time view of the data that is summarized on the reporting graphs? The system is crashing before I can see the related activity on the charts given the update rate on them in the webgui.

 

[rruss]

Well, I take back what I said about RAM...

I can't explain how a system with 16GB of RAM is swapping, but I watched the free memory dropping and dropping as I was doing transfers (which I expect as the cache fills). My assumption was that the cache would be sacrificed to give RAM to processes that needed it... but the processes associated with the iSCSI writes seemed to get starved for memory. The free RAM hit zero, and the system swapped... and then all was lost, and the system eventually dropped the network connection and had to be rebooted via IPMI.

Here is what the memory/swap looked like when I repeated the process again to confirm it:

I did some searching around the forum to see about ways to limit the cache size, but the consensus seems to be that it should do this automatically. vfs.zfs.arc_max is set reporting nearly 16GB. I've never set anything to manually manipulate this... what should it read on a system with 16GB of RAM?

 

[Mirfster]

I could easily bump to 32GB or even 48G on my configuration.

Give it more RAM and see how things go, but for iSCSI I would say put as much as much as you can in right now.

vfs.zfs.arc_max is set reporting nearly 16GB

Since you mentioned "vfs.zfs.arc_max"; would I be correct in assuming that you enabled autotune?

 

[rruss]

I did not have autotune enabled, and there were no tunables set in my system. When I queried vfs.zfs.arc_max with sysctl -a and it showed 15.5GB.

I checked autotune and rebooted, and now it shows many tunables and vfs.zfs.arc_max is 13.6GB. What is the recommendation with autotune? Should it be used, or no?

I'm going to order more memory tomorrow - adding 16GB is about $100, but adding 32GB jumps to $320. If it was just the price, I'd buy a pair of the 16GB modules - but it's only available from ebay sources, so I'd prefer to get 2x8GB from a trusted retailer.

 

[Mirfster]

I checked autotune and rebooted, and now it shows many tunables and vfs.zfs.arc_max is 13.6GB. What is the recommendation with autotune? Should it be used, or no?

Opps, I was not asking you to enable autotune... I was asking if you did enable it. If so, I was going to recommend that you disable it; delete the configurations that it set and reboot.

I'm going to order more memory tomorrow - adding 16GB is about $100, but adding 32GB jumps to $320. If it was just the price, I'd buy a pair of the 16GB modules - but it's only available from ebay sources, so I'd prefer to get 2x8GB from a trusted retailer.

Not saying that this will be the final cure. I was assuming that you already had the RAM and could put it in as a test...

Keep this quote in mind (from @jgreco 's thread):

What people want -> old repurposed 486 with 32MB RAM and a dozen cheap SATA disks in RAIDZ2

What people need -> E5-1637v3 with 128GB RAM and a dozen decent SATA disks, mirrored

 

[rruss]

Originally, I did not have autotune set, and my tunables list was blank. With this configuration, the memory management seems... broken. If I do a couple of nfs transfers to/from linux hosts on the igb0 interface (which fills the cache), and then do an iSCSI transfer, on igb1, which requires some CPU on the freenas box, the "free" memory drops very rapidly to 0 and then the system swaps. I can stop the activity immediately, and even with memory free the swap stays in use and it never seems to really recover. If I let it continue to struggle while swapping it eventually loses it's NICs, drops all networking, and essentially dies, requiring a power cycle.

From what I'm reading, my assumption is that none of this work to set vfs.zfs.arc_max should be required. FreeNAS should just drop cache as necessary to give active processes the memory they need to operate - at the expense of the performance of the read/write operations happening on NFS.

With all of that in mind, I'm aware of jgreco's recommendations... and we're not talking about re-purposing some ancient hardware here. This board was a highly recommended platform for FreeNAS from very experienced members of this forum (judging by post count).

Is it possible that the cache is filled with write operations that haven't been committed to disk, so freenas can't drop them quickly enough to handle the other operations that are going on?

I'm using ECC RAM, Intel NICs, stripped mirrors, keeping the system up to date, etc. I'm following the guidelines as best I can discern them. So yes - more RAM might help mask the problem... but it seems like a genuine problem that a system would swap before simply dropping cached zfs data. I've never seen good things happen after a system starts swapping... especially in a network where another box depends on freenas for all disk activity (iSCSI).

I verified the write speeds of my arrays with dd... and all are very fast, so I don't expect that I've got a lot of cached writes in memory that haven't been committed to disk and therefore can't be dropped from the cache to make room for the urgent needs of system processes.

Thanks again for your thoughts Mirfster - it's very helpful to bounce this stuff off the forum to help think about it and correct the things I may not be thinking about quite right.

 

[rs225]

I would say keep looking at limiting the ARC or things like that.

I am completely on a limb saying this (and therefore could be wrong), but I believe this is basically a bug, and it is a bug that has been identified for a long time by somebody on the FreeBSD side, but for whatever reason, no permanent fix has been merged into FreeBSD. I don't have the PR handy. The
'bug' I am referring to is the memory behavior (being pushed into swap unnecessarily), not the crash per se.

I haven't run into it myself, but enough people have that I would say it isn't just you.

 

[rruss]

I'll know more on Monday night after installing the additional 16GB.

I have a specific sequence of events that can crash it every time... is there a set of data I can collect for developers that would make this easier to track down and fix?

 

[Mirfster]

[System] - [Advanced] - "Save Debug". That will provide tons of info.

 

[rruss]

I saw that - If I run it right after a crash and reboot, will it have good data about the system behavior leading up to the point where it started swapping?

If so, I'll give it one more good crash and reboot and save debug before the memory upgrade.

 

[Mirfster]

Yep, it should give all that info.

 

[Robert Trevellyan]

Someone who knows better may have to correct me here, but ...

Block storage is a different beast from network sharing. The host expects hard-drive-class performance, and the time it takes ZFS to give back RAM it's using for ARC introduces enough of a delay for bad things to happen. It appears to me that you need to add more RAM and to set a limit on ARC size.

 

[rruss]

I installed the additional 16GB of RAM last night. Just to be sure I was starting clean, I ran a memory test (memtest86+ that comes with g4l), and let that run overnight. The RAM passed all tests.

I booted with autotune on, deleted all the tunables except vfs.zfs.arc_max which I set to 16GB, turned autotune off, and rebooted. At this point, I should have a system that will use 1/2 fo the physical RAM for ARC, and leave the rest untouched (if I understand vfs.zfs.arc_max correctly).

Once the freenas box booted, I booted the ESXi host and started just one VM which has its storage on the iSCSI hosted by freenas. This iSCSI is dedicated to this one VM, so the drive fills the 100G of the zvol. Sync settings are the default.

With nothing else happening on the freenas, ESXi, or any other machine on the network - I did a copy (scp) on the linux VM of a 32GB file on the physical HDD on another linux host to the local disk (ie, the iSCSI) on the ESXi VM, and watched the processes and memory on freenas with "top -s1". The ram went from 29G free... to ZERO and started swapping over the course of maybe 30 or 40 seconds!!

I'm at a complete loss. I did a "save debug".

My thoughts at this point - perhaps the test iSCSI I built on a zvol with block size intentionally set to match iSCSI at 512 bytes, using 100% of the zvol for the iSCSI is particularly unstable?

I only wanted one RAID'd file storage system to maintain on my network. This behavior makes me think that I can't have iSCSI on freenas and will need to install a hardware raid card and a couple of HDDs in the ESXi host to get the storage off freenas.

 

[Mirfster]

Here is an observation, you have 4 drives... How are they configured (mirrored, RaidZ1, RaidZ2)?

I ask because for certain things you would want different configurations. For NFS and CIFS; I would think that RaidZ2 is the preferred method. For iSCSI then mirrored vdevs; iSCSI and RaidZ2 is not a good thing...

 

[rruss]

They are a stripe of mirrors...

 

[Mirfster]

I booted with autotune on, deleted all the tunables except vfs.zfs.arc_max which I set to 16GB, turned autotune off, and rebooted.

I think that there is more to it than just setting "vfs.zfs.arc_max". While I am not thoroughly versed in all of this, you may want to try removing this setting as well and trying another test.

Maybe you will get lucky and one of the resident iSCSI gurus @jgreco will chime in. Of course, he may just tell you some bad news as well... ;)

 

[flanello]

Have you dedup on?
Can you check which process needs all the memory? Perhaps this could give clue what’s going on.

 

[rruss]

dedup is off.

And whatever process is eating the memory is invisible to "ps aux". "top" shows the memory disappearing... and even at the point that it starts to swap, "ps aux | sort -nk +4" continues to show manage.py using 0.8% of memory, and it just stays there as the top memory consumer at 0.8% as free memory slides to 0. So whatever is using it, appears to be invisible to the ps command.

I crawled through all the things generated by the "save debug", and it's all just stuff describing my configuration and log files, and nothing indicates what might be going wrong. Any ideas on what to look at in that package for a memory overflow issue like this?