Hi All,
first of all, I would like to thank iXsystems guys and FreeBSD team for amazing piece of software FreeNAS is. I've been using it for more than 5 years and always been happy with stability and feature set FreeNAS offers.
Recently, decided to restore my server lab used for development and devops muscle training and rebuilt the primary storage server that exposes NFS to four VM hypervisors. Details are listed below, to keep it short, it is a 8x HDD Stripe-Mirror with single SSD divided into ZIL and L2ARC and Swap, Xeon, 16GB ECC, networking over bonded 4x 1Gbe using LACP.
Soon after setting it up, I've noticed the system hangs unexpectedly without printing any output neither in console nor in any logs, here's what happened:
1. Initial setup, 8x HDD stripe mirror + Intel 520 SSD, 8GB ZIL, 196GB L2ARC, no swap, LACP + jumbo frames, default tunables - system freezed in couple of hours, initially responsive to ping, but impossible to log into SSH, seemed like stuck on IO. Before it happened, noticed relatively high CPU usage on System and IRQ.
2. Thought it might be SSD, so replaced it with Intel 320 SSD, left 8GB, 128GB L2ARC, no swap, LACP + jumbo frames, default tunables - again system freezed without crashing in couple of hours.
3. Suspected Jumbo Frames to be the culprit, so reverted back to 1500 MTU - system freezed again but after a longer time.
4. Had an L2ARC hunch, set it in tunables to 12GB and increased l2arc_write_max and l2arc_write_boost to 128MB - system freezed again but this time worked even longer, more than 24 hours, although I wasn't doing any load tests in meanwhile.
5. Finally noticed there is no swap (lol) so I added 16GB parition on the SSD and swapon'ed it - system freezed but again after more than a day, noticed FreeNAS started swapping soon before crashing.
6. Turned on autotune, go 13GB for L2ARC and started doing load tests - freezed again in around 12 hours or so, noticed swapping again soon before system freezed
7. Bless you guys for the "Remote Graphite Server Hostname" option to upload carbon stats, done that, decreased L2ARC to 10GB, started load test to see what's happening - freezed again and rebooted itself around an hour later early morning.
8. Retried load tests and watched Grafana - same result, freezed but this time got a pattern in Grafana dashboard.
So, it seems, FreeNAS, a kernel or some process, is running of of memory, regardless of the L2ARC size, at some point it starts to swap heavily and soon after that, poof... freezed, without even saying oh f*** y** in any logs.
Snapshots of the Grafana dashboards:
- 3 hours before the freeze: https://snapshot.raintank.io/dashboard/snapshot/9xM3Z6ropc6eVBZIUqtC0737Yvw3Y9X0?orgId=2
- 24 hours before the freeze: https://snapshot.raintank.io/dashboard/snapshot/zIYgjKmangIGNhKV6XbeoqNrwW9GhuUp?orgId=2
Have a look on the ZFS Hit Ratio, Memory, Swap, and SSD IOPS.
Hardware:
- Intel S3420GP mobo
- Xeon X3420
- 16GB DDR3 ECC
- 2x onboard Intel NICs used for management only
- 4x Intel 82575GB 1GBe, bonded in LACP, no jumbo currently
- 2x Marvell 88SE9235 controllers, two separate 4x SATA controllers, on dedicated to two HDD, one to SSD
Attached: lspci, ps aux, sysctl -a, vmstat, zfs-stat taken soon before freeze.
Hope that helps, in meanwhile I'm going to downgrade to 9.10 (?) to see if the issue occurs again, after all need a stable storage :)
first of all, I would like to thank iXsystems guys and FreeBSD team for amazing piece of software FreeNAS is. I've been using it for more than 5 years and always been happy with stability and feature set FreeNAS offers.
Recently, decided to restore my server lab used for development and devops muscle training and rebuilt the primary storage server that exposes NFS to four VM hypervisors. Details are listed below, to keep it short, it is a 8x HDD Stripe-Mirror with single SSD divided into ZIL and L2ARC and Swap, Xeon, 16GB ECC, networking over bonded 4x 1Gbe using LACP.
Soon after setting it up, I've noticed the system hangs unexpectedly without printing any output neither in console nor in any logs, here's what happened:
1. Initial setup, 8x HDD stripe mirror + Intel 520 SSD, 8GB ZIL, 196GB L2ARC, no swap, LACP + jumbo frames, default tunables - system freezed in couple of hours, initially responsive to ping, but impossible to log into SSH, seemed like stuck on IO. Before it happened, noticed relatively high CPU usage on System and IRQ.
2. Thought it might be SSD, so replaced it with Intel 320 SSD, left 8GB, 128GB L2ARC, no swap, LACP + jumbo frames, default tunables - again system freezed without crashing in couple of hours.
3. Suspected Jumbo Frames to be the culprit, so reverted back to 1500 MTU - system freezed again but after a longer time.
4. Had an L2ARC hunch, set it in tunables to 12GB and increased l2arc_write_max and l2arc_write_boost to 128MB - system freezed again but this time worked even longer, more than 24 hours, although I wasn't doing any load tests in meanwhile.
5. Finally noticed there is no swap (lol) so I added 16GB parition on the SSD and swapon'ed it - system freezed but again after more than a day, noticed FreeNAS started swapping soon before crashing.
6. Turned on autotune, go 13GB for L2ARC and started doing load tests - freezed again in around 12 hours or so, noticed swapping again soon before system freezed
7. Bless you guys for the "Remote Graphite Server Hostname" option to upload carbon stats, done that, decreased L2ARC to 10GB, started load test to see what's happening - freezed again and rebooted itself around an hour later early morning.
8. Retried load tests and watched Grafana - same result, freezed but this time got a pattern in Grafana dashboard.
So, it seems, FreeNAS, a kernel or some process, is running of of memory, regardless of the L2ARC size, at some point it starts to swap heavily and soon after that, poof... freezed, without even saying oh f*** y** in any logs.
Snapshots of the Grafana dashboards:
- 3 hours before the freeze: https://snapshot.raintank.io/dashboard/snapshot/9xM3Z6ropc6eVBZIUqtC0737Yvw3Y9X0?orgId=2
- 24 hours before the freeze: https://snapshot.raintank.io/dashboard/snapshot/zIYgjKmangIGNhKV6XbeoqNrwW9GhuUp?orgId=2
Have a look on the ZFS Hit Ratio, Memory, Swap, and SSD IOPS.
Hardware:
- Intel S3420GP mobo
- Xeon X3420
- 16GB DDR3 ECC
- 2x onboard Intel NICs used for management only
- 4x Intel 82575GB 1GBe, bonded in LACP, no jumbo currently
- 2x Marvell 88SE9235 controllers, two separate 4x SATA controllers, on dedicated to two HDD, one to SSD
Attached: lspci, ps aux, sysctl -a, vmstat, zfs-stat taken soon before freeze.
Hope that helps, in meanwhile I'm going to downgrade to 9.10 (?) to see if the issue occurs again, after all need a stable storage :)
