danb35
Hall of Famer
- Joined
- Aug 16, 2011
- Messages
- 15,504
tl;dr: In the last few weeks, my NFS performance has fallen off a cliff, and I can't figure out why--between systems on a 10G link, I'm seeing data rates under 10 MByte/sec.
Among other things, my FreeNAS server acts as a datastore for my three-node Proxmox cluster. Until recently (more on that later), the VM images were stored on my data pool, consisting of four six-disk RAIDZ2 vdevs (I know--I figured the number of vdevs would still allow for decent IOPS). The pool is about 35% full right now. And for most of this year, it's performed just fine in this configuration. I never did any benchmarking, but I never felt I had a reason to. About a few weeks ago, I started noticing that performance of my VMs was very poor, particularly in I/O. The NFS connection became slow enough and unstable enough that Proxmox would frequently disable that storage backend entirely.
I tried a few simple things--turning off NFS on the FreeNAS box and then re-enabling it, upgrading to -U4.1. No help there. So I figured the RAIDZ2 pool was catching up with me, and built a new pool, currently of 3 x 2-disk mirrors of 3 TB SAS disks. This didn't help either--I'm seeing the same poor performance from both pools.
My FreeNAS server, all the Proxmox systems, and my test FreeNAS server all have Chelsio T420-CR NICs. They're connected via a Dell PowerConnect X1052 and a small Mikrotik switch. iperf3 between any two systems reports 5+ Gbit/sec--not ideal, but ample for my needs.
So to try to quantify the problem, I logged into one of the Proxmox servers, changed to one of the NFS mountpoints (the RAIDZ2 pool), and ran
So this points the problem to the FreeNAS box--performance is very poor with multiple clients, whether with Linux or FreeBSD. Both datasets are sync=disabled. What else should I check?
Hardware:
Among other things, my FreeNAS server acts as a datastore for my three-node Proxmox cluster. Until recently (more on that later), the VM images were stored on my data pool, consisting of four six-disk RAIDZ2 vdevs (I know--I figured the number of vdevs would still allow for decent IOPS). The pool is about 35% full right now. And for most of this year, it's performed just fine in this configuration. I never did any benchmarking, but I never felt I had a reason to. About a few weeks ago, I started noticing that performance of my VMs was very poor, particularly in I/O. The NFS connection became slow enough and unstable enough that Proxmox would frequently disable that storage backend entirely.
I tried a few simple things--turning off NFS on the FreeNAS box and then re-enabling it, upgrading to -U4.1. No help there. So I figured the RAIDZ2 pool was catching up with me, and built a new pool, currently of 3 x 2-disk mirrors of 3 TB SAS disks. This didn't help either--I'm seeing the same poor performance from both pools.
My FreeNAS server, all the Proxmox systems, and my test FreeNAS server all have Chelsio T420-CR NICs. They're connected via a Dell PowerConnect X1052 and a small Mikrotik switch. iperf3 between any two systems reports 5+ Gbit/sec--not ideal, but ample for my needs.
So to try to quantify the problem, I logged into one of the Proxmox servers, changed to one of the NFS mountpoints (the RAIDZ2 pool), and ran
dd if=bigfile of=/dev/null for a few large files (hundreds of MB). The results weren't super-consistent--highest was 10.6 MByte/sec, lowest was 5.9--but they were still far lower than I would have expected. I tried it with a file on the mirrored pool, and had about the same result--9.3 MB/sec. Wondering if this was something unique to Linux or Debian's configuration, I tried it on my test FreeNAS box--mounted that same export at the CLI and ran the same command. Results were even worse--about 1.6 MB/sec.So this points the problem to the FreeNAS box--performance is very poor with multiple clients, whether with Linux or FreeBSD. Both datasets are sync=disabled. What else should I check?
Hardware:
FreeNAS 11.3-U4.1
SuperMicro SuperStorage Server 6047R-E1R36L (Motherboard: X9DRD-7LN4F-JBOD, Chassis: SuperChassis 847E16-R1K28LPB)
2 x Xeon E5-2670, 128 GB RAM, Chelsio T420E-CR
Pool: 6 x 6 TB RAIDZ2, 6 x 4 TB RAIDZ2, 6 x 8 TB RAIDZ2, 6 x 12 TB RAIDZ2; 3 x 2 x 3 TB mirrors
SuperMicro SuperStorage Server 6047R-E1R36L (Motherboard: X9DRD-7LN4F-JBOD, Chassis: SuperChassis 847E16-R1K28LPB)
2 x Xeon E5-2670, 128 GB RAM, Chelsio T420E-CR
Pool: 6 x 6 TB RAIDZ2, 6 x 4 TB RAIDZ2, 6 x 8 TB RAIDZ2, 6 x 12 TB RAIDZ2; 3 x 2 x 3 TB mirrors
Dell PowerEdge C6100
2 x Xeon X5650, 48 GB RAM, Chelsio T420E-CR
2 x Xeon X5650, 48 GB RAM, Chelsio T420E-CR
