I felt it best to stop hogging someone else's thread and decided to make my own. The following continues from my posts here. I'll include a tl;dr as I'm also making the full version very detailed to hopefully save time with questions, but hopefully not deter comments.
Tl;dr
Read speeds can saturate 10GBe, write speeds can't break half. Any further suggestions?
Much longer version:
Setup/Hardware
I have an r620 that I am using as a virtual machine box. The PM OS and VM virtual disks are stored on the single RAID1 virtual disk made of a pair of Samsung 860 EVO SSDs. Specs of the system are as follows:
Proxmox v6.1
2x e5-2630L
128gb 1333MHz DDR3 ECC RAM
H710 Mini mono RAID card
I also have an R720XD LFF that runs FreeNAS, which is used for storage of large files (ie: movies) that will be accessed over my LAN or via the VM for various computing needs (ie: transcoding for Plex). The 12x 12TB drives are split into a pool consisting of 2x 6-drive RaidZ2 VDEVs. Specs for this are as follows:
FreeNAS 11.3RC1
2x e5-2630
128gb 1333MHz DDR3 ECC RAM
LSI 9208-8i HBA
12x WD 12tb EasyStore drives, shucked.
I have one of the 1gbe ports on each machine connected to my local network and the two machines are direct connected to each other via SFP+ DAC on a 10Gb port.
All file sharing between devices is done using SMB/CIFS as the files need to be accessible by Windows machines and VMs and *nix VMs and SMB is, to me in my limited experience, the easiest way to do this with FreeNAS.
FN Expectations:
Based on this (https://www.ixsystems.com/blog/zfs-pool-performance-2/) information, and given the R/W speeds I saw prior to shucking, the following should be the theoretical limits for my pool for sequential reads (given compression off):
VM Setups
Fedora:
Fedora 31 fresh install with latest updates applied.
VirtIO used for NIC drivers
4CPU cores/8GB of RAM (RAM may be different, if specified in following data tables)
Ubuntu:
Ubuntu 19.10
VirtIO used for NIC drivers
Docker active: no dockers actively performing tasks, and no docker accesses FreeNAS mount Plex but performing no tasks (test folder already scanned to library and no one has access to Plex so no transcoding occurring) 4CPU cores/8GB of RAM (RAM may be different, if specified in following data tables)
Win10VM:
Win10 fresh install, latest updates applied
VirtIO used for NIC drivers
4CPU cores/8GB of RAM (RAM may be different, if specified in following data tables)
Troubleshooting
I've been running a gauntlet of testing through a variety of parameter changes. I won't post ALL of my testing results here, I'll just pick up from where I left off in the referenced post from above. At this point, the changes that actually led to an impact are as follows:
1) Jumbo frames on both FreeNAS and proxmox, along with in the VMs themselves.
2) Setting CPU performance in both machine's BIOSes to max performance.
* I'll put a special note here that setting the CPU on FreeNAS to max performance had the most impact, and I'll reference this in a bit.
3) Turning off HyperThreading on FreeNAS box (I may have to retest this particular setting as I changed CPU performance settings after this so it might do nothing)
The key tests I have been performing are iperf3, dd, pv copies, and copies within Nautilus. After a lovely suggestion from @Donny Davis , I keep an eye on CPU core utilization of smbd on FreeNAS during all of these tests, though I only recorded the results for some.
The non-DD files that I am copying for testing are compressed movie files, so I would expect they don't compress much in ZFS, if at all.
To start, I run iperf3 to ensure that it's not a network issue.
FreeNAS means from the WebUI shell, proxmox means from the webUI shell.
So, it's not a NIC issue between the boxes. The following are the results I get from DD, Nautilus transfers, and pv copies (note: compression is still on unless specified otherwise):
DD
File Transfer Nautilus
File Transfer pv
* "Delete and redo": first value is the first time I copy from FN to the VM. After the copy, I immediately delete from the VM. The second value is copying the same file back to the VM again.
Yes, I intentionally set the RAM to 40GB so as to remove the physical disk speed (SSD) of the VM as a limiting factor (it's only a SATA drive). Even with this, I'm still not getting full speed (10GBe line speed/theoretical drive read speeds, they're identical based on my math above in "FN Expectations"). However, write speeds to FN are spot on or higher(except dd, which I'd expect higher from as it's just a giant file of zeros which should be VERY compressible, which still seems odd).
What I noticed after that round of testing is that read speeds will peg a CPU core to 100% BUT they will operate at line speed. Reads from FN are far less but never peg a CPU core to 100%.
Even though smbd never used a full core, I still noticed an improvement from previous testing where I did NOT have the FN box CPU performance set to max vs the testing above where I *did*. So, on a whim, I decided to swap the processors between the two boxes (as they aren't identical, see above "Setup") just to see what would happen, expecting some sort of change to the results I was getting, mostly expecting worse read results as the processors currently in my FN box are a generation older, though they aren't the low-power version. Those results follow:
IPERF3
DD
File Transfer Nautilus/Explorer (Windows)
File Transfer pv
Interestingly, swapping in the older, but not low-power version, processor into the FN box improved my write speeds. Still, during reads, FreeNAS smbd never pegs a core at 100%, which I would expect to be 100% when the speed isn't full line speed signifying that I am CPU bound on my read speeds, but this may be very wrong.
However, given that changes to the CPU did impact read speeds, I'm still wondering if I am CPU bound and the next step is to move to a CPU with higher per-thread clock speed.
I've also included some testing done in a Win10VM that is on Proxmox, which has (by far) the worst performance of all of the testing. For the record, I am using VirtIO NIC drivers for all VMs. I haven't done any testing at these phases with my Ubuntu VM as Fedora was also performing a bit better in all tests.
DD on FreeNAS directly
I should have included this earlier, but so it is separate, here are the results of running DD on FreeNAS directly when FreeNAS had the 2630L V2s in (I forgot to re-run with the 2630s):
Final Thoughts and Request for Help
I think I've reached the limits of my knowledge and testing as to what is going on. At this point, I'm thinking these are the most likely :
1) I am still CPU bound on the FN box and need to step up to higher single thread performance.
2) There is something in proxmox VMs related to data ingress that is throttling speeds (not sure how to test this, recommendations welcome)
3) A configuration in FN is hindering my read performance over the network, or possibly in general (see DD on FreeNAS directly results).
4) NIC problem? That's a straight guess.
My two biggest points of confusion are:
1) During reads from FreeNAS over the "network", smbd never fully utilizes a CPU core, as it does on Writes to FreeNAS over the network.
2) Even performing DD on FreeNAS directly, my write speeds are higher than read, though not as large a gap (read speed gets closer to write speed) while, strangely, the results are flipped based on theoretical values.
Any observations or expectations I've made are obviously up for debate and refute as I'm very new to FreeNAS/ZFS. Any observations that you see that might shed some light are appreciated, and I'll gladly do further testing. I'm just looking to maximize the performance of my set-up to help with future use cases.
Cheers
Tl;dr
Read speeds can saturate 10GBe, write speeds can't break half. Any further suggestions?
Much longer version:
Setup/Hardware
I have an r620 that I am using as a virtual machine box. The PM OS and VM virtual disks are stored on the single RAID1 virtual disk made of a pair of Samsung 860 EVO SSDs. Specs of the system are as follows:
Proxmox v6.1
2x e5-2630L
128gb 1333MHz DDR3 ECC RAM
H710 Mini mono RAID card
- 2x Samsung 860 Evo in Raid1
I also have an R720XD LFF that runs FreeNAS, which is used for storage of large files (ie: movies) that will be accessed over my LAN or via the VM for various computing needs (ie: transcoding for Plex). The 12x 12TB drives are split into a pool consisting of 2x 6-drive RaidZ2 VDEVs. Specs for this are as follows:
FreeNAS 11.3RC1
2x e5-2630
128gb 1333MHz DDR3 ECC RAM
LSI 9208-8i HBA
12x WD 12tb EasyStore drives, shucked.
- Before shucking, I tested each on a windows PC and received ~225MB/s R, 200MB/s W sequential
I have one of the 1gbe ports on each machine connected to my local network and the two machines are direct connected to each other via SFP+ DAC on a 10Gb port.
All file sharing between devices is done using SMB/CIFS as the files need to be accessible by Windows machines and VMs and *nix VMs and SMB is, to me in my limited experience, the easiest way to do this with FreeNAS.
FN Expectations:
Based on this (https://www.ixsystems.com/blog/zfs-pool-performance-2/) information, and given the R/W speeds I saw prior to shucking, the following should be the theoretical limits for my pool for sequential reads (given compression off):
- Seq Read: V * ((N – p) * D)
- V = number of VDEVs in pool
- N = total number of drives in VDEV
- P = parity width, or 2 for RaidZ2
- D = Slowest drive speed, in MB/s for given operation in pool (identical for all drives)
- Seq Read should be = 2 * ((6-2)*225) = 900MB/s
- Seq Write = 2 * ((6 – 2) * 200) = 800MB/s
VM Setups
Fedora:
Fedora 31 fresh install with latest updates applied.
VirtIO used for NIC drivers
4CPU cores/8GB of RAM (RAM may be different, if specified in following data tables)
Ubuntu:
Ubuntu 19.10
VirtIO used for NIC drivers
Docker active: no dockers actively performing tasks, and no docker accesses FreeNAS mount Plex but performing no tasks (test folder already scanned to library and no one has access to Plex so no transcoding occurring) 4CPU cores/8GB of RAM (RAM may be different, if specified in following data tables)
Win10VM:
Win10 fresh install, latest updates applied
VirtIO used for NIC drivers
4CPU cores/8GB of RAM (RAM may be different, if specified in following data tables)
Troubleshooting
I've been running a gauntlet of testing through a variety of parameter changes. I won't post ALL of my testing results here, I'll just pick up from where I left off in the referenced post from above. At this point, the changes that actually led to an impact are as follows:
1) Jumbo frames on both FreeNAS and proxmox, along with in the VMs themselves.
2) Setting CPU performance in both machine's BIOSes to max performance.
* I'll put a special note here that setting the CPU on FreeNAS to max performance had the most impact, and I'll reference this in a bit.
3) Turning off HyperThreading on FreeNAS box (I may have to retest this particular setting as I changed CPU performance settings after this so it might do nothing)
The key tests I have been performing are iperf3, dd, pv copies, and copies within Nautilus. After a lovely suggestion from @Donny Davis , I keep an eye on CPU core utilization of smbd on FreeNAS during all of these tests, though I only recorded the results for some.
The non-DD files that I am copying for testing are compressed movie files, so I would expect they don't compress much in ZFS, if at all.
To start, I run iperf3 to ensure that it's not a network issue.
Host 1 | Host 2 | Client | Host | Results (Gbps) | Results (MB/s) |
FreeNAS | Proxmox | Proxmox | FreeNAS | 9.9 | 1238 |
FreeNAS | Proxmox | 9.6 | 1200 |
So, it's not a NIC issue between the boxes. The following are the results I get from DD, Nautilus transfers, and pv copies (note: compression is still on unless specified otherwise):
DD
Host | File Size Used | Write to FN Speed (MB/s) | Read from FN Speed (MB/s) | FN SMBD Observations | Setup Comments |
Fedora VM | 16GB | 753, 820 | 590 | 90% on write, 67% on read | RAM 40GB |
File Transfer Nautilus
Host | Location | Write to FN Speed (MB/s) | Read from FN Speed (MB/s) | FN SMBD Observations | Setup Comments |
Fedora VM | Physical | Too fast (no number shown) | 320, 390 (delete and redo)* | N/A | RAM 40GB |
File Transfer pv
Host | Location | Write to FN Speed (MB/s) | Read from FN Speed (MB/s) | SMBD comments | Setup Comments |
Fedora VM | Physical | 1007 | 345, 410 (delete and redo)* | N/A | RAM 40GB, new file |
Fedora VM | Physical | 938 | 290, 415 (delete and redo)* | N/A | RAM 40GB, new file |
Yes, I intentionally set the RAM to 40GB so as to remove the physical disk speed (SSD) of the VM as a limiting factor (it's only a SATA drive). Even with this, I'm still not getting full speed (10GBe line speed/theoretical drive read speeds, they're identical based on my math above in "FN Expectations"). However, write speeds to FN are spot on or higher
What I noticed after that round of testing is that read speeds will peg a CPU core to 100% BUT they will operate at line speed. Reads from FN are far less but never peg a CPU core to 100%.
Even though smbd never used a full core, I still noticed an improvement from previous testing where I did NOT have the FN box CPU performance set to max vs the testing above where I *did*. So, on a whim, I decided to swap the processors between the two boxes (as they aren't identical, see above "Setup") just to see what would happen, expecting some sort of change to the results I was getting, mostly expecting worse read results as the processors currently in my FN box are a generation older, though they aren't the low-power version. Those results follow:
IPERF3
Host 1 | Host 2 | Client | Host | Results (Gbps) | Results (MB/s) |
FreeNAS | Proxmox | Proxmox | FreeNAS | 9.9 | 1238 |
FreeNAS | Proxmox | 9.7 | 1213 | ||
FreeNAS | Fedora VM | Fedora | FN | 9.8 | 1225 |
FN | Fedora | 9.6 | 1200 | ||
FreeNAS | Win10VM | Win10 | FN | 5.1 | 638 |
FN | Win10 | 2.8 | 350 |
DD
Chrono | Host | File Size Used | Write to FN Speed (MB/s) | Read from FN Speed (MB/s) | FN SMBD Observations | Setup Comments |
30 | Fedora VM | 16GB | 811, 812 | 808, 781 | N/A | RAM 40GB |
File Transfer Nautilus/Explorer (Windows)
Chrono | Host | Location | Write to FN Speed (MB/s) | Read from FN Speed (MB/s) | FN SMBD Observations | Setup Comments |
31 | Fedora VM | Physical | Too fast | 370, 510(delete and redo)* | N/A | RAM 40GB |
32 | Fedora VM | Ramdisk* | 1100 | 445, 580(delete and redo)* | N/A | RAM 40GB |
35 | Win10 VM | Physical | 1000 | 477 | 55%R 95%W | RAM 40GB |
File Transfer pv
Chrono | Host | Location | Write to FN Speed (MB/s) | Read from FN Speed (MB/s) | FN SMBD Observations | Setup Comments |
33 | Fedora VM | Ramdisk* | 793, 798 | 532 - 537 | 95%W | RAM 40GB, new file |
34 | Fedora VM | Physical | 949 | 476 | 56%R | RAM 40GB, new file |
However, given that changes to the CPU did impact read speeds, I'm still wondering if I am CPU bound and the next step is to move to a CPU with higher per-thread clock speed.
I've also included some testing done in a Win10VM that is on Proxmox, which has (by far) the worst performance of all of the testing. For the record, I am using VirtIO NIC drivers for all VMs. I haven't done any testing at these phases with my Ubuntu VM as Fedora was also performing a bit better in all tests.
DD on FreeNAS directly
I should have included this earlier, but so it is separate, here are the results of running DD on FreeNAS directly when FreeNAS had the 2630L V2s in (I forgot to re-run with the 2630s):
Host | File Size Used | Dataset used | Write Speed (MB/s) | Read Speed (MB/s) |
FreeNAS Shell | 400GB | new, empty dataset with comp = off | 1030 | 800 |
FreeNAS Shell | 800GB | new, empty dataset with comp = off | 840 | 725 |
FreeNAS Shell | 400GB | same Movies dataset as over the network tests, comp = lz4 | 1699 | 5753 |
FreeNAS Shell | 1000GB | same Movies dataset as over the network tests, comp = lz4 | 1711 | 5676 |
Final Thoughts and Request for Help
I think I've reached the limits of my knowledge and testing as to what is going on. At this point, I'm thinking these are the most likely :
1) I am still CPU bound on the FN box and need to step up to higher single thread performance.
2) There is something in proxmox VMs related to data ingress that is throttling speeds (not sure how to test this, recommendations welcome)
3) A configuration in FN is hindering my read performance over the network, or possibly in general (see DD on FreeNAS directly results).
4) NIC problem? That's a straight guess.
My two biggest points of confusion are:
1) During reads from FreeNAS over the "network", smbd never fully utilizes a CPU core, as it does on Writes to FreeNAS over the network.
2) Even performing DD on FreeNAS directly, my write speeds are higher than read, though not as large a gap (read speed gets closer to write speed) while, strangely, the results are flipped based on theoretical values.
Any observations or expectations I've made are obviously up for debate and refute as I'm very new to FreeNAS/ZFS. Any observations that you see that might shed some light are appreciated, and I'll gladly do further testing. I'm just looking to maximize the performance of my set-up to help with future use cases.
Cheers
Last edited: