Hectactical
Cadet
- Joined
- Feb 23, 2019
- Messages
- 2
Hello Everyone,
Hello! I'm a relatively new user to FreeNAS, and wanted to share an issue I had recently. I use my appliance for mostly iSCSI duties to serve as backend storage for a few ESX VM hosts. While the hardware varies (in both generation and brand) between all of these components, pretty much everything uses an Intel X520-DA2. Originally selected the 2 port variants to reserve one port for iSCSI and the other for Management/Shares (Not iSCSI)/etc.
While 10gbps is a little overkill for home, the cables/adapters are getting fairly inexpensive, and had scored a deal on a Quanta 24 port SFP+ switch about a year ago. At the time, my storage couldn't saturate it, but 10g simplified the cabling a bit, eliminating the need for link aggregation for the most part on my servers. Last fall, I rebuilt my storage server using FreeNAS. I have been very pleased with the performance and functionality, though when I finally got around to benchmarking the storage subsystems from a windows VM, noticed that CrystalDiskMark (quick and dirty testing) was reporting around 524MB/Sec read and 1194MB/Sec write. While this isn't slow, it struck me as odd that both HDD and SSD Datastores were reporting about the same level of performance.
The VMWare datastores correspond to the HDD/SSD pools, each with 8 drives in a mirrored/striped volume. I've added zvols on top of those, and attached them as devices to iSCSI targets. Adding them to ESXi was a breeze!
Anyway, after much testing, I ended up punting a bit and swapping the X520-DA2 in the FreeNAS appliance to an X710-DA2 and noted much improved performance on the TX front. Question -- has anyone seen something similar? Have done a bit of searching around the forums and google/bing, but so far been pretty inconclusive. Are there any tunables I may have missed?
Selected the X710 based on a STH article:
https://www.servethehome.com/buyers...as-servers/top-picks-freenas-nics-networking/
For those interested, here's the details of testing/etc:
==============
Original System Specs
==============
Rosewill RSV-L4412
Intel 1245v2 Socket 1155 (Ivy Bridge @ 3.4 GHz 77W TDP)
Tyan S5510 (4 PCI-E 8x slot variant)
Kingston 32 GB RAM (4x8 ECC DDR3 UDIMM)
Intel X520-DA2 (Using Cisco HB-CU-3M Twinax Cables)
LSI 9300-8i 12Gb/s HBA
Adaptec AEC-82885T 12Gb/s Expander
8 x Seagate 3tb SATA 6Gb/s Drives (Grouped into an RAID10 Volume)
8 x Mixed 480 GB SSD SATA 6Gb/s Drives (Grouped into an RAID10 Volume) - in an external enclosure (planning to move to an internal eventually)
===========
ESXi Hosts Specs
===========
2 x Dell R230 Xeon 1220v5 w/32GB DDR4 ECC
1 x Generic Dual 2680v2 w/128GB DDR3 ECC
All use Intel x520-DA2 (Using Cisco HB-CU-3M Twinax Cables)
All using v6.7 U1
==========
Network Switch
==========
Ubiquiti 16 Port Edgeswitch (12 SFP+, 4 RJ45)
==============
Testing Methodology
==============
Primarily iPerf. iPerf already installed on FreeNAS by default (ran iperf -s from shell). Installed iPerf Offline Bundle on ESXi Hosts:
https://vibsdepot.v-front.de/wiki/index.php/Iperf
General process, upload to datastore, install offline bundle via SSH (enable on host via web/console):
After Installing, ran iPerf from ESXi hosts with a total test runtime for 60 seconds, displaying speed reports every 10 seconds, with a bidirectional switch to test both TX/RX. Observed results similar to the following:
172.16.192.21 = FreeNAS
172.16.192.15 = ESXi Host
Note in the second phase of the test, iPerf is testing TX from the perspective of the FreeNAS. This would correspond to to READ activity from an iSCSI Client perspective. Host to Host iPerfs looked clean:
172.16.192.11 = ESXi Host
172.16.192.15 = ESXi Host
While not quite 10gbps, the below result shows the host is capable of more than 5.47Gbps on RX. Examining the switchports on the network hardware, no port errors observed.
===========
Troubleshooting
===========
- Swapped FreeNAS iSCSI port Twinax cable to another known good.
- Found that FreeNAS motherboard is only PCI-E 8x on Slots 7 & 8 (Closest to CPU). The remaining ones are 4x, while the connector is 8x.
- Purchased x710-da2 for FreeNAS **
Hello! I'm a relatively new user to FreeNAS, and wanted to share an issue I had recently. I use my appliance for mostly iSCSI duties to serve as backend storage for a few ESX VM hosts. While the hardware varies (in both generation and brand) between all of these components, pretty much everything uses an Intel X520-DA2. Originally selected the 2 port variants to reserve one port for iSCSI and the other for Management/Shares (Not iSCSI)/etc.
While 10gbps is a little overkill for home, the cables/adapters are getting fairly inexpensive, and had scored a deal on a Quanta 24 port SFP+ switch about a year ago. At the time, my storage couldn't saturate it, but 10g simplified the cabling a bit, eliminating the need for link aggregation for the most part on my servers. Last fall, I rebuilt my storage server using FreeNAS. I have been very pleased with the performance and functionality, though when I finally got around to benchmarking the storage subsystems from a windows VM, noticed that CrystalDiskMark (quick and dirty testing) was reporting around 524MB/Sec read and 1194MB/Sec write. While this isn't slow, it struck me as odd that both HDD and SSD Datastores were reporting about the same level of performance.
The VMWare datastores correspond to the HDD/SSD pools, each with 8 drives in a mirrored/striped volume. I've added zvols on top of those, and attached them as devices to iSCSI targets. Adding them to ESXi was a breeze!
Anyway, after much testing, I ended up punting a bit and swapping the X520-DA2 in the FreeNAS appliance to an X710-DA2 and noted much improved performance on the TX front. Question -- has anyone seen something similar? Have done a bit of searching around the forums and google/bing, but so far been pretty inconclusive. Are there any tunables I may have missed?
Selected the X710 based on a STH article:
https://www.servethehome.com/buyers...as-servers/top-picks-freenas-nics-networking/
For those interested, here's the details of testing/etc:
==============
Original System Specs
==============
Rosewill RSV-L4412
Intel 1245v2 Socket 1155 (Ivy Bridge @ 3.4 GHz 77W TDP)
Tyan S5510 (4 PCI-E 8x slot variant)
Kingston 32 GB RAM (4x8 ECC DDR3 UDIMM)
Intel X520-DA2 (Using Cisco HB-CU-3M Twinax Cables)
LSI 9300-8i 12Gb/s HBA
Adaptec AEC-82885T 12Gb/s Expander
8 x Seagate 3tb SATA 6Gb/s Drives (Grouped into an RAID10 Volume)
8 x Mixed 480 GB SSD SATA 6Gb/s Drives (Grouped into an RAID10 Volume) - in an external enclosure (planning to move to an internal eventually)
===========
ESXi Hosts Specs
===========
2 x Dell R230 Xeon 1220v5 w/32GB DDR4 ECC
1 x Generic Dual 2680v2 w/128GB DDR3 ECC
All use Intel x520-DA2 (Using Cisco HB-CU-3M Twinax Cables)
All using v6.7 U1
==========
Network Switch
==========
Ubiquiti 16 Port Edgeswitch (12 SFP+, 4 RJ45)
==============
Testing Methodology
==============
Primarily iPerf. iPerf already installed on FreeNAS by default (ran iperf -s from shell). Installed iPerf Offline Bundle on ESXi Hosts:
https://vibsdepot.v-front.de/wiki/index.php/Iperf
General process, upload to datastore, install offline bundle via SSH (enable on host via web/console):
Code:
esxcli software vib install -d /vmfs/volumes/[VMFS Name]/ iperf-2.0.5-1-offline_bundle.zip --no-sig-check Installation Result Message: Operation finished successfully. Reboot Required: false VIBs Installed: DAST_bootbank_iperf_2.0.5-1 VIBs Removed: VIBs Skipped:
After Installing, ran iPerf from ESXi hosts with a total test runtime for 60 seconds, displaying speed reports every 10 seconds, with a bidirectional switch to test both TX/RX. Observed results similar to the following:
172.16.192.21 = FreeNAS
172.16.192.15 = ESXi Host
Code:
/opt/iperf/bin/iperf -c 172.16.192.21 -t 60 -i 10 -r ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 64.0 KByte (default) ------------------------------------------------------------ ------------------------------------------------------------ Client connecting to 172.16.192.21, TCP port 5001 TCP window size: 256 KByte (default) ------------------------------------------------------------ [ 5] local 172.16.192.15 port 11787 connected with 172.16.192.21 port 5001 [ ID] Interval Transfer Bandwidth [ 5] 0.0-10.0 sec 11.5 GBytes 9.89 Gbits/sec [ 5] 10.0-20.0 sec 11.5 GBytes 9.89 Gbits/sec [ 5] 20.0-30.0 sec 11.5 GBytes 9.89 Gbits/sec [ 5] 30.0-40.0 sec 11.5 GBytes 9.89 Gbits/sec [ 5] 40.0-50.0 sec 11.5 GBytes 9.89 Gbits/sec [ 5] 50.0-60.0 sec 11.5 GBytes 9.89 Gbits/sec [ 5] 0.0-60.0 sec 69.1 GBytes 9.89 Gbits/sec [ 4] local 172.16.192.15 port 5001 connected with 172.16.192.21 port 29047 [ 4] 0.0-10.0 sec 6.34 GBytes 5.45 Gbits/sec [ 4] 10.0-20.0 sec 6.37 GBytes 5.47 Gbits/sec [ 4] 20.0-30.0 sec 6.37 GBytes 5.47 Gbits/sec [ 4] 30.0-40.0 sec 6.37 GBytes 5.47 Gbits/sec [ 4] 40.0-50.0 sec 6.37 GBytes 5.47 Gbits/sec [ 4] 50.0-60.0 sec 6.37 GBytes 5.47 Gbits/sec [ 4] 0.0-60.0 sec 38.2 GBytes 5.47 Gbits/sec
Note in the second phase of the test, iPerf is testing TX from the perspective of the FreeNAS. This would correspond to to READ activity from an iSCSI Client perspective. Host to Host iPerfs looked clean:
172.16.192.11 = ESXi Host
172.16.192.15 = ESXi Host
Code:
------------------------------------------------------------ Client connecting to 172.16.192.11, TCP port 5001 TCP window size: 107 KByte (default) ------------------------------------------------------------ [ 5] local 172.16.192.15 port 55124 connected with 172.16.192.11 port 5001 [ 5] 0.0-180.0 sec 164 GBytes 7.82 Gbits/sec [ 4] local 172.16.192.15 port 5001 connected with 172.16.192.11 port 21841 [ 4] 0.0-180.0 sec 174 GBytes 8.32 Gbits/sec
While not quite 10gbps, the below result shows the host is capable of more than 5.47Gbps on RX. Examining the switchports on the network hardware, no port errors observed.
===========
Troubleshooting
===========
- Swapped FreeNAS iSCSI port Twinax cable to another known good.
Result: No Change
- Found that FreeNAS motherboard is only PCI-E 8x on Slots 7 & 8 (Closest to CPU). The remaining ones are 4x, while the connector is 8x.
X520-DA2 is PCI-E 2.0 8x
Was in 4x slot; moved to 8x slot; moved expander to farthest slot from CPU, Slot 4.
Result: No Change
- Purchased x710-da2 for FreeNAS **
Swapped adapters, ended up appearing as separate NICs, deleted originals, and reconfigured new to match original.
Result: Much improved performance for TX on FreeNAS
Code:
opt/iperf/bin/iperf -c 172.16.192.21 -t 60 -i 15 -r------------------------------------------------------------Server listening on TCP port 5001TCP window size: 64.0 KByte (default)------------------------------------------------------------------------------------------------------------------------Client connecting to 172.16.192.21, TCP port 5001TCP window size: 256 KByte (default)------------------------------------------------------------[ 5] local 172.16.192.15 port 56773 connected with 172.16.192.21 port 5001[ ID] Interval Transfer Bandwidth[ 5] 0.0-15.0 sec 17.3 GBytes 9.89 Gbits/sec[ 5] 15.0-30.0 sec 17.3 GBytes 9.89 Gbits/sec[ 5] 30.0-45.0 sec 17.3 GBytes 9.89 Gbits/sec[ 5] 45.0-60.0 sec 17.3 GBytes 9.89 Gbits/sec[ 5] 0.0-60.0 sec 69.1 GBytes 9.89 Gbits/sec[ 4] local 172.16.192.15 port 5001 connected with 172.16.192.21 port 49812[ 4] 0.0-15.0 sec 17.3 GBytes 9.89 Gbits/sec[ 4] 15.0-30.0 sec 17.3 GBytes 9.90 Gbits/sec[ 4] 30.0-45.0 sec 17.1 GBytes 9.81 Gbits/sec[ 4] 45.0-60.0 sec 17.2 GBytes 9.83 Gbits/sec[ 4] 0.0-60.0 sec 68.9 GBytes 9.86 Gbits/sec
EDIT: Grammer
Attachments
Last edited:
