acquacow
Explorer
- Joined
- Sep 7, 2018
- Messages
- 51
Hi,
New to FreeNAS, have it setup on a supermicro xeon-D 1541, 16GB of DRAM, with four Intel S3500 480GB SSDs as one storage pool and eight 7200RPM HGST 4TB drives as a second pool. Switch is a Netgear XS708T, jumbo frames were on, but I've shut them off everywhere currently while troubleshooting.
I had it setup and working and was getting great reads:
and writes:
I was using 30GB of ISOs as my test data and everything was great. I originally only had 5 HDDs in it until I could prove out that I could max speeds with an SSD tier. HDD speeds were good, at the expected ~150MB/sec per drive. Once I saw the SSD speeds and saw they were consistent over several hours of testing, I shut the box down and bought 3 more HDDs to max out the storage available. Once the box came back up, I deleted my HDD pool, re-created it as a 1x8 arrangement as raidz2. Writes were still great and I started moving data into it, but when I decided to test reads again, they were only 150MB/sec. And then I tested the SSD tier and the reads there were also 150MB/sec...
I broke out iperf to try and eliminate the networking as an issue, things seem fine here:
#freenas as client
#windows as client
Freenas as a client is a tad slower, but it doesn't have 4.5GHz cores like my desktop does. It's still way more throughput than the 1.2GBit I'm seeing on reads from FreeNAS volumes.
Compression is off, atime is off, dedupe is off...
ix0 looked like this when jumbo frames were on:
ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether ac:1f:6b:60:52:7c
hwaddr ac:1f:6b:60:52:7c
inet 192.168.2.28 netmask 0xffffff00 broadcast 192.168.2.255
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (10Gbase-T <full-duplex>)
status: active
I've set the SSD stripe up as 1x4 and as 2x2 with no change in performance.
Same thing:
I tried with SMB Multichannel on/off and there was no difference there.
I tried starting a second read while the first was going only to have the speeds both run at 50%
I can still copy data at full speed across my network to other VMs just fine, just not FreeNAS.
Thoughts?
Thanks,
-- Dave
New to FreeNAS, have it setup on a supermicro xeon-D 1541, 16GB of DRAM, with four Intel S3500 480GB SSDs as one storage pool and eight 7200RPM HGST 4TB drives as a second pool. Switch is a Netgear XS708T, jumbo frames were on, but I've shut them off everywhere currently while troubleshooting.
I had it setup and working and was getting great reads:
and writes:
I was using 30GB of ISOs as my test data and everything was great. I originally only had 5 HDDs in it until I could prove out that I could max speeds with an SSD tier. HDD speeds were good, at the expected ~150MB/sec per drive. Once I saw the SSD speeds and saw they were consistent over several hours of testing, I shut the box down and bought 3 more HDDs to max out the storage available. Once the box came back up, I deleted my HDD pool, re-created it as a 1x8 arrangement as raidz2. Writes were still great and I started moving data into it, but when I decided to test reads again, they were only 150MB/sec. And then I tested the SSD tier and the reads there were also 150MB/sec...
I broke out iperf to try and eliminate the networking as an issue, things seem fine here:
#freenas as client
Code:
root@freenas:~ # iperf3 -c 192.168.2.200 -w64k -P6 Connecting to host 192.168.2.200, port 5201 [ 5] local 192.168.2.28 port 63491 connected to 192.168.2.200 port 5201 [ 7] local 192.168.2.28 port 56686 connected to 192.168.2.200 port 5201 [ 9] local 192.168.2.28 port 47797 connected to 192.168.2.200 port 5201 [ 11] local 192.168.2.28 port 62493 connected to 192.168.2.200 port 5201 [ 13] local 192.168.2.28 port 35516 connected to 192.168.2.200 port 5201 [ 15] local 192.168.2.28 port 25206 connected to 192.168.2.200 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 1.00-2.00 sec 158 MBytes 1.32 Gbits/sec 0 64.0 KBytes [ 7] 1.00-2.00 sec 156 MBytes 1.31 Gbits/sec 0 64.0 KBytes [ 9] 1.00-2.00 sec 132 MBytes 1.11 Gbits/sec 0 64.0 KBytes [ 11] 1.00-2.00 sec 132 MBytes 1.11 Gbits/sec 0 64.0 KBytes [ 13] 1.00-2.00 sec 154 MBytes 1.29 Gbits/sec 0 64.0 KBytes [ 15] 1.00-2.00 sec 158 MBytes 1.32 Gbits/sec 0 64.0 KBytes [SUM] 1.00-2.00 sec 889 MBytes 7.46 Gbits/sec 0
#windows as client
Code:
PS O:\Download\iperf-3.1.3-win64> .\iperf3.exe -c 192.168.2.28 -w64k -P6 Connecting to host 192.168.2.28, port 5201 [ 4] local 192.168.2.200 port 51622 connected to 192.168.2.28 port 5201 [ 6] local 192.168.2.200 port 51623 connected to 192.168.2.28 port 5201 [ 8] local 192.168.2.200 port 51624 connected to 192.168.2.28 port 5201 [ 10] local 192.168.2.200 port 51625 connected to 192.168.2.28 port 5201 [ 12] local 192.168.2.200 port 51626 connected to 192.168.2.28 port 5201 [ 14] local 192.168.2.200 port 51627 connected to 192.168.2.28 port 5201 [ ID] Interval Transfer Bandwidth [ 4] 1.00-2.00 sec 161 MBytes 1.35 Gbits/sec [ 6] 1.00-2.00 sec 164 MBytes 1.37 Gbits/sec [ 8] 1.00-2.00 sec 225 MBytes 1.88 Gbits/sec [ 10] 1.00-2.00 sec 248 MBytes 2.08 Gbits/sec [ 12] 1.00-2.00 sec 161 MBytes 1.35 Gbits/sec [ 14] 1.00-2.00 sec 159 MBytes 1.33 Gbits/sec [SUM] 1.00-2.00 sec 1.09 GBytes 9.37 Gbits/sec
Freenas as a client is a tad slower, but it doesn't have 4.5GHz cores like my desktop does. It's still way more throughput than the 1.2GBit I'm seeing on reads from FreeNAS volumes.
Compression is off, atime is off, dedupe is off...
ix0 looked like this when jumbo frames were on:
ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether ac:1f:6b:60:52:7c
hwaddr ac:1f:6b:60:52:7c
inet 192.168.2.28 netmask 0xffffff00 broadcast 192.168.2.255
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (10Gbase-T <full-duplex>)
status: active
I've set the SSD stripe up as 1x4 and as 2x2 with no change in performance.
Same thing:
I tried with SMB Multichannel on/off and there was no difference there.
I tried starting a second read while the first was going only to have the speeds both run at 50%
I can still copy data at full speed across my network to other VMs just fine, just not FreeNAS.
Thoughts?
Thanks,
-- Dave
Last edited:
