SOLVED Migrated Win2k8R2/RAID card to FreeNAS/ZFS - Write speed went from >100MB/s to ~60MB/s

[NobleKangaroo]

Hi FreeNAS community,

I recently migrated from Windows Server 2008 R2 to FreeNAS. I had tried FreeNAS a long time ago in the past, but never stuck with it as AD integration was shabby, which was a show-stopper for me. It's much better these days, and with my ever-growing media collection I needed the ability to use more than 8 drives, so I figured I'd give it a go again. Unfortunately after swapping OSes and RAID cards, I am seeing much slower performance.

Here's my previous setup:
OS: Windows Server 2008 R2 x64
Case: Norco RPC-4220 (20-bay)
Mobo: SuperMicro MBD-X9SCA-F-O (LGA1155)
CPU: Intel Pentium G2020 (2 cores @ 2.9GHz)
RAM: Kingston 32GB DDR3 ECC (1600 MHz)
Drives: 8x 4TB Western Digital Red (5400 RPM)
NIC: Intel PRO/1000 VT Quad Port EXPI9404VT
Card: Areca ARC-1224-8i
RAID: All drives in RAID 6 (~24 TiB usable)

And what I'm working with at the moment:
OS: FreeNAS-9.10.2-U2 (e1497f2)
Case: Norco RPC-4220 (20-bay)
Mobo: SuperMicro MBD-X9SCA-F-O (LGA1155)
CPU: Intel Pentium G2020 (2 cores @ 2.9GHz)
RAM: Kingston 32GB DDR3 ECC (1600 MHz)
Drives: 12x 4TB Western Digital Red (5400 RPM)
NIC: Intel PRO/1000 VT Quad Port EXPI9404VT
Card: LSI SAS9211-8i (IT mode, P20 firmware)
RAID: 2x RaidZ2, striped (~28.1 TiB usable)

On Win2k8R2, I was seeing write speeds around 100-120 MB/s with a simple CIFS share. With FreeNAS, I'm currently seeing right around 60 MB/s writes speeds on a simple SMB share. CPU and memory usage don't top out (spikes around 40%, nothing too crazy) and network isn't saturated. I would ideally like to fully saturate my 1 Gbps connection, or at best get back up to 100+ MB/s transfer rates. Any recommendations would be appreciated. I can provide screenshots or more info if needed.

Thank you for your help.

 

[Ericloewe]

What's the extra NIC for? The motherboard has two perfectly good NICs already-

 

[NobleKangaroo]

What's the extra NIC for? The motherboard has two perfectly good NICs already-

Thanks for the reply. Was a purchase I made a couple years back, before I realized that LACP doesn't work the way I thought it did. I just left it as is, as I don't have any other use for it currently.

 

[Ericloewe]

Ok, let's start with iPerf, to narrow down the problem. Post the results one you have them (CODE tags, please).

 

[NobleKangaroo]

Ok, let's start with iPerf, to narrow down the problem. Post the results one you have them (CODE tags, please).

iPerf results, default settings, over 3 minutes.

Code:

bin/iperf.exe -c 192.168.1.19 -P 1 -i 1 -p 5001 -f k -t 180
------------------------------------------------------------
Client connecting to 192.168.1.19, TCP port 5001
TCP window size: 63.0 KByte (default)
------------------------------------------------------------
[164] local 192.168.1.31 port 54644 connected with 192.168.1.19 port 5001
[ ID] Interval	   Transfer	 Bandwidth
[164]  0.0- 1.0 sec  105768 KBytes  866451 Kbits/sec
[164]  1.0- 2.0 sec  98088 KBytes  803537 Kbits/sec
[164]  2.0- 3.0 sec  107752 KBytes  882704 Kbits/sec
[164]  3.0- 4.0 sec  109824 KBytes  899678 Kbits/sec
[164]  4.0- 5.0 sec  105768 KBytes  866451 Kbits/sec
[164]  5.0- 6.0 sec  109976 KBytes  900923 Kbits/sec
[164]  6.0- 7.0 sec  104352 KBytes  854852 Kbits/sec
[164]  7.0- 8.0 sec  107488 KBytes  880542 Kbits/sec
[164]  8.0- 9.0 sec  109848 KBytes  899875 Kbits/sec
[164]  9.0-10.0 sec  107712 KBytes  882377 Kbits/sec
[164] 10.0-11.0 sec  111792 KBytes  915800 Kbits/sec
[164] 11.0-12.0 sec  103768 KBytes  850067 Kbits/sec
[164] 12.0-13.0 sec  108800 KBytes  891290 Kbits/sec
[164] 13.0-14.0 sec  110520 KBytes  905380 Kbits/sec
[164] 14.0-15.0 sec  111736 KBytes  915341 Kbits/sec
[164] 15.0-16.0 sec  109712 KBytes  898761 Kbits/sec
[164] 16.0-17.0 sec  105544 KBytes  864616 Kbits/sec
[164] 17.0-18.0 sec  109912 KBytes  900399 Kbits/sec
[164] 18.0-19.0 sec  108824 KBytes  891486 Kbits/sec
[164] 19.0-20.0 sec  110792 KBytes  907608 Kbits/sec
[ ID] Interval	   Transfer	 Bandwidth
[164] 20.0-21.0 sec  104832 KBytes  858784 Kbits/sec
[164] 21.0-22.0 sec  101672 KBytes  832897 Kbits/sec
[164] 22.0-23.0 sec  107168 KBytes  877920 Kbits/sec
[164] 23.0-24.0 sec  111232 KBytes  911213 Kbits/sec
[164] 24.0-25.0 sec  109672 KBytes  898433 Kbits/sec
[164] 25.0-26.0 sec  110408 KBytes  904462 Kbits/sec
[164] 26.0-27.0 sec  103344 KBytes  846594 Kbits/sec
[164] 27.0-28.0 sec  105880 KBytes  867369 Kbits/sec
[164] 28.0-29.0 sec  109624 KBytes  898040 Kbits/sec
[164] 29.0-30.0 sec  111376 KBytes  912392 Kbits/sec
[164] 30.0-31.0 sec  106488 KBytes  872350 Kbits/sec
[164] 31.0-32.0 sec  102592 KBytes  840434 Kbits/sec
[164] 32.0-33.0 sec  109624 KBytes  898040 Kbits/sec
[164] 33.0-34.0 sec  108424 KBytes  888209 Kbits/sec
[164] 34.0-35.0 sec  111472 KBytes  913179 Kbits/sec
[164] 35.0-36.0 sec  109944 KBytes  900661 Kbits/sec
[164] 36.0-37.0 sec  101016 KBytes  827523 Kbits/sec
[164] 37.0-38.0 sec  110160 KBytes  902431 Kbits/sec
[164] 38.0-39.0 sec  100832 KBytes  826016 Kbits/sec
[164] 39.0-40.0 sec  107808 KBytes  883163 Kbits/sec
[ ID] Interval	   Transfer	 Bandwidth
[164] 40.0-41.0 sec  109536 KBytes  897319 Kbits/sec
[164] 41.0-42.0 sec  99216 KBytes  812777 Kbits/sec
[164] 42.0-43.0 sec  106568 KBytes  873005 Kbits/sec
[164] 43.0-44.0 sec  105704 KBytes  865927 Kbits/sec
[164] 44.0-45.0 sec  110984 KBytes  909181 Kbits/sec
[164] 45.0-46.0 sec  108728 KBytes  890700 Kbits/sec
[164] 46.0-47.0 sec  104696 KBytes  857670 Kbits/sec
[164] 47.0-48.0 sec  107952 KBytes  884343 Kbits/sec
[164] 48.0-49.0 sec  109136 KBytes  894042 Kbits/sec
[164] 49.0-50.0 sec  108080 KBytes  885391 Kbits/sec
[164] 50.0-51.0 sec  107744 KBytes  882639 Kbits/sec
[164] 51.0-52.0 sec  103408 KBytes  847118 Kbits/sec
[164] 52.0-53.0 sec  107208 KBytes  878248 Kbits/sec
[164] 53.0-54.0 sec  108424 KBytes  888209 Kbits/sec
[164] 54.0-55.0 sec  105176 KBytes  861602 Kbits/sec
[164] 55.0-56.0 sec  110904 KBytes  908526 Kbits/sec
[164] 56.0-57.0 sec  108616 KBytes  889782 Kbits/sec
[164] 57.0-58.0 sec  108216 KBytes  886505 Kbits/sec
[164] 58.0-59.0 sec  110968 KBytes  909050 Kbits/sec
[164] 59.0-60.0 sec  108248 KBytes  886768 Kbits/sec
[ ID] Interval	   Transfer	 Bandwidth
[164] 60.0-61.0 sec  110856 KBytes  908132 Kbits/sec
[164] 61.0-62.0 sec  106856 KBytes  875364 Kbits/sec
[164] 62.0-63.0 sec  108400 KBytes  888013 Kbits/sec
[164] 63.0-64.0 sec  110104 KBytes  901972 Kbits/sec
[164] 64.0-65.0 sec  103072 KBytes  844366 Kbits/sec
[164] 65.0-66.0 sec  111616 KBytes  914358 Kbits/sec
[164] 66.0-67.0 sec  102712 KBytes  841417 Kbits/sec
[164] 67.0-68.0 sec  109880 KBytes  900137 Kbits/sec
[164] 68.0-69.0 sec  111352 KBytes  912196 Kbits/sec
[164] 69.0-70.0 sec  110960 KBytes  908984 Kbits/sec
[164] 70.0-71.0 sec  110920 KBytes  908657 Kbits/sec
[164] 71.0-72.0 sec  106624 KBytes  873464 Kbits/sec
[164] 72.0-73.0 sec  110920 KBytes  908657 Kbits/sec
[164] 73.0-74.0 sec  111216 KBytes  911081 Kbits/sec
[164] 74.0-75.0 sec  107424 KBytes  880017 Kbits/sec
[164] 75.0-76.0 sec  108856 KBytes  891748 Kbits/sec
[164] 76.0-77.0 sec  107728 KBytes  882508 Kbits/sec
[164] 77.0-78.0 sec  111296 KBytes  911737 Kbits/sec
[164] 78.0-79.0 sec  108208 KBytes  886440 Kbits/sec
[164] 79.0-80.0 sec  108216 KBytes  886505 Kbits/sec
[ ID] Interval	   Transfer	 Bandwidth
[164] 80.0-81.0 sec  106576 KBytes  873071 Kbits/sec
[164] 81.0-82.0 sec  105736 KBytes  866189 Kbits/sec
[164] 82.0-83.0 sec  112144 KBytes  918684 Kbits/sec
[164] 83.0-84.0 sec  110560 KBytes  905708 Kbits/sec
[164] 84.0-85.0 sec  106760 KBytes  874578 Kbits/sec
[164] 85.0-86.0 sec  104752 KBytes  858128 Kbits/sec
[164] 86.0-87.0 sec  103704 KBytes  849543 Kbits/sec
[164] 87.0-88.0 sec  110408 KBytes  904462 Kbits/sec
[164] 88.0-89.0 sec  108656 KBytes  890110 Kbits/sec
[164] 89.0-90.0 sec  109224 KBytes  894763 Kbits/sec
[164] 90.0-91.0 sec  109408 KBytes  896270 Kbits/sec
[164] 91.0-92.0 sec  73456 KBytes  601752 Kbits/sec
[164] 92.0-93.0 sec  62320 KBytes  510525 Kbits/sec
[164] 93.0-94.0 sec  89024 KBytes  729285 Kbits/sec
[164] 94.0-95.0 sec  109672 KBytes  898433 Kbits/sec
[164] 95.0-96.0 sec  110720 KBytes  907018 Kbits/sec
[164] 96.0-97.0 sec  102568 KBytes  840237 Kbits/sec
[164] 97.0-98.0 sec  104160 KBytes  853279 Kbits/sec
[164] 98.0-99.0 sec  107456 KBytes  880280 Kbits/sec
[164] 99.0-100.0 sec  103888 KBytes  851050 Kbits/sec
[ ID] Interval	   Transfer	 Bandwidth
[164] 100.0-101.0 sec  110640 KBytes  906363 Kbits/sec
[164] 101.0-102.0 sec  102608 KBytes  840565 Kbits/sec
[164] 102.0-103.0 sec  91024 KBytes  745669 Kbits/sec
[164] 103.0-104.0 sec  110768 KBytes  907411 Kbits/sec
[164] 104.0-105.0 sec  110160 KBytes  902431 Kbits/sec
[164] 105.0-106.0 sec  107296 KBytes  878969 Kbits/sec
[164] 106.0-107.0 sec  101784 KBytes  833815 Kbits/sec
[164] 107.0-108.0 sec  110008 KBytes  901186 Kbits/sec
[164] 108.0-109.0 sec  106448 KBytes  872022 Kbits/sec
[164] 109.0-110.0 sec  109392 KBytes  896139 Kbits/sec
[164] 110.0-111.0 sec  110656 KBytes  906494 Kbits/sec
[164] 111.0-112.0 sec  104696 KBytes  857670 Kbits/sec
[164] 112.0-113.0 sec  105136 KBytes  861274 Kbits/sec
[164] 113.0-114.0 sec  111456 KBytes  913048 Kbits/sec
[164] 114.0-115.0 sec  109232 KBytes  894829 Kbits/sec
[164] 115.0-116.0 sec  109144 KBytes  894108 Kbits/sec
[164] 116.0-117.0 sec  107904 KBytes  883950 Kbits/sec
[164] 117.0-118.0 sec  100640 KBytes  824443 Kbits/sec
[164] 118.0-119.0 sec  109080 KBytes  893583 Kbits/sec
[164] 119.0-120.0 sec  108272 KBytes  886964 Kbits/sec
[ ID] Interval	   Transfer	 Bandwidth
[164] 120.0-121.0 sec  109680 KBytes  898499 Kbits/sec
[164] 121.0-122.0 sec  105184 KBytes  861667 Kbits/sec
[164] 122.0-123.0 sec  107752 KBytes  882704 Kbits/sec
[164] 123.0-124.0 sec  108568 KBytes  889389 Kbits/sec
[164] 124.0-125.0 sec  104200 KBytes  853606 Kbits/sec
[164] 125.0-126.0 sec  98752 KBytes  808976 Kbits/sec
[164] 126.0-127.0 sec  104200 KBytes  853606 Kbits/sec
[164] 127.0-128.0 sec  107608 KBytes  881525 Kbits/sec
[164] 128.0-129.0 sec  105408 KBytes  863502 Kbits/sec
[164] 129.0-130.0 sec  109248 KBytes  894960 Kbits/sec
[164] 130.0-131.0 sec  108728 KBytes  890700 Kbits/sec
[164] 131.0-132.0 sec  105912 KBytes  867631 Kbits/sec
[164] 132.0-133.0 sec  105064 KBytes  860684 Kbits/sec
[164] 133.0-134.0 sec  101352 KBytes  830276 Kbits/sec
[164] 134.0-135.0 sec  104056 KBytes  852427 Kbits/sec
[164] 135.0-136.0 sec  110688 KBytes  906756 Kbits/sec
[164] 136.0-137.0 sec  107056 KBytes  877003 Kbits/sec
[164] 137.0-138.0 sec  108904 KBytes  892142 Kbits/sec
[164] 138.0-139.0 sec  110216 KBytes  902889 Kbits/sec
[164] 139.0-140.0 sec  110200 KBytes  902758 Kbits/sec
[ ID] Interval	   Transfer	 Bandwidth
[164] 140.0-141.0 sec  109888 KBytes  900202 Kbits/sec
[164] 141.0-142.0 sec  105552 KBytes  864682 Kbits/sec
[164] 142.0-143.0 sec  110008 KBytes  901186 Kbits/sec
[164] 143.0-144.0 sec  110704 KBytes  906887 Kbits/sec
[164] 144.0-145.0 sec  106168 KBytes  869728 Kbits/sec
[164] 145.0-146.0 sec  110440 KBytes  904724 Kbits/sec
[164] 146.0-147.0 sec  105400 KBytes  863437 Kbits/sec
[164] 147.0-148.0 sec  109200 KBytes  894566 Kbits/sec
[164] 148.0-149.0 sec  109032 KBytes  893190 Kbits/sec
[164] 149.0-150.0 sec  108856 KBytes  891748 Kbits/sec
[164] 150.0-151.0 sec  107688 KBytes  882180 Kbits/sec
[164] 151.0-152.0 sec  99784 KBytes  817431 Kbits/sec
[164] 152.0-153.0 sec  109632 KBytes  898105 Kbits/sec
[164] 153.0-154.0 sec  101184 KBytes  828899 Kbits/sec
[164] 154.0-155.0 sec  107752 KBytes  882704 Kbits/sec
[164] 155.0-156.0 sec  111848 KBytes  916259 Kbits/sec
[164] 156.0-157.0 sec  101896 KBytes  834732 Kbits/sec
[164] 157.0-158.0 sec  109960 KBytes  900792 Kbits/sec
[164] 158.0-159.0 sec  108520 KBytes  888996 Kbits/sec
[164] 159.0-160.0 sec  106584 KBytes  873136 Kbits/sec
[ ID] Interval	   Transfer	 Bandwidth
[164] 160.0-161.0 sec  108488 KBytes  888734 Kbits/sec
[164] 161.0-162.0 sec  103784 KBytes  850199 Kbits/sec
[164] 162.0-163.0 sec  103232 KBytes  845677 Kbits/sec
[164] 163.0-164.0 sec  110328 KBytes  903807 Kbits/sec
[164] 164.0-165.0 sec  110408 KBytes  904462 Kbits/sec
[164] 165.0-166.0 sec  107192 KBytes  878117 Kbits/sec
[164] 166.0-167.0 sec  104856 KBytes  858980 Kbits/sec
[164] 167.0-168.0 sec  110584 KBytes  905904 Kbits/sec
[164] 168.0-169.0 sec  110176 KBytes  902562 Kbits/sec
[164] 169.0-170.0 sec  109472 KBytes  896795 Kbits/sec
[164] 170.0-171.0 sec  108800 KBytes  891290 Kbits/sec
[164] 171.0-172.0 sec  103968 KBytes  851706 Kbits/sec
[164] 172.0-173.0 sec  110048 KBytes  901513 Kbits/sec
[164] 173.0-174.0 sec  106640 KBytes  873595 Kbits/sec
[164] 174.0-175.0 sec  109248 KBytes  894960 Kbits/sec
[164] 175.0-176.0 sec  110720 KBytes  907018 Kbits/sec
[164] 176.0-177.0 sec  100064 KBytes  819724 Kbits/sec
[164] 177.0-178.0 sec  98472 KBytes  806683 Kbits/sec
[164] 178.0-179.0 sec  109160 KBytes  894239 Kbits/sec
[164] 179.0-180.0 sec  112144 KBytes  918684 Kbits/sec

Pretty solid near-gigabit speeds. For testing, it's safe to assume I'll be staying with the additional NIC for now.

 

[SweetAndLow]

Test both iperf with the server being on your client Ave freebsd. Both direction is important.

You also said you went from 100mbps to 60MB/s that is about 6x faster. I think you should be getting 1gbps speeds but what your are telling us is still much faster.

Sent from my Nexus 5X using Tapatalk

 

[NobleKangaroo]

Test both iperf with the server being on your client Ave freebsd. Both direction is important.

You also said you went from 100mbps to 60MB/s that is about 6x faster. I think you should be getting 1gbps speeds but what your are telling us is still much faster.

Sent from my Nexus 5X using Tapatalk

Sorry for the confusion there. Accidental lack of capitalization. Meant that I went from 100MBps to 60 MBps.

iPerf the other way, from NAS as client to PC as server

Code:

bin/iperf.exe -s -P 0 -i 1 -p 5001 -f k
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 8.00 KByte (default)
------------------------------------------------------------
[300] local 192.168.1.31 port 5001 connected with 192.168.1.19 port 22309
[ ID] Interval	   Transfer	 Bandwidth
[300]  0.0- 1.0 sec  110587 KBytes  905930 Kbits/sec
[300]  1.0- 2.0 sec  104654 KBytes  857325 Kbits/sec
[300]  2.0- 3.0 sec  101170 KBytes  828787 Kbits/sec
[300]  3.0- 4.0 sec  107401 KBytes  879831 Kbits/sec
[300]  4.0- 5.0 sec  112113 KBytes  918428 Kbits/sec
[300]  5.0- 6.0 sec  110547 KBytes  905600 Kbits/sec
[300]  6.0- 7.0 sec  108752 KBytes  890896 Kbits/sec
[300]  7.0- 8.0 sec  104402 KBytes  855260 Kbits/sec
[300]  8.0- 9.0 sec  107609 KBytes  881532 Kbits/sec
[300]  9.0-10.0 sec  111867 KBytes  916415 Kbits/sec
[300] 10.0-11.0 sec  109233 KBytes  894835 Kbits/sec
[300] 11.0-12.0 sec  107537 KBytes  880945 Kbits/sec
[300] 12.0-13.0 sec  107556 KBytes  881101 Kbits/sec
[300] 13.0-14.0 sec  106869 KBytes  875474 Kbits/sec
[300] 14.0-15.0 sec  109692 KBytes  898593 Kbits/sec
[300] 15.0-16.0 sec  104782 KBytes  858371 Kbits/sec
[300] 16.0-17.0 sec  103802 KBytes  850349 Kbits/sec
[300] 17.0-18.0 sec  99370 KBytes  814040 Kbits/sec
[300] 18.0-19.0 sec  105336 KBytes  862913 Kbits/sec
[300] 19.0-20.0 sec  107002 KBytes  876561 Kbits/sec
[ ID] Interval	   Transfer	 Bandwidth
[300] 20.0-21.0 sec  104838 KBytes  858831 Kbits/sec
[300] 21.0-22.0 sec  108198 KBytes  886355 Kbits/sec
[300] 22.0-23.0 sec  103353 KBytes  846664 Kbits/sec
[300] 23.0-24.0 sec  108308 KBytes  887259 Kbits/sec
[300] 24.0-25.0 sec  109247 KBytes  894950 Kbits/sec
[300] 25.0-26.0 sec  109968 KBytes  900859 Kbits/sec
[300] 26.0-27.0 sec  109909 KBytes  900375 Kbits/sec
[300] 27.0-28.0 sec  107496 KBytes  880606 Kbits/sec
[300] 28.0-29.0 sec  107821 KBytes  883267 Kbits/sec
[300] 29.0-30.0 sec  108429 KBytes  888249 Kbits/sec
[300] 30.0-31.0 sec  110102 KBytes  901958 Kbits/sec
[300] 31.0-32.0 sec  110541 KBytes  905552 Kbits/sec
[300] 32.0-33.0 sec  105869 KBytes  867275 Kbits/sec
[300] 33.0-34.0 sec  109916 KBytes  900433 Kbits/sec
[300] 34.0-35.0 sec  111340 KBytes  912097 Kbits/sec
[300] 35.0-36.0 sec  110140 KBytes  902266 Kbits/sec
[300] 36.0-37.0 sec  108563 KBytes  889352 Kbits/sec
[300] 37.0-38.0 sec  108400 KBytes  888009 Kbits/sec
[300] 38.0-39.0 sec  107672 KBytes  882048 Kbits/sec
[300] 39.0-40.0 sec  110811 KBytes  907767 Kbits/sec
[ ID] Interval	   Transfer	 Bandwidth
[300] 40.0-41.0 sec  110171 KBytes  902518 Kbits/sec
[300] 41.0-42.0 sec  110814 KBytes  907790 Kbits/sec
[300] 42.0-43.0 sec  101507 KBytes  831548 Kbits/sec
[300] 43.0-44.0 sec  107678 KBytes  882095 Kbits/sec
[300] 44.0-45.0 sec  110805 KBytes  907718 Kbits/sec
[300] 45.0-46.0 sec  108944 KBytes  892473 Kbits/sec
[300] 46.0-47.0 sec  109951 KBytes  900715 Kbits/sec
[300] 47.0-48.0 sec  106994 KBytes  876491 Kbits/sec
[300] 48.0-49.0 sec  107137 KBytes  877670 Kbits/sec
[300] 49.0-50.0 sec  109524 KBytes  897217 Kbits/sec
[300] 50.0-51.0 sec  109846 KBytes  899855 Kbits/sec
[300] 51.0-52.0 sec  110891 KBytes  908416 Kbits/sec
[300] 52.0-53.0 sec  107126 KBytes  877574 Kbits/sec
[300] 53.0-54.0 sec  105781 KBytes  866559 Kbits/sec
[300] 54.0-55.0 sec  106364 KBytes  871331 Kbits/sec
[300] 55.0-56.0 sec  110442 KBytes  904745 Kbits/sec
[300] 56.0-57.0 sec  110082 KBytes  901795 Kbits/sec
[300] 57.0-58.0 sec  108383 KBytes  887876 Kbits/sec
[300] 58.0-59.0 sec  110092 KBytes  901875 Kbits/sec
[300] 59.0-60.0 sec  108136 KBytes  885847 Kbits/sec
[ ID] Interval	   Transfer	 Bandwidth
[300] 60.0-61.0 sec  110657 KBytes  906499 Kbits/sec
[300] 61.0-62.0 sec  109514 KBytes  897139 Kbits/sec
[300] 62.0-63.0 sec  100188 KBytes  820736 Kbits/sec
[300] 63.0-64.0 sec  109817 KBytes  899621 Kbits/sec
[300] 64.0-65.0 sec  109919 KBytes  900457 Kbits/sec
[300] 65.0-66.0 sec  109944 KBytes  900658 Kbits/sec
[300] 66.0-67.0 sec  106993 KBytes  876488 Kbits/sec
[300] 67.0-68.0 sec  109876 KBytes  900105 Kbits/sec
[300] 68.0-69.0 sec  110816 KBytes  907803 Kbits/sec
[300] 69.0-70.0 sec  111122 KBytes  910309 Kbits/sec
[300] 70.0-71.0 sec  105953 KBytes  867968 Kbits/sec
[300] 71.0-72.0 sec  110089 KBytes  901853 Kbits/sec
[300] 72.0-73.0 sec  110120 KBytes  902100 Kbits/sec
[300] 73.0-74.0 sec  108881 KBytes  891952 Kbits/sec
[300] 74.0-75.0 sec  107512 KBytes  880735 Kbits/sec
[300] 75.0-76.0 sec  92627 KBytes  758798 Kbits/sec
[300] 76.0-77.0 sec  99774 KBytes  817347 Kbits/sec
[300] 77.0-78.0 sec  97607 KBytes  799597 Kbits/sec
[300] 78.0-79.0 sec  108755 KBytes  890922 Kbits/sec
[300] 79.0-80.0 sec  107840 KBytes  883429 Kbits/sec
[ ID] Interval	   Transfer	 Bandwidth
[300] 80.0-81.0 sec  108418 KBytes  888164 Kbits/sec
[300] 81.0-82.0 sec  109091 KBytes  893671 Kbits/sec
[300] 82.0-83.0 sec  103393 KBytes  846993 Kbits/sec
[300] 83.0-84.0 sec  111724 KBytes  915241 Kbits/sec
[300] 84.0-85.0 sec  109632 KBytes  898107 Kbits/sec
[300] 85.0-86.0 sec  110251 KBytes  903173 Kbits/sec
[300] 86.0-87.0 sec  109622 KBytes  898022 Kbits/sec
[300] 87.0-88.0 sec  105558 KBytes  864730 Kbits/sec
[300] 88.0-89.0 sec  109989 KBytes  901034 Kbits/sec
[300] 89.0-90.0 sec  111200 KBytes  910953 Kbits/sec
[300] 90.0-91.0 sec  109680 KBytes  898502 Kbits/sec
[300] 91.0-92.0 sec  108872 KBytes  891883 Kbits/sec
[300] 92.0-93.0 sec  100956 KBytes  827032 Kbits/sec
[300] 93.0-94.0 sec  107568 KBytes  881195 Kbits/sec
[300] 94.0-95.0 sec  111723 KBytes  915234 Kbits/sec
[300] 95.0-96.0 sec  109166 KBytes  894291 Kbits/sec
[300] 96.0-97.0 sec  107005 KBytes  876586 Kbits/sec
[300] 97.0-98.0 sec  106313 KBytes  870919 Kbits/sec
[300] 98.0-99.0 sec  108922 KBytes  892287 Kbits/sec
[300] 99.0-100.0 sec  109672 KBytes  898433 Kbits/sec
[ ID] Interval	   Transfer	 Bandwidth
[300] 100.0-101.0 sec  109237 KBytes  894873 Kbits/sec
[300] 101.0-102.0 sec  110056 KBytes  901581 Kbits/sec
[300] 102.0-103.0 sec  100443 KBytes  822833 Kbits/sec
[300] 103.0-104.0 sec  109791 KBytes  899404 Kbits/sec
[300] 104.0-105.0 sec  110808 KBytes  907743 Kbits/sec
[300] 105.0-106.0 sec  109422 KBytes  896383 Kbits/sec
[300] 106.0-107.0 sec  109170 KBytes  894321 Kbits/sec
[300] 107.0-108.0 sec  104611 KBytes  856977 Kbits/sec
[300] 108.0-109.0 sec  107150 KBytes  877777 Kbits/sec
[300] 109.0-110.0 sec  108297 KBytes  887169 Kbits/sec
[300] 110.0-111.0 sec  110214 KBytes  902875 Kbits/sec
[300] 111.0-112.0 sec  109357 KBytes  895855 Kbits/sec
[300] 112.0-113.0 sec  102585 KBytes  840373 Kbits/sec
[300] 113.0-114.0 sec  110402 KBytes  904415 Kbits/sec
[300] 114.0-115.0 sec  108126 KBytes  885768 Kbits/sec
[300] 115.0-116.0 sec  110178 KBytes  902581 Kbits/sec
[300] 116.0-117.0 sec  109492 KBytes  896961 Kbits/sec
[300] 117.0-118.0 sec  106850 KBytes  875311 Kbits/sec
[300] 118.0-119.0 sec  109586 KBytes  897727 Kbits/sec
[300] 119.0-120.0 sec  111744 KBytes  915407 Kbits/sec
[ ID] Interval	   Transfer	 Bandwidth
[300] 120.0-121.0 sec  109829 KBytes  899723 Kbits/sec
[300] 121.0-122.0 sec  110764 KBytes  907381 Kbits/sec
[300] 122.0-123.0 sec  107964 KBytes  884442 Kbits/sec
[300] 123.0-124.0 sec  105506 KBytes  864304 Kbits/sec
[300] 124.0-125.0 sec  109082 KBytes  893599 Kbits/sec
[300] 125.0-126.0 sec  110935 KBytes  908776 Kbits/sec
[300] 126.0-127.0 sec  108268 KBytes  886932 Kbits/sec
[300] 127.0-128.0 sec  109640 KBytes  898168 Kbits/sec
[300] 128.0-129.0 sec  105688 KBytes  865798 Kbits/sec
[300] 129.0-130.0 sec  109695 KBytes  898618 Kbits/sec
[300] 130.0-131.0 sec  110760 KBytes  907348 Kbits/sec
[300] 131.0-132.0 sec  110486 KBytes  905101 Kbits/sec
[300] 132.0-133.0 sec  104777 KBytes  858335 Kbits/sec
[300] 133.0-134.0 sec  108570 KBytes  889404 Kbits/sec
[300] 134.0-135.0 sec  108428 KBytes  888243 Kbits/sec
[300] 135.0-136.0 sec  98349 KBytes  805671 Kbits/sec
[300] 136.0-137.0 sec  99988 KBytes  819099 Kbits/sec
[300] 137.0-138.0 sec  97015 KBytes  794743 Kbits/sec
[300] 138.0-139.0 sec  107278 KBytes  878824 Kbits/sec
[300] 139.0-140.0 sec  106508 KBytes  872510 Kbits/sec
[ ID] Interval	   Transfer	 Bandwidth
[300] 140.0-141.0 sec  110498 KBytes  905201 Kbits/sec
[300] 141.0-142.0 sec  108808 KBytes  891356 Kbits/sec
[300] 142.0-143.0 sec  103643 KBytes  849041 Kbits/sec
[300] 143.0-144.0 sec  108859 KBytes  891774 Kbits/sec
[300] 144.0-145.0 sec  108609 KBytes  889723 Kbits/sec
[300] 145.0-146.0 sec  111089 KBytes  910044 Kbits/sec
[300] 146.0-147.0 sec  109261 KBytes  895063 Kbits/sec
[300] 147.0-148.0 sec  107936 KBytes  884208 Kbits/sec
[300] 148.0-149.0 sec  106332 KBytes  871072 Kbits/sec
[300] 149.0-150.0 sec  110316 KBytes  903712 Kbits/sec
[300] 150.0-151.0 sec  110981 KBytes  909154 Kbits/sec
[300] 151.0-152.0 sec  111090 KBytes  910046 Kbits/sec
[300] 152.0-153.0 sec  108032 KBytes  884997 Kbits/sec
[300] 153.0-154.0 sec  110006 KBytes  901171 Kbits/sec
[300] 154.0-155.0 sec  109864 KBytes  900008 Kbits/sec
[300] 155.0-156.0 sec  109806 KBytes  899530 Kbits/sec
[300] 156.0-157.0 sec  108799 KBytes  891279 Kbits/sec
[300] 157.0-158.0 sec  106723 KBytes  874274 Kbits/sec
[300] 158.0-159.0 sec  110813 KBytes  907782 Kbits/sec
[300] 159.0-160.0 sec  111180 KBytes  910783 Kbits/sec
[ ID] Interval	   Transfer	 Bandwidth
[300] 160.0-161.0 sec  108088 KBytes  885454 Kbits/sec
[300] 161.0-162.0 sec  110259 KBytes  903242 Kbits/sec
[300] 162.0-163.0 sec  108794 KBytes  891241 Kbits/sec
[300] 163.0-164.0 sec  108542 KBytes  889174 Kbits/sec
[300] 164.0-165.0 sec  111253 KBytes  911386 Kbits/sec
[300] 165.0-166.0 sec  107651 KBytes  881879 Kbits/sec
[300] 166.0-167.0 sec  108548 KBytes  889227 Kbits/sec
[300] 167.0-168.0 sec  108205 KBytes  886414 Kbits/sec
[300] 168.0-169.0 sec  106854 KBytes  875351 Kbits/sec
[300] 169.0-170.0 sec  109671 KBytes  898425 Kbits/sec
[300] 170.0-171.0 sec  109040 KBytes  893255 Kbits/sec
[300] 171.0-172.0 sec  108894 KBytes  892058 Kbits/sec
[300] 172.0-173.0 sec  109388 KBytes  896109 Kbits/sec
[300] 173.0-174.0 sec  109129 KBytes  893986 Kbits/sec
[300] 174.0-175.0 sec  110053 KBytes  901551 Kbits/sec
[300] 175.0-176.0 sec  111459 KBytes  913069 Kbits/sec
[300] 176.0-177.0 sec  107141 KBytes  877698 Kbits/sec
[300] 177.0-178.0 sec  106774 KBytes  874694 Kbits/sec
[300] 178.0-179.0 sec  109991 KBytes  901043 Kbits/sec
[300] 179.0-180.0 sec  110875 KBytes  908290 Kbits/sec
[ ID] Interval	   Transfer	 Bandwidth
[300]  0.0-180.0 sec  19459072 KBytes  885548 Kbits/sec

 

[Ericloewe]

Are you using an encrypted pool, by any chance?

 

[SweetAndLow]

What is the performance of your pool? Create a dataset without compression. And run dd if=/dev/zero of=/mnt/tank/newdataset/10G.dat bs=1M count=10000

Sent from my Nexus 5X using Tapatalk

 

[NobleKangaroo]

Are you using an encrypted pool, by any chance?

And BINGO was his name-o. I just deleted and re-configured my zpool, without encryption, and I'm now seeing >105MB/s write and >110MB/s read. Just want to say thank you for that. I was driving myself bonkers, twiddling with just about everything else, and it came down to that... Though that's quite a hefty overhead I must say.

So, that said, what's the recommended route to increasing speed if encryption is wanted? I can either create two zpools and split my drives (ie have 8 drives for Plex and 4 for private data), but is there anything else outside of that, perhaps from a ZFS standpoint? I've done the recommended reading and understand that drives are RAIDed together to create vdevs, and vdevs are combined (striped?) to create volumes/zpools. Is there any way to create two volumes on a group of vdevs? I wouldn't think that's possible, given how everything works under the hood, but I could be missing something.

 

[SweetAndLow]

Get a CPU that supports aes-ni.

Sent from my Nexus 5X using Tapatalk

 

[Ericloewe]

Get a CPU that supports aes-ni.

Sent from my Nexus 5X using Tapatalk

This. With AES-NI, encryption is almost transparent, performance-wise.

However, encryption also makes it much easier to lose all your data, so don't do it without a good reason.

 

[NobleKangaroo]

Get a CPU that supports aes-ni.

I just now saw that while re-reading the PowerPoint (Page 31: "CPUs without AES-NI suffer a significant performance penalty."). Definitely something I missed when reading it the first time. Clear case of RTFM. :)

However, encryption also makes it much easier to lose all your data, so don't do it without a good reason.

Are you saying that in the sense of "if you lose your key, you lose your data" or is there something else I am unaware of?



Side note - marking this closed. Thanks to all who helped. It's greatly appreciated.

 

[Ericloewe]

Are you saying that in the sense of "if you lose your key, you lose your data" or is there something else I am unaware of?

Yeah, Bugs are also more likely and the procedures are more complex.