I've got three pools I have been doing some simple dd reads with a 200GB file sending to /dev/null with bs=1M. The pools are similar with number of disks (2 have the same number of disks) and they are all RaidZ3. In the testing where you see 3 in a row of the same test it was done sequentially with no delay. The two similar pools vol1main and vol2main have very different read speeds. vol1main is almost half the speed of vol2main. See below.
mainpool -12 x 3.0TB Seagate NAS drives ST3000VN000-1HJ166 RaidZ3
Reads 611730047 bytes/sec
vol2main -10 x 8.0TB WD80EMAZ-00WJTA0 RaidZ3 - non utilized pool
Reads 742083901 bytes/sec
vol1main -10 x 4.0TB Seagate NAS Drives ST4000VN008-2DR166 RaidZ3 - non utilized pool
Reads 415811235 bytes/sec <-- almost half the speed of the identically configured vol2main
I have tried configuring vol1main (configuration 2 below) to a stripe of 10 disks and I get reads of 373867656 bytes/sec
I've looked at "zpool -v iostat vol1main 1" during the read and the disks all look to be reading at the same time and at similar xfer rates.
I do not know what could be causing the discrepancy in read speed between the problem pool (vol1main) and the other pools. It is almost half the speed of the other pools when configured in the similar RaidZ3. I even split the pool in half and made two 5 disk RaidZ pools thinking if there was something funny going on with a drive or port that the results on the two pools would be different. They weren't.
I've tested each drive in the vol1main pool with "dd if=/dev/daX of=/dev/null bs=1M count=200000" and I can sustain ~198,000,000 Bytes/sec
Looking for any suggestions on things to try.
Test Detail and Results
mainpool -12 x 3.0TB Seagate NAS drives ST3000VN000-1HJ166 RaidZ3
Reads 611730047 bytes/sec
vol2main -10 x 8.0TB WD80EMAZ-00WJTA0 RaidZ3 - non utilized pool
Reads 742083901 bytes/sec
vol1main -10 x 4.0TB Seagate NAS Drives ST4000VN008-2DR166 RaidZ3 - non utilized pool
Reads 415811235 bytes/sec <-- almost half the speed of the identically configured vol2main
I have tried configuring vol1main (configuration 2 below) to a stripe of 10 disks and I get reads of 373867656 bytes/sec
I've looked at "zpool -v iostat vol1main 1" during the read and the disks all look to be reading at the same time and at similar xfer rates.
I do not know what could be causing the discrepancy in read speed between the problem pool (vol1main) and the other pools. It is almost half the speed of the other pools when configured in the similar RaidZ3. I even split the pool in half and made two 5 disk RaidZ pools thinking if there was something funny going on with a drive or port that the results on the two pools would be different. They weren't.
I've tested each drive in the vol1main pool with "dd if=/dev/daX of=/dev/null bs=1M count=200000" and I can sustain ~198,000,000 Bytes/sec
Looking for any suggestions on things to try.
Test Detail and Results
Code:
Setup Test file (~200GB) root@zeus:/mnt/vol2main/cifsshare # dd if=/mnt/vol2main/cifsshare/200g.dd of=/mnt/vol1main/200g.dd bs=1M 200000+0 records in 200000+0 records out 209715200000 bytes transferred in 320.639679 secs (654052551 bytes/sec) System Freenas 11.1U6 128GB system ram, IBM x3650 M3 server , 2x Intel(R) Xeon(R) CPU X5650 @ 2.67GHz Disks are in two DS426 units. 1 cable to each ds4246 from lsi 9207-8e. The drives in the pools are spread equally between the 2 disk enclosures. The thought is to balance the load across the two cables. TESTING mainpool 12 x 3.0TB Seagate NAS drives ST3000VN000-1HJ166 RaidZ3 1: dd if=/mnt/mainpool/200g.dd of=/dev/null bs=1M 200000+0 records in 200000+0 records out 209715200000 bytes transferred in 342.823114 secs (611730047 bytes/sec) 2: dd if=/mnt/mainpool/200g.dd of=/dev/null bs=1M 200000+0 records in 200000+0 records out 209715200000 bytes transferred in 349.308256 secs (600372870 bytes/sec) 3: dd if=/mnt/mainpool/200g.dd of=/dev/null bs=1M 200000+0 records in 200000+0 records out 209715200000 bytes transferred in 361.240751 secs (580541368 bytes/sec) Note: Variations in the throughput are most likely due to the mainpool being home to several ISCSI VM disks. vol2main 10 x 8.0TB WD80EMAZ-00WJTA0 RaidZ3 1: dd if=/mnt/vol2main/cifsshare/200g.dd of=/dev/null bs=1M 200000+0 records in 200000+0 records out 209715200000 bytes transferred in 282.603085 secs (742083901 bytes/sec) 2: dd if=/mnt/vol2main/cifsshare/200g.dd of=/dev/null bs=1M 200000+0 records in 200000+0 records out 209715200000 bytes transferred in 283.664567 secs (739306999 bytes/sec) 3: dd if=/mnt/vol2main/cifsshare/200g.dd of=/dev/null bs=1M 200000+0 records in 200000+0 records out 209715200000 bytes transferred in 283.451890 secs (739861710 bytes/sec) Problem Child (Vol1main) Configuration 1 vol1main 10 x 4.0TB Seagate NAS Drives ST4000VN008-2DR166 RaidZ3 1: dd if=/mnt/vol1main/200g.dd of=/dev/null bs=1M 200000+0 records in 200000+0 records out 209715200000 bytes transferred in 504.351932 secs (415811235 bytes/sec) 2: dd if=/mnt/vol1main/200g.dd of=/dev/null bs=1M 200000+0 records in 200000+0 records out 209715200000 bytes transferred in 508.249956 secs (412622170 bytes/sec) 3: dd if=/mnt/vol1main/200g.dd of=/dev/null bs=1M 200000+0 records in 200000+0 records out 209715200000 bytes transferred in 506.531198 secs (414022277 bytes/sec) Configuration 2 vol1main 10 x 4.0TB Seagate NAS Drives ST4000VN008-2DR166 Stripe 1: dd if=/mnt/vol1main/200g.dd of=/dev/null bs=1M 200000+0 records in 200000+0 records out 209715200000 bytes transferred in 560.934321 secs (373867656 bytes/sec) I split the config for this one into 2 equal pools thinking that if there was something weird going on with one of the drives/ports that the results would be different. They weren't. Configuration 3 vol1main 5 x 4.0TB Seagate NAS Drives ST4000VN008-2DR166 first 5 disks RaidZ 1: dd if=/mnt/vol1main/200g.dd of=/dev/null bs=1M 200000+0 records in 200000+0 records out 209715200000 bytes transferred in 650.448330 secs (322416386 bytes/sec) vol3main 5 x 4.0TB Seagate NAS Drives ST4000VN008-2DR166 second 5 disks RaidZ 1: dd if=/mnt/vol3main/200g.dd of=/dev/null bs=1M 200000+0 records in 200000+0 records out 209715200000 bytes transferred in 651.932346 secs (321682459 bytes/sec)
Last edited:
