So wait, this is unclear... where does iSCSI come into this, then? Where does the RAID50 come into this?
Assuming this is a system you can play around with, you'd be well advised to start at the beginning.
If you're using RAID50 on the PERC5/i, don't. Do what noobsauce80 quoted above. Complex pseudo-RAID levels are often not implemented fully in silicon and can be stressy on the poor little CPU's in cheap RAID controllers. If they ARE implemented in silicon, then they're craptacular and have strange performance characteristics resulting from teeny stripe sizes and other tradeoffs commonly made in low end RAID controllers.
Have the controller export the disks to the host as JBOD. Then start there; measure your performance to each disk individually and in aggregate.
Do this BEFORE you create any filesystems, because while the read test is not destructive, the write test
is data destructive.
Code:
[jgreco@storageX] /# camcontrol devlist
<NECVMWar VMware IDE CDR10 1.00> at scbus1 target 0 lun 0 (pass0,cd0)
<VMware Virtual disk 1.0> at scbus2 target 0 lun 0 (pass1,da0)
<OCZ-AGILITY3 2.15> at scbus4 target 0 lun 0 (pass2,ada0)
<ST3000DM001-1CH166 CC43> at scbus5 target 0 lun 0 (pass3,ada1)
<ST3000DM001-1CH166 CC43> at scbus6 target 0 lun 0 (pass4,ada2)
<ST3000DM001-9YN166 CC4H> at scbus7 target 0 lun 0 (pass5,ada3)
<ST3000DM001-9YN166 CC4H> at scbus8 target 0 lun 0 (pass6,ada4)
[jgreco@storageX] /# dd if=/dev/ada1 of=/dev/null bs=1048576
^C3853+0 records in
3853+0 records out
4040163328 bytes transferred in 24.200801 secs (166943371 bytes/sec)
[jgreco@storageX] /# fg %1
dd if=/dev/ada1 of=/dev/null bs=1048576
^C14278+0 records in
14278+0 records out
14971568128 bytes transferred in 84.456721 secs (177269114 bytes/sec)
[jgreco@storageX] /# fg %2
dd if=/dev/ada2 of=/dev/null bs=1048576
^C14917+0 records in
14917+0 records out
15641608192 bytes transferred in 87.686877 secs (178380264 bytes/sec)
[jgreco@storageX] /# fg %3
dd if=/dev/ada3 of=/dev/null bs=1048576
^C15829+0 records in
15829+0 records out
16597909504 bytes transferred in 91.159528 secs (182075422 bytes/sec)
[jgreco@storageX] /# fg %4
dd if=/dev/ada4 of=/dev/null bs=1048576
^C16728+0 records in
16728+0 records out
17540579328 bytes transferred in 94.785611 secs (185055296 bytes/sec)
[jgreco@storageX] /#
I've only let each one run a minute or two because I have other things to do today, but you should ideally let them play out and note the speeds, which should all be very similar if you have the same model of disk. Then you do a write test, using "if=/dev/zero of=/dev/ada${foo}" as appropriate. I'm not doing that because I don't have a spare system up and running at the moment.
People are always anxious to skip this step because they think it's just fine and dandy to do it on top of a filesystem layered on top of the disks, but this step can identify poorly performing disk devices (possible future fail!) or bad cabling or troubled controller ports or PCI bus bottlenecks or any number of other things that are WICKED HARD to debug when all you know is that your two-dozen-disk filesystem is performing pretty poorly and you don't know why.
Note that testing one drive individually and then all of them in parallel are not the same test. The speeds you get from one disk and from all disks at the same time should be very similar, or else you've got bottlenecks. Important test.
Then you can create your proposed filesystem and repeat the single file dd test.
Code:
[jgreco@storageX] /mnt/storageX# dd if=/dev/zero of=testfile bs=1048576 count=65536
65536+0 records in
65536+0 records out
68719476736 bytes transferred in 613.280034 secs (112052363 bytes/sec)
[jgreco@storageX] /mnt/storageX# dd if=testfile of=/dev/null bs=1048576
65536+0 records in
65536+0 records out
68719476736 bytes transferred in 240.292488 secs (285982626 bytes/sec)
Now unfortunately this shows slower write speeds than FreeNAS would offer by default; the side effect of the responsiveness tweaking indicated by bug 1531 is lower write performance - but you can be doing a scrub, large sequential read, and large sequential write on this server without losing responsiveness on this system, the way it's been tuned.
Once you've gotten to this point and you're happy, THEN - and only then - should you layer iSCSI on top of it all. iSCSI adds another layer of complexity to it all
If you just look at the end result of throwing something together and then it doesn't perform the way you'd like, you kind of go "hrm" and then waffle about playing find-the-issue. Build it in layers and test each layer.
