This question below is primarily for anyone out there with an MD1000 with an m1015, but any feedback/thoughts are greatly appreciated.
My configuration:
FreeNAS-9.10.2-U1 (USB)
Dell R610 (latest BIOS) 12GB RAM E5620/2.4 Ghz
m1015 cross flashed to latest P20 IT mode (20.00.07.00)
Dell MD1000 A.04 (unified mode) populated with 14x1GB Hitachi HUA72101 A74A (SATA + interposers)
MD1000 connected via single SAS cable from m1015 to EMM#1
I've bench-marked the system (compression off, using dd if=/dev/zero of=/mnt/vol1/dataset1/ddfile bs=2048k count=10000 and dd of=/dev/null if=/mnt/vol1/dataset1/ddfile bs=2048k count=10000). Generalized results:
1. For reference, a single disk, read/write ~80MB/s
2. Any other configuration read/write <200MB/S (Mirror, 6 disk RAIDZ2, 4xMirror, 2xRAIDZ2-7 disk) with individual drive performance well below 80MB/s, CPU load negligible
My thoughts/observations:
1. Based on the (admittedly) little I know, I would have thought the dd results with a 2xRAIDZ2 (7 disk) would be somewhere around 500-600 MB/s for this hardware before tapping out the SAS 6Gbps/4 lane interface.
2. Seeing as how the individual drive performance drops off roughly proportionately to the number of drives in a volume tested, I suspect there is an issue between the m1015 and the MD1000 resulting in only a single SAS lane being used.
3. I'm not necessarily trying to eke out every last drop of performance. Rather, my concern is that there is an underlying hardware or configuration problem. This system is for home use.
4. It looks to me like the m1015 has LEDs to indicate the SAS lane activity (marked CR2-CR9 on the board). I am seeing only a single LED lit when under load - CR5 or CR9 depending on which SAS port is connected to the MD1000.
5. I've tried swapping SAS cables, pulling an EMM, and changing EMM's with no change noted in behavior.
6. SMART tests look ok for every drive.
7. When in the m1015 BIOS, I can't expand the MD1000 enclosure to view individual drives and the signaling summary info is blank. Not sure if this is an issue or to be expected. FreeNAS has no issue seeing any drives.
Question 1 (for anyone with m1015 and MD1000): What level of performance are you seeing with your system using dd? Are my expectations way off?
Question 2 (for anyone with m1015): If you physically look at your m1015, is there more than one LED lit while the system is under load?
Question 3: Is there anyway to determine the number of SAS lanes active on a SAS HBA from within FreeNAS/FreeBSD?
Question 4: Should a single process (i.e. dd) activate/use all available SAS lanes? Or are lanes place into use as additional processes request/write data (similar to LACP)?
Question 5: Am I doing something wrong or are my expectations on performance or thought processes way off?
I am obviously new to FreeNas, so any help or direction is greatly appreciated. BTW, there is no data on this system yet, so if it would be helpful for me to provide specific dd results using differing drive configurations, I am able to make changes as needed.
Thank you in advance!
Rob
My configuration:
FreeNAS-9.10.2-U1 (USB)
Dell R610 (latest BIOS) 12GB RAM E5620/2.4 Ghz
m1015 cross flashed to latest P20 IT mode (20.00.07.00)
Dell MD1000 A.04 (unified mode) populated with 14x1GB Hitachi HUA72101 A74A (SATA + interposers)
MD1000 connected via single SAS cable from m1015 to EMM#1
I've bench-marked the system (compression off, using dd if=/dev/zero of=/mnt/vol1/dataset1/ddfile bs=2048k count=10000 and dd of=/dev/null if=/mnt/vol1/dataset1/ddfile bs=2048k count=10000). Generalized results:
1. For reference, a single disk, read/write ~80MB/s
2. Any other configuration read/write <200MB/S (Mirror, 6 disk RAIDZ2, 4xMirror, 2xRAIDZ2-7 disk) with individual drive performance well below 80MB/s, CPU load negligible
My thoughts/observations:
1. Based on the (admittedly) little I know, I would have thought the dd results with a 2xRAIDZ2 (7 disk) would be somewhere around 500-600 MB/s for this hardware before tapping out the SAS 6Gbps/4 lane interface.
2. Seeing as how the individual drive performance drops off roughly proportionately to the number of drives in a volume tested, I suspect there is an issue between the m1015 and the MD1000 resulting in only a single SAS lane being used.
3. I'm not necessarily trying to eke out every last drop of performance. Rather, my concern is that there is an underlying hardware or configuration problem. This system is for home use.
4. It looks to me like the m1015 has LEDs to indicate the SAS lane activity (marked CR2-CR9 on the board). I am seeing only a single LED lit when under load - CR5 or CR9 depending on which SAS port is connected to the MD1000.
5. I've tried swapping SAS cables, pulling an EMM, and changing EMM's with no change noted in behavior.
6. SMART tests look ok for every drive.
7. When in the m1015 BIOS, I can't expand the MD1000 enclosure to view individual drives and the signaling summary info is blank. Not sure if this is an issue or to be expected. FreeNAS has no issue seeing any drives.
Question 1 (for anyone with m1015 and MD1000): What level of performance are you seeing with your system using dd? Are my expectations way off?
Question 2 (for anyone with m1015): If you physically look at your m1015, is there more than one LED lit while the system is under load?
Question 3: Is there anyway to determine the number of SAS lanes active on a SAS HBA from within FreeNAS/FreeBSD?
Question 4: Should a single process (i.e. dd) activate/use all available SAS lanes? Or are lanes place into use as additional processes request/write data (similar to LACP)?
Question 5: Am I doing something wrong or are my expectations on performance or thought processes way off?
I am obviously new to FreeNas, so any help or direction is greatly appreciated. BTW, there is no data on this system yet, so if it would be helpful for me to provide specific dd results using differing drive configurations, I am able to make changes as needed.
Thank you in advance!
Rob