Supermicro X10SRH-CLN4F Server Performance Tests - Weird Results?

[depasseg]

I have an m1015 I can try.

My 3008 is flashed to v5 which matches the driver version.

 

[HeloJunkie]

@depasseg - that would be great as I am seeing the same problem with both my M1015 (which works great in another supermicro non-backplane system) as well as the onboard 3008. I did order a new cable today that should be here tomorrow (FSS8643 to normal sata) and expect to test the 3008 directly to the hard drives. Frankly I am not sure what to expect since the 1015 is doing the exact same thing with the same drivers I am using in other systems. But the 14.04 LTS thing is throwing me for a loop as well - the fact that it appears to run ok under Linux but not bsd.

Eventually we will get this figured out and move on!

 

[Ericloewe]

The mpr SAS3 driver is due for a major update soon, so I'd expect fixes for this kind of issues.

 

[depasseg]

@HeloJunkie - Does the M1015 use the same connector as the one on the MB? Or did you need a new cable or an adapter? I won't be able to look at it until next weekend, so I figured if I needed to order anything, I should do it now.

Out of curiousity, which 3008 FW did you use? I originally used phase 6 and then downgraded to phase 5.

This is what I did:
https://forums.freenas.org/index.ph...nas-primary-for-plex.24519/page-6#post-159232

 

[Ericloewe]

The M1015 needs an SFF-8087 to SFF-WahteverTheNewOneIsCalled cable if you'll be using it with an SAS3 backplane.

 

[HeloJunkie]

@depasseg

I had to purchase the SFF-8087 to SFF-8642 cable from Newegg to get the M1015 to work with the backplane. I am expecting this cable today to try a test of the 3008 directly to the drives.

On the firmware, I also started with 6 but went down to 5 at your suggestion:

I have that same exact server. Only difference is CPU, RAM and I'm using SAS drives instead of SATA.

I also downgraded my 3008 FW to match the Freenas driver version (5).

Code:

[root@plexnas] ~# dmesg | grep mpr
mpr0: <LSI SAS3008> port 0xe000-0xe0ff mem 0xfb200000-0xfb20ffff irq 26 at device 0.0 on pci1
mpr0: IOCFacts :
mpr0: Firmware: 05.00.00.00, Driver: 05.255.05.00-fbsd

 

[TXAG26]

@depasseg

I had to purchase the SFF-8087 to SFF-8642 cable from Newegg to get the M1015 to work with the backplane. I am expecting this cable today to try a test of the 3008 directly to the drives.

On the firmware, I also started with 6 but went down to 5 at your suggestion:

Code:

[root@plexnas] ~# dmesg | grep mpr
mpr0: <LSI SAS3008> port 0xe000-0xe0ff mem 0xfb200000-0xfb20ffff irq 26 at device 0.0 on pci1
mpr0: IOCFacts :
mpr0: Firmware: 05.00.00.00, Driver: 05.255.05.00-fbsd

Did you ever have a chance to try hooking the 3008 directly to the SATA drives with the new cable? I'm looking to pull the trigger on the same board, but initially, will not need the LSI expander. I will just use SFF-8642 to SATA cables until I add more drives and pick up another expander.

 

[HeloJunkie]

Did you ever have a chance to try hooking the 3008 directly to the SATA drives with the new cable? I'm looking to pull the trigger on the same board, but initially, will not need the LSI expander. I will just use SFF-8642 to SATA cables until I add more drives and pick up another expander.

Yes I did actually, I ran into the same problem unless I switch from FN to Linux. I tried loading FreebSD, I tried loading NAS4Free, both had the very same issue, but when I load Ubuntu or Gentoo, the problem goes away completely. In any case, since there are several other people running that same board without any issues, I decided to go ahead and put test data on the box and test performance even though it looked like there may be an issue. Long story short, I can completely saturate 2Gbps of a 10G link (tested with iperf to 6+Gbps) on reads and as high as 5Gbps (tested with iperf to 9.41Gbps) on writes. Now, mind you this was with nfs and a single 12 Drive vdev RAIDZ2. I have since moved to 2 x 6 Drive VDEVS RAIDZ2. I suspect I will see better performance, but I have not tested it as of yet.

You can read about this performance testing and my attempt to speed up NFS HERE.

So in the end, even with a perceived performance issue when doing DDs directly to the /dev/da device, with 2 vdevs in RAIDZ2 with sata drives, this motherboard absolutely rocks. If you were not going to run 10G, then this board will completely saturate a gig link each direction all day long.

 

[TXAG26]

Excellent news! Since the real world performance seems great, I wonder if it's some strange testing error that only shows up during specific benchmarking environments? My environment is still just 1GB, so I'm not expecting speed issues anytime soon.

Do you know what revision of the X10SRH-CLN4F you have? I've seen photos online showing Rev. 1.00 and Rev. 1.00a.

 

[HeloJunkie]

I have the 1.00a revision.

 

[TXAG26]

Thanks for the info, I placed an order for the X10SRH-CLN4F this evening. Can't wait to get this new system setup. Hopefully the next couple of FreeNAS v9.3 updates bring some more stability to the LSI 3008 drivers.

 

[HeloJunkie]

As a further test I have moved 11TB of Movies/Music/TV Shows to this box, then moved it off, then back on again with no more problems all via NFS....

 

[Ericloewe]

Thanks for the info, I placed an order for the X10SRH-CLN4F this evening. Can't wait to get this new system setup. Hopefully the next couple of FreeNAS v9.3 updates bring some more stability to the LSI 3008 drivers.

There seems to be a new firmware/driver combination on the horizon, which is said to iron out most of the big issues. No ETA that I know of, though, and it'll need a bit before showing up on FreeNAS after that.

 

[TXAG26]

There seems to be a new firmware/driver combination on the horizon, which is said to iron out most of the big issues. No ETA that I know of, though, and it'll need a bit before showing up on FreeNAS after that.

That is welcomed news. Is there a site or feed that I can watch for news on updated drivers/firmware (besides the forums here)?

 

[Ericloewe]

That is welcomed news. Is there a site or feed that I can watch for news on updated drivers/firmware (besides the forums here)?

You can try checking the bug tracker for mpr stuff. It's mostly about information that trickles down from the devs, who are in contact with the LSI software guys.

 

[depasseg]

I know it's been a little while, but I have an interesting data point. I just picked up an SC847 enclosure (SAS2). And hooked it up to the external port of my freenas1 server (LSI 3008). Experienced the same exact slowdown occurances on the SC847 drives. The first one was screaming at ~150MB, then as additional tests (I was running badblocks) started, the speeds slowed until the stabilized around 35MB. This is running badblocks -ws /dev/da14_through_39. What's a little interesting is that the last 6 drives are 6TB, so I had to change the block size to 4096. These 6 drives are running at 75MB. All drives are SATA (not SAS like the others I mentioned earlier in this thread).

 

[pclausen]

A few pages back the question about checking / upgrading the firmware on the backplane came up. Unless things have changed compared to the Supermicro SAS2 backplanes, you can easily check the firmware version and upgrade to the latest.

Here's a screenshot of me doing it to one of my 846 expanders:

 

[depasseg]

Too bad that looks like windows. :-(

 

[pclausen]

Ah, but they do have linux flavor as well.

ftp://ftp.supermicro.com/utility/ExpanderXtools_Lite/Linux/64bits/

 

[Cpuroast]

I thought I'd contribute to this thread since I'm currently testing pretty much identical hardware and so far both serial and parallel performance is looking good.

SuperStorage Server 5048R-E1CR36L
Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz
64GB DDR4-2133
LSI SAS3008
24x SEAGATE ST4000NM0023 0004

dev.mpr.0.%desc: LSI SAS3008
dev.mpr.0.%driver: mpr
dev.mpr.0.%location: slot=0 function=0 handle=\_SB_.PCI0.BR1A.H000
dev.mpr.0.%pnpinfo: vendor=0x1000 device=0x0097 subvendor=0x15d9 subdevice=0x0808 class=0x010700
dev.mpr.0.%parent: pci1
dev.mpr.0.debug_level: 3
dev.mpr.0.disable_msix: 0
dev.mpr.0.disable_msi: 0
dev.mpr.0.firmware_version: 05.00.00.00
dev.mpr.0.driver_version: 05.255.05.00-fbsd
dev.mpr.0.io_cmds_active: 24
dev.mpr.0.io_cmds_highwater: 440
dev.mpr.0.chain_free: 2048
dev.mpr.0.chain_free_lowwater: 2048
dev.mpr.0.max_chains: 2048
dev.mpr.0.chain_alloc_fail: 0

ses0 at mpr0 bus 0 scbus0 target 32 lun 0
ses0: <LSI SAS3x40 0601> Fixed Enclosure Services SCSI-5 device
ses0: Serial Number
ses0: 1200.000MB/s transfers
ses0: Command Queueing enabled
ses0: SCSI-3 ENC Device

ses1 at mpr0 bus 0 scbus0 target 63 lun 0
ses1: <LSI SAS3x28 0601> Fixed Enclosure Services SCSI-5 device
ses1: Serial Number
ses1: 1200.000MB/s transfers
ses1: Command Queueing enabled
ses1: SCSI-3 ENC Device

Performing initial serial array read (baseline speeds)
Fri Jul 3 06:41:34 PDT 2015
Fri Jul 3 07:35:35 PDT 2015
Completed: initial serial array read (baseline speeds)

Array's average speed is 178.657 MB/sec per disk

Disk Disk Size MB/sec %ofAvg
------- ---------- ------ ------
da0 3815447MB 173 97
da1 3815447MB 183 102
da2 3815447MB 190 106
da3 3815447MB 167 94
da4 3815447MB 185 104
da5 3815447MB 171 96
da6 3815447MB 174 97
da7 3815447MB 172 96
da8 3815447MB 185 104
da9 3815447MB 185 104
da10 3815447MB 174 98
da11 3815447MB 181 101
da12 3815447MB 176 99
da13 3815447MB 184 103
da14 3815447MB 183 102
da15 3815447MB 181 102
da16 3815447MB 172 96
da17 3815447MB 183 102
da18 3815447MB 180 101
da19 3815447MB 171 96
da20 3815447MB 182 102
da21 3815447MB 173 97
da22 3815447MB 176 99
da23 3815447MB 185 104

Performing initial parallel array read
Fri Jul 3 07:35:35 PDT 2015
The disk da0 appears to be 3815447 MB.
Disk is reading at about 173 MB/sec
This suggests that this pass may take around 367 minutes

Serial Parall % of
Disk Disk Size MB/sec MB/sec Serial
------- ---------- ------ ------ ------
da0 3815447MB 173 173 100
da1 3815447MB 183 182 100
da2 3815447MB 190 190 100
da3 3815447MB 167 167 100
da4 3815447MB 185 185 100
da5 3815447MB 171 171 100
da6 3815447MB 174 174 100
da7 3815447MB 172 172 100
da8 3815447MB 185 185 100
da9 3815447MB 185 185 100
da10 3815447MB 174 176 101
da11 3815447MB 181 181 100
da12 3815447MB 176 176 100
da13 3815447MB 184 184 100
da14 3815447MB 183 183 100
da15 3815447MB 181 181 100
da16 3815447MB 172 172 100
da17 3815447MB 183 183 100
da18 3815447MB 180 180 100
da19 3815447MB 171 172 100
da20 3815447MB 182 181 99
da21 3815447MB 173 173 99
da22 3815447MB 176 172 98
da23 3815447MB 185 180 97

Checking the Disk I/O graphs in the FreeNAS WebUI also shows that parallel reads are all normal, all 24 disks are doing ~170-175MB/sec.