Supermicro X10SRH-CLN4F Server Performance Tests - Weird Results?

Status
Not open for further replies.

depasseg

FreeNAS Replicant
Joined
Sep 16, 2014
Messages
2,874
And is Command Queuing enabled listed for all your DA devices?

Any chance you are using disk encryption?
 

HeloJunkie

Patron
Joined
Oct 15, 2014
Messages
300
No disk encryption on any of the disks I have tested with to date. I thought NCQ was pretty much forced on by the drive manufacturers. I know when I take the drives out of the box and test them while not connected to the SAS Expander / Backplane they work just fine, no drop in tps. When I plug that same drive back into the LSI expander, the problem comes right back!
 

HeloJunkie

Patron
Joined
Oct 15, 2014
Messages
300
At this point, I'd contact SuperMicro. I don't think I applied any updates that would have affected the expander. But I guess that could be a difference. Mine is at 4.1 (0401)

Out of curiosity, what is the output of dmesg | grep ses0

Supermicro has been contacted via my vendor. I'll run the command tomorrow when I get back in front of it tomorrow.

Thank you very much for the help and insight - it helps a great deal to have someone with the exact same system to bounce tests between!
 

souporman

Explorer
Joined
Feb 3, 2015
Messages
57
I'm using the same board (but in the 5048 chassis), with WD Red 6TB drives. dd if=/dev/da0 of=/dev/null bs=8M on da14 gives me ~175MB/s. Running a second instance gives me ~140MB/s, and adding a third instance gives me ~130MB/sec.
 

HeloJunkie

Patron
Joined
Oct 15, 2014
Messages
300
Thanks @souporman - I appreciate the input, gives me more to go to SM with on this issue!
 

souporman

Explorer
Joined
Feb 3, 2015
Messages
57
btw, I've also flashed the FW down to v5-IT. Oh, and I have like 6(with more every day) 5048 chassis that exhibit the same behavior as what I wrote above.
 

HeloJunkie

Patron
Joined
Oct 15, 2014
Messages
300
Supermicro is shipping out a new backplane today!
 

vikingboy

Explorer
Joined
Aug 3, 2014
Messages
71
Im glad you shared this information, benchmarking FreeNAS gets some tough press on here but I think your scenario shows why benching is a useful tool in verifying components meet expectations. Hope the new backplane sorts this for you.
 

HeloJunkie

Patron
Joined
Oct 15, 2014
Messages
300
@depasseg

Here is the output from mine:

Code:
[root@plexnas] ~# dmesg |grep ses0
ses0 at mpr0 bus 0 scbus0 target 20 lun 0
ses0: <LSI SAS3x28 0601> Fixed Enclosure Services SCSI-5 device 
ses0: Serial Number        
ses0: 1200.000MB/s transfers
ses0: Command Queueing enabled
ses0: SCSI-3 ENC Device
ses0: da0,pass0: Element descriptor: 'Slot00'
ses0: da0,pass0: SAS Device Slot Element: 1 Phys at Slot 0
ses0:  phy 0: SATA device
ses0:  phy 0: parent 5003048001cc6ebf addr 5003048001cc6e80
ses0: da1,pass1: Element descriptor: 'Slot01'
ses0: da1,pass1: SAS Device Slot Element: 1 Phys at Slot 1
ses0:  phy 0: SATA device
ses0:  phy 0: parent 5003048001cc6ebf addr 5003048001cc6e81
ses0: da2,pass2: Element descriptor: 'Slot02'
ses0: da2,pass2: SAS Device Slot Element: 1 Phys at Slot 2
ses0:  phy 0: SATA device
ses0:  phy 0: parent 5003048001cc6ebf addr 5003048001cc6e82
ses0: da3,pass3: Element descriptor: 'Slot03'
ses0: da3,pass3: SAS Device Slot Element: 1 Phys at Slot 3
ses0:  phy 0: SATA device
ses0:  phy 0: parent 5003048001cc6ebf addr 5003048001cc6e83
ses0: da4,pass4: Element descriptor: 'Slot04'
ses0: da4,pass4: SAS Device Slot Element: 1 Phys at Slot 4
ses0:  phy 0: SATA device
ses0:  phy 0: parent 5003048001cc6ebf addr 5003048001cc6e84
ses0: da5,pass5: Element descriptor: 'Slot05'
ses0: da5,pass5: SAS Device Slot Element: 1 Phys at Slot 5
ses0:  phy 0: SATA device
ses0:  phy 0: parent 5003048001cc6ebf addr 5003048001cc6e85
ses0: da6,pass6: Element descriptor: 'Slot06'
ses0: da6,pass6: SAS Device Slot Element: 1 Phys at Slot 6
ses0:  phy 0: SATA device
ses0:  phy 0: parent 5003048001cc6ebf addr 5003048001cc6e86
ses0: da7,pass7: Element descriptor: 'Slot07'
ses0: da7,pass7: SAS Device Slot Element: 1 Phys at Slot 7
ses0:  phy 0: SATA device
ses0:  phy 0: parent 5003048001cc6ebf addr 5003048001cc6e87
ses0: da8,pass8: Element descriptor: 'Slot08'
ses0: da8,pass8: SAS Device Slot Element: 1 Phys at Slot 8
ses0:  phy 0: SATA device
ses0:  phy 0: parent 5003048001cc6ebf addr 5003048001cc6e88
ses0: da9,pass9: Element descriptor: 'Slot09'
ses0: da9,pass9: SAS Device Slot Element: 1 Phys at Slot 9
ses0:  phy 0: SATA device
ses0:  phy 0: parent 5003048001cc6ebf addr 5003048001cc6e89
ses0: da10,pass10: Element descriptor: 'Slot10'
ses0: da10,pass10: SAS Device Slot Element: 1 Phys at Slot 10
ses0:  phy 0: SATA device
ses0:  phy 0: parent 5003048001cc6ebf addr 5003048001cc6e8a
ses0: da11,pass11: Element descriptor: 'Slot11'
ses0: da11,pass11: SAS Device Slot Element: 1 Phys at Slot 11
ses0:  phy 0: SATA device
ses0:  phy 0: parent 5003048001cc6ebf addr 5003048001cc6e8b


From what I can see, it all looks good!
 

depasseg

FreeNAS Replicant
Joined
Sep 16, 2014
Messages
2,874
It does. In fact yours looks better than mine. My da10 and da11 devices aren't named and yours are. hmmmmm
upload_2015-4-18_21-22-40.png
 

souporman

Explorer
Joined
Feb 3, 2015
Messages
57
Since you guys have SAS3008 on board... have either of you gotten SAS3IRCU to work in FreeNAS? I can get the UEFI version to work, but when I run it in FreeNAS it tells me there are no controllers found.
 

depasseg

FreeNAS Replicant
Joined
Sep 16, 2014
Messages
2,874
I don't have sas3ircu. Running sas3flash provides a "no LSI adapters found message".
 

leonroy

Explorer
Joined
Jun 15, 2012
Messages
77
Supermicro have pretty good support, glad they shipped out a new BP. As an aside I know testing can be fun and educational, but why not pickup a TrueNAS box and avoid all this system integration rigmarole? Is the pricing that much better when you roll your own?
 

HeloJunkie

Patron
Joined
Oct 15, 2014
Messages
300
Since you guys have SAS3008 on board... have either of you gotten SAS3IRCU to work in FreeNAS? I can get the UEFI version to work, but when I run it in FreeNAS it tells me there are no controllers found.

Same here...no adaptor found when I run it. Maybe because the controller has been flashed to IT mode...?
 

HeloJunkie

Patron
Joined
Oct 15, 2014
Messages
300
Supermicro have pretty good support, glad they shipped out a new BP. As an aside I know testing can be fun and educational, but why not pickup a TrueNAS box and avoid all this system integration rigmarole? Is the pricing that much better when you roll your own?

My goal with "rolling my own" if far more than simply saving a few dollars on the cost of the hardware. You do save quite a bit when you buy a supermicro box and do it yourself, at least in my case. I have gotten quotes both for me personally and for the school where I work and frankly, being on a tight budget at the school (and not so tight on people time) doing it ourselves was a lot less expensive. With almost 100TB deployed (soon to be 140TB), the cost difference was almost ten thousand dollars and climbing.

However, one could have argued the cost of the time, energy and testing on my end might have been worth the cost to outsource it to IX, however we still have to support it ourselves, inhouse. The testing process is a very intimate way to understand not only the hardware, but the freenas OS itself. If one is not afraid to get their hands dirty and learn the hardware and the software, test every conceivable configuration, learn a huge amount about freenas in the process (as I have done with each new server deployment), I think we are far more able to support the solution once deployed.

We run LTSP, Proxmox, OpenStack, and Linux. With almost 95% of our 1,500 network clients running Linux in some form or another, I feel my team is well equipped to handle the bsd/FreeNas stuff, but like anything else you're going to hang your hat on, I think you better know as much about it as possible before using it in production, and what better way than doing it yourself? We have a long way to go in completely understanding FreeNas, but one thing I can say is that I know it far more than I ever knew my $2.8M NetApp that I purchased when I owned data centers!
 

HeloJunkie

Patron
Joined
Oct 15, 2014
Messages
300
It does. In fact yours looks better than mine. My da10 and da11 devices aren't named and yours are. hmmmmm

I should get the new backplane today, it will be interesting to see how it does and what it looks like in comparison to the one I have now.
 

souporman

Explorer
Joined
Feb 3, 2015
Messages
57
Supermicro have pretty good support, glad they shipped out a new BP. As an aside I know testing can be fun and educational, but why not pickup a TrueNAS box and avoid all this system integration rigmarole? Is the pricing that much better when you roll your own?
I have both. I paid ~$70k for my TrueNAS and $15k for my FreeNAS. I also got 13 more TB useable and used 4 less u in my rackspace.

Are they exactly the same? No. TrueNAS uses SAS 7k disks whereas I'm using SATA. Both saturate a 10GbE and both have kicked 2 disks since I've owned them (7 months). I'm actually building out 8u worth of Supermicro these days and getting 300TB useable for it in RAIDz3.

The TrueNAS has been great, though, and it's been a pleasure to work with iX support. It's a heck of a lot cheaper than the cool million we dropped on our EMC Isilon cluster, which has been an absolute nightmare to administer.
 

HeloJunkie

Patron
Joined
Oct 15, 2014
Messages
300
I was going to go for a shiny iX unit just to have one, but then I look at the Neptune and it looks exactly like the Supermicro server I have now :smile: I too have spent millions on solutions and I am sooooo happy Freenas is around!

That is one of the tests that I have not done yet is with a 10G link. My new box has a 10G Intel card in it designed to go back-to-back with my Plex server (not that it needs it, but just cause I can). Once I get it all set up I will test between the two. RIght now I can saturate a 1G link no sweat.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
My goal with "rolling my own" if far more than simply saving a few dollars on the cost of the hardware. You do save quite a bit when you buy a supermicro box and do it yourself, at least in my case. I have gotten quotes both for me personally and for the school where I work and frankly, being on a tight budget at the school (and not so tight on people time) doing it ourselves was a lot less expensive. With almost 100TB deployed (soon to be 140TB), the cost difference was almost ten thousand dollars and climbing.

About 16 months ago I did data recovery for a school. The reason? They wanted to save that money by going with a home-brew setup rather than an iX setup. The entire school district lost their largest pool, backups were later determined to be borked beyond being useful, and they reconsidered that money saved by not going iX. They run TrueNAS now. ;)

The sad truth. It was actually their choice of hardware that did them in. Not the software.
 

HeloJunkie

Patron
Joined
Oct 15, 2014
Messages
300
Yea, we tend to stick with Supermicro for that exact reason!
 
Status
Not open for further replies.
Top