New 32 4TB HDD Server

[gpsguy]

Code tags preserve the formatting of your message.

Before you insert your chart, etc. you insert a "code tag". After the chart you add another one.

In my example below, I reversed the square brackets, so you could see what is needed. To make it work substitute [ for ] and vice versa.

]code[xxxxx]/code[


Sent from my phone

 

[Mr_N]

Who told you that? You're supposed to keep the driver and firmware at the same version.

Was told as long as you keep the firmware of the hba ahead of the driver version in freenas its all good, but there might be problems if its behind, obviously if you want to keep it the same thats going to work too.

That's 4Gb/s per lane theoretical which for estimation sake take 75% of that for a decent gestimate. That card has 8 lanes of PCIex2.

So he could throw a lot more drives at that card w/o issue. I think he'd be stoked to get 6Gb/s sustained for the over all array.

You sure? I thought the entire x8 pci 2.0 was 4GByte/s (500MByte/s per lane) ... so a x16 would be 8GByte/s total... then moving upto PCI 3.0 you get 1GByte/s per lane...

 

[aufalien]

Yes but the card has 8 lanes of PCIe 2.0. So what I meant by PCIeX2 was PCIe 2.0.

 

[Mr_N]

I agree its probably not the cause of the performance, but with 36 hdd's he's limiting each drive down to around 113MB/s if loading it up with sequential reads/writes, suggesting you can throw plenty more drives at it is only going to reduce that number further...

 

[Rand]

My questions is I am getting a lot of error messages about the LSI 9211-8i "IOC fault, resetting" and the maximum speed i can get for write is 55MB/s.

I would think that continuously resetting of the BUS on the single HBA *will* have a serious impact on performance.
I suppose you are using a port multiplier to get those x8 card to 36 interfaces? Is there a chance to test it without (ie only 8 drives)?

Not to start a whole new discussion but SATA drives behind SAS port multipliers have been known not to always work completely as expected... (Edit: have been said to not always work completely as expected...)

 

[Ericloewe]

I would think that continuously resetting of the BUS on the single HBA *will* have a serious impact on performance.
I suppose you are using a port multiplier to get those x8 card to 36 interfaces? Is there a chance to test it without (ie only 8 drives)?

Not to start a whole new discussion but SATA drives behind SAS port multipliers have been known not to always work completely as expected...

Can you elaborate? General consensus seems to be that SAS expanders work fine with SATA drives. The SAS spec was certainly designed for that.

 

[Rand]

eg
http://garrett.damore.org/2010/08/why-sas-sata-is-not-such-great-idea.html
http://hardforum.com/showthread.php?t=1548145

Not sure it applies to FreeBSD in general and FreeNas in particular but back when i started looking at that stuff it was a hotly discussed issue.
O/c i started out with nexenta which was directly affected.
Might or might not be a reason to worry.

 

[Ericloewe]

The first link sounds more like fear mongering... All the talk of "conversion" is crap. SAS includes a protocol for encapsulating ATA in an SAS link, so that the all the SAS devices can speak SCSI to each other while carrying the ATA commands between drive and OS.

What is possible is that certain vendors have shady implementations that don't work like they should. LSI stuff is known to work well, though. Cyberjock, for instance, has an LSI 2008 controller and LSI 24-port expander and he's happy with the setup.

 

[cyberjock]

Putting SATA on SAS isn't a bad idea so long as you don't go cheap with things like SAS expanders and such.

All too often people buy a SAS expander that is the cheapest they can find in the world and get upset when it's not 'quite right'. As someone who has put SATA on SAS for 6 years running and never had a problem I can vouch that it *can* and *does* work properly when cost isn't your primary factor. If cost is your primary factor then may God have mercy on your data.

For the record I've been running my M1015 with an Intel SAS Expander for more than a year without a problem.

 

[Rand]

Ok, glad to hear that. Maybe i was too paranoid ;)

 

[Bigtexun]

"All too often people buy a SAS expander that is the cheapest they can find in the world" This is an interesting statement, given that a cheap $75 controller is recommended, and the only recommendations in the official hardware posts state simply "a compatibleSAS expander".

You can't give people a hard time for buying the "cheapest they can find in the world" if there are no specific statements as to what they should buy, or even what they should look for. There is precious little information on the SAS expanders, and most of us can't afford to do what the professional integrators do... They buy 3 or 4 of everything, and put a team of engineers together to work through all of the options, to determine which one they should buy for a production line.

So how about some specific part numbers... When I do searches, I can't find any SAS expanders that say "not the cheapest in the world", so that doesn't seem to be sufficient information. And price isn't a good measure of value anyway, I see the identical parts with prices all over the place.

I mean I understand your point about not being cheap... But I see you and others harp and harp about using recommended hardware, but then you don't complete the recommendations and then complain about people being cheap.

I am very happy you said you are running with an Intel expander... but we need more info, and it would be nice to have info on more than one option. SAS expanders are not commodity items, and many models are difficult to find new, and they are often for a specific chassis.

I've already been burned buying things that other people got working with old versions of FreeNAS... only to find out those systems stopped working after upgrades. So I'm ready to sip the kool aid, where is it?

And before someone gives me a lecture on just using bare controllers... There are ZERO high density controller recommendations. To do high density, I have been forced to do non-recommended configurations. Not everyone is building a small one filesystem baby NAS. I'm not looking for performance, I have 4 SAN's for performance, one has 100 drives on it. I'm building NAS servers for slow bulk storage, and FreeNAS and similar solutions excel at good performance that exceeds my needs. I'm archiving code trees for a manufacturer with an ever growing set of hardware versions of an imbedded product, that gets forked every time we get a new customer. It is just an archive of old stuff we don't need to access except for iso 9000 reasons. FreeBSD was always the OS of choice for this sort of archiving, starting with the beta that first got ZFS. Between ZFS and filesystem snapshots, FreeBSD was instantly the best and most efficient solution back then. I'm trying to make the jump to FreeNAS. I've made the jump at home, now I'm trying to do it at the office. I have my drive backplanes, a stack of the recommended LSI 9211-8i's, now I just need the expanders.

 

[diehard]

I honestly can't think of any SAS expanders that are not LSI based.

For a 32 drive server i wouldnt really recommend any enclosure other than a Supermicro one.. and getting the E26 chassis will give you redundant expanders.

 

[aufalien]

Well, I've had luck with any of the Intel ones as well as CI Designs. The CI Designs are of the type that integrate into your chassis.

 

[Ericloewe]

Well, I've had luck with any of the Intel ones as well as CI Designs. The CI Designs are of the type that integrate into your chassis.

All of Intel's SAS stuff uses LSI controllers/expanders, so it should work well.

 

[TXAG26]

I'll echo the Intel LSI-based SAS expanders. I've been using the 24-port version for about a month with no issues with a reverse break-out cable to hook up to the 8x onboard LSI 2308 SATA ports. I'm not seeing any performance loss either (compared to when each drive was directly hooked up to the LSI 2308. The 24-port expanders actually only have 20 actual SAS ports. 4 ports are used to connect to HBA.

Remember, the biggest thing is getting the break-out cables right (if your setup needs them)! There are two types: "Forward" and "Reverse" and they both look the same, but are not interchangeable!

 

[jgreco]

behind SAS port multipliers have been known not to always work

Please, terminology, people. There's no such thing as a "SAS port multiplier". There are SATA port multipliers and they typically suck in the worst way because they're made by some no-label company and sold in a Shenzhen back alley to manufacturers of cheap PC expansion hardware. Then there are SAS expanders, most of which are made by LSI these days, which are manufactured to attach large fleets of disks in enterprise and datacenter environments. These are not guaranteed to be 100%, but if you take the time to qualify the hardware appropriately, these will be as close to rock solid as you can get given that you are introducing a third device and a protocol conversion into what is normally a two-device (controller, drive) equation.

 

[diehard]

holy balls i thought jgreco died

 

[cyberjock]

holy balls i thought jgreco died

Naa, I didn't give him permission to die.

 

[Bigtexun]

For what it's worth, under Free BSD 8.0 beta, with the beta version of ZFS, the crappy cheap SATA port multipliers in my numerous massive drive chassis worked flawlessly. The only problems came after upgrading the OS to something more current. The guy that replaced me at that job pretty much lost every array he upgraded... I don't know what he upgraded to, I just know he will strangle me to death if he ever sees me.

I have not tracked every version of FreeBSD, but I do know that the raw drive performance has varied between versions, sometimes getting worse, and sometimes getting better. It could be as simple as code changes that improved performance that triggered problems with the multipliers. The expanders are very similar devices, but they operate on the SAS level, and encapsulate the SATA packets, so it is possible that the SAS layer is the reason they work better. Clearly the people making the port multipliers needed to give a pile of the for free to the FreeBSD developers... But they work great in linux or window$, so there you go...

I begrudgingly have been using Linux for most of my performance filesystems applications, since FreeBSD lost it's performance edge back in the mid to late 1990's... With version 5.x there were several years where the kernel guys were stuck in the mud, and filesystem I/O went to the crapper and remained there for a few years. 6.x was supposed to fix everything, but it was only 80% of the performance of Linux... And when it finally recovered, the years of momentum Linux had gained pretty much wiped out most commercial application support for FreeBSD, and a LOT of the open-source support. My search engine datacenter had to migrate to linux during that time for performance reasons, and once you do something like that, there is no going back...

But I guess the point is don't blame everything on the end user buying something "cheap". FreeBSD really lost it's market when they got stuck in the mud... And the big data users are not happy about hardware failing because of a software update. But I see a lot of talk here that blames everything on the hardware, hardware that is /solid/ under other operating systems. I'm a huge fan of *BSD. I used to buy licenses directly from Rob Kolstad when he ran BSDi, back in the day. But is is a struggle to justify the work, only specific features like ZFS provide a clear enough win... But this idea that you can't buy the same hardware that works great everywhere else gets really old really fast.

I'm chugging the kool aid as fast as I can, and I have been drinking it for 20 years, but this forum is oozing with the kind of attitude that you can only get away with when you are beating up a n00b. I get it, not everything is supported. But don't blame it on being cheap, or crappy. Admit the reality, FreeBSD doesn't have working drivers for everything. Don't say "card X has some problems" Say the truth, "we don't have working drivers for card x". Some of the rhetoric I see here is really bringing back bad memories of why 5.x was a disaster, and 6.x was too untrusted for anyone to go back. Sounds more like politics or religion than computer engineering.

I'm just shocked...Shocked I tell you... That an advanced NAS project like this is stuck behind a single 8 port controller recommendation.

And if you wonder why I rant... I'm an engineer. Statements like "I honestly can't think of any SAS expanders that are not LSI based." appear to be true... so statements like "All too often people buy a SAS expander that is the cheapest they can find in the world" only conveys emotion, and very little fact. If that had been from any end-user I would be entirely understanding... that is how I feel about SuperMicro... I understand feelings. But this is a support forum, and those words came from someone I'm supports to trust. I think your statement is valid, and you made it for all the right reasons. But as an official representative of the company, I expect a little less emotion and a little more fact from the official messaging.

But that's OK... Rob Kolstad was also very emotional... I worked with him on an anti-spam appliance, and his design actually increased the spam volume of the beta testers by about 100 fold... He created a user honey pot where all emails addresses worked... yeah, he got really excited as all of his beta servers melted under the load. The good ol' days...

 

[jgreco]

holy balls i thought jgreco died

Naa, I didn't give him permission to die.

"Oh, he's not dead... not yet."

Just been working a contract gig that's been eating all my time. May be seeing more of me soon.

Made my day though, guys. ;-)