So you want some hardware suggestions.

[dark7np]

I wonder what you guys think about this Ebay offer:
http://www.ebay.de/itm/130734807487
Sorry it's German, but I think the technical stuff should be more or less clear...

Intel S5400SF Board, 32 GB Ram (DDR2 FB), with 2 Quad-Core Xeons (X5335), for 300€. It has 6 SATA Ports. One thing I noticed is that is has only one x16 PCIe slot for expansion. I was quite fond of it at first, until I thought about adding Controllers to it... Would probably be more suited as Board for a virtualization host. For this price though? If it would have more slots? What do you think?

What also bugs me is the Intel homepage for this Board. It is insanely difficult to find the technical specs of this board - as in a listing of the IO options. It seems to have custom add addon-Cards for Ethernet / Infiniband and others. Is that common for Server Boards? How hard is it to get such cards?

As far as Cases go: http://www.ebay.de/itm/161217099507 Supermicro 933T - 15 3.5" Hot-Swap SATA for 150€. Anything wrong with that? I guess a used case and backplane should not be much trouble?

 

[jgreco]

Just from the pix/without looking it up, the board strikes me as a 1U server board, probably custom design. Stay away unless you prove me wrong to your satisfaction (not necessarily hard to do btw).

Probably designed for HPC is my guess. Which could mean cheap parts and upgrades if you don't need best-of-the-best; the HPC crew changes gear like a normal person changes underwear.... But this could also mean it all becomes nightmare unobtanium in a year.

As for the Supermicro, be sure to know what exactly the backplane is all about.

 

[dark7np]

A 1U PCIe riser card was mentioned somewhere. It did kinda suggest HPC to me too, with Dual-Sockets, the Ram Slots, Infiniband and the lack of other slots.

What about the backplane? There are hotplug HDD trays and 15 normal SATA looking connectors on the inside. I have to admit, I know that in principle SATA can have expanders and all that, but I've never had anything to do with them. I don't know what a backplane is all about. My understanding so far is that it's just "cables" in board form. At least that seems to be what this one is. Are there ones with built-in expanders? Do expanders need to be supported by the HBA? Any Pointers to background info in that domain?

The price of the Supermicro is quite charming, IMHO. A new one costs almost 1k € I think... And 15 drive bays should easily be enough for a while in my case.

 

[HoneyBadger]

Unfortunately you've got an SSI-EEB motherboard and an EATX case - while the two are the same dimensions (12"x13") they have different mounting holes.

That system will also be a power-hungry SOB as well with 32GB spread across up to 16 FB-DIMMs. Not sure if that factors into your decision much (especially if you're planning on tucking a dozen or more drives in there) but thought I'd mention it.

 

[jgreco]

As for the chassis, these fall into a few categories depending on specific backplane features.

I don't know which one that unit has offhand. 4G wireless is crap this morning.

A Supermicro "TQ" style terminates every drive at a separate SAS (possibly two) connector, which for your purposes can be used as a SATA connector. DIY'ers should have complete familiarity with that. Downsides are needing a controller channel per port; for example, a SC846 TQ will require you to have 24 SATA/SAS channels, or three M1015's worth of controllers. Plus all the ugly SATA cabling.

A Supermicro "A" style is, IIRC, basically the same thing but brings it out to SFF8087 connectors. Same downsides except less ugly cabling. An SC846 "A" will need three M1015's and six SFF8087 cables.

A Supermicro "BE" style has an SAS Expander. An SC846BE16 will have a single SFF8087 for the host connection and will only require half of a single M1015 to attach. This sounds like a great deal - and it is - but only for spinny hard drives. Because you have a 24Gbps SAS link and 24 drives, all 24 drives are sharing that link, limiting each drive to about ~1Gbps. Contemporary drives are able to do a bit more than that. On the other hand, your system is unlikely to be pushing THAT hard at all drives simultaneously. Which is why I think it is a pretty good choice for up to maybe 24 hard drives.

These chassis are designed for use in a server rack, machine room, or data center and may be ... a bit noisy. Just be aware. The chassis fans can be "fixed" if that's a problem but the supply fans can't, really.

 

[jgreco]

Oh and PS you can usually cram a SATA drive into a SAS bay, but the other way around does not work!

 

[EvanVanVan]

This is a similar comparison to my first post, so I'm probably won't find anyone with either of these PSUs. But I bought a Corsair HX650 Gold from newegg for $99 after rebates (with thousands of good reviews, http://www.newegg.com/Product/Product.aspx?Item=N82E16817139012 ). Now I see that Newegg is selling a Rosewill FORTRESS-650 Platinum series for $79 after rebates (with only 100 good reviews, http://www.newegg.com/Product/Product.aspx?Item=N82E16817182083 ).

Will I be better off in the long run with the Platinum PSU over the Gold one? and should I pull the trigger and return my (unopened Corsair PSU)?

 

[jgreco]

Your RoI in terms of energy saved probably won't recover the return shipping cost for at least a year or two.

 

[EvanVanVan]

Well, thanks you're probably right. Although, while making this post I did just realize I actually paid $109 before the rebate was in effect. So I'm going to call Newegg tomorrow and see if they'll refund me the $10...

Thanks again

 

[pirateghost]

This is a similar comparison to my first post, so I'm probably won't find anyone with either of these PSUs. But I bought a Corsair HX650 Gold from newegg for $99 after rebates (with thousands of good reviews, http://www.newegg.com/Product/Product.aspx?Item=N82E16817139012 ). Now I see that Newegg is selling a Rosewill FORTRESS-650 Platinum series for $79 after rebates (with only 100 good reviews, http://www.newegg.com/Product/Product.aspx?Item=N82E16817182083 ).

Will I be better off in the long run with the Platinum PSU over the Gold one? and should I pull the trigger and return my (unopened Corsair PSU)?

I tend to stay far away from rosewill except for small unimportant items.

Sent from my Nexus 5

 

[jgreco]

Hi to all,

I am considering building my own NAS.
The main function I want, is for it to be able to stream media (1080p) at a good rate.
What motherboard and CPU would you recommand.

Here is a first version:
Motherboard: ASUS H81I-PLUS
Processor: Pentium G3220

Please note that I don't want to build a server grade NAS.

Thank you,
John

Then just pick any random crap you like and have a go at it. "Server grade" is a term that describes an x86/x64 platform designed for your use model. It is typically a bit cheaper than prosumer boards and will include stuff like Intel ethernets and proper ECC support. I think there's a sticky at the top of this forum called "So you want some hardware suggestions" that was written for people just like you. Read it, follow it, be happy. Do your own thing at your own risk.

 

[diehard]

jgreco, have you had any bad experiences with SAS expanders with SATA drives? There seems to be so much mixed info out there regarding their use with ZFS.. it's hard to form a consensus. I still think this fits into the "hardware recommendations" category.

 

[HoneyBadger]

The best I can find from the SAS expanders with SATA drives is "it's not recommended." I did a lot of searching myself and the best I can find is below.

Due to the lack of WWN on SATA drives there's not a 100% reliable way to make sure you're finding the disk you expect to. On this forum though I've seen someone have weird hangs on boot with SATA SSDs attached to a SAS expander.

Personally I try to avoid SAS expanders even if I'm not using SATA just because it's one more layer between ZFS and the disks. And I'm sure that if I had an issue, the first thing someone would ask me to test is "can you connect it directly and see if that resolves it?"

 

[jgreco]

jgreco, have you had any bad experiences with SAS expanders with SATA drives? There seems to be so much mixed info out there regarding their use with ZFS.. it's hard to form a consensus. I still think this fits into the "hardware recommendations" category.

Great question.

I do have a dozen SATA drives hanging off an SAS expander (Supermicro 846BE26 backplane) and it is one of the least expensive options for attaching large numbers of drives.

But yes, there are real people with real SATA drives that have caused SAS expanders to do Bad Things(tm), most of which appear to be in the realm of failures causing other drives to also "lock up" or "error out in nasty ways." This seems to be related to specific combinations of drive and controller firmware. So I have two suggestions to mitigate:

1) Be very aggressive with drive replacement. Do not allow failing SATA drives to remain in an array.
2) Do not use such a pool for a mission-critical application.

One alternative is to avoid the SAS expander by using direct channels to an HBA. A typical ServeRAID M1015 (or other LSI -8i HBA) has 8 ports for 8 drives. You can easily put three in a system to handle 24 drives and as a bonus you get no SAS 24Gbps link contention. Nice.

The other alternative is to move to SAS nearline drives, which gives you the ability to multipath as well. The Constellation ES.3 drives are probably the least expensive/best specced SAS drives out there. The ST4000NM0023 offers a 128MB cache and 7200RPM spindle speed, 5 year warranty, and a 512 native sector size (kinda watch out for that) which runs around 175MB/sec. The downside is that they're bleepin' expensive (~$350) compared to SATA (~$150).

So, there is a significant financial incentive to stay with SATA if possible. SATA with an SAS expander is probably cheapest per-TB assuming you have more drives than fingers. SATA with SAS HBA is probably okay up to maybe 24-32 drives and omits the expander drama, but is of course adding some per-drive cost on for the HBA ports.

But if you're really terrified of SAS expander lockups etc., please carefully consider your use case. Here it is archival storage so "no big" if there were to be a problem requiring a pry-the-drive-out/reboot event - and there's never been a problem that required that.

 

[diehard]

Good info.

I built a 36 drive system with SATA drives on a 846EL1 (847E16 chassis) and a LSI 9207-8i and have been running mostly backup stuff and a couple VM's off it but he more i have been reading about their "toxic relationship" the more i have been getting nervous. Even though its just backup data.. if it were to go down it would take down some mission critical VM's (Windows and ESXi just love when you pull drives unexpectedly)

So far it has been good on mostly older ES.2 drives. If we were to go with another system do you think it would be wise to go with a Supermicro JBOD backplane and just use SAS > SATA breakout cables? Only downside would be it wouldnt as clean, right? The allure of paying $150 for a drive is so great...

Edit: Now that i think about it, the breakout cables would present the same problem that a backplane would present.. damn.

 

[jgreco]

I assume you mean an arrangement like an -8i HBA to SFF8087<--->SAS breakout cables that attach to the individual drive connectors on a TQ backplane? That is what I meant by my first alternative above.

(the term "JBOD" isn't sufficiently clear here because the SuperMicro JBOD expander chassis are actually SAS expander based, see BPN-SAS2-846EL2 vs the individually connectorized BPN-SAS-846TQ, so referring to individual lanes as "JBOD" is just confusing.)

It isn't the backplane that's a problem - that's just a circuit board. For TQ it just passes signals through and distributes power (more or less). It's the SAS expander chip on the fancier backplanes like the EL2 that is of concern.

The question is, how paranoid do you want to get. If you do not have a combination of drives/expander/controller that is prone to lockups, do you want to proactively mitigate a nonexistent problem?

Also, we're talking SATA ES.2's here? I'm going to go out on a bit of a limb here and say I'd expect they're more likely to be fine on an SAS expander. You might want to find out what the story is on configuring ERC for those units, most of the lockups I've heard of are somehow related to failing drives. But at this point you're just dealing with my hunches, which you have no particular reason to trust.

 

[diehard]

Yeah i was meaning the Supermicro backpane's with individual SATA/SAS connections, sorry. I updated my post with a little more specific info on what i was using.

They are the SATA ES.2's.. and they have been going good for about 6 months now. The only "problem" is after management saw how cheap storage could be .. wants to put more of it into my hands. Trying to move away from compellent and their $1000 1TB drives.

I might have convinced them i could replicate the system and get new 3TB WD Red's (or alternatively other cheap 5400rpm SATA drives) and make another box. I really should have just kept my mouth shut lol. But im rambling.

Thanks for the help/advice , much appreciated.

 

[panz]

I don't like Expander backplanes: I had that choice when I decided to buy one case (24 drives with 1 backplane every 4 drives = 6 Mini-SAS SFF-8087 cables) or the other one (again 24 drives, all driven with a single Mini-SAS cable + Expander backplane).

If you are in trouble it is very difficult to find a replacement part for that expander backplane.

 

[Z300M]

The best I can find from the SAS expanders with SATA drives is "it's not recommended." I did a lot of searching myself and the best I can find is below.

Due to the lack of WWN on SATA drives there's not a 100% reliable way to make sure you're finding the disk you expect to. On this forum though I've seen someone have weird hangs on boot with SATA SSDs attached to a SAS expander.

Personally I try to avoid SAS expanders even if I'm not using SATA just because it's one more layer between ZFS and the disks. And I'm sure that if I had an issue, the first thing someone would ask me to test is "can you connect it directly and see if that resolves it?"

I don't see myself using an SAS expander any time soon, if ever, but I'm curious about your suggestion that it could be difficult to find the correct disk. If

zpool status

tells me that a drive shows errors and

glabel status

tells me the (a)daN information for that drive

and smartctl -a /dev/(a)daN

tells me the drive model and serial number and

all my drives are marked with a readily visible serial number,*

where is the problem in locating the correct drive?

*My drives are mounted in iStarUSA drive cages with the serial numbers marked clearly on the front of each hot-swap adapter.

 

[jgreco]

panz, your complaint about expander backplanes is kind of odd. You're basically in trouble with big complex hardware if ANY of it fails. Could be an SAS expander chip, yes, but even avoiding that, if you have a 24 drive chassis, what do you do if you snap off an SAS/SATA connector on the backplane? If you have an HBA with 8 ports, what do you do if that fails on your 12 drive pool? etc.

It is probably wise to give the issue some thought but it probably doesn't warrant totally avoiding a given bit of hardware.