BUILD First Build - Am i doing it right?

[Eli Singer]

Being doing a lot of reading hopefully to get something right out of my First FreeNAS build.
The suggested build will have 8 HGST 4TB drives to get me going, and i would like to have the option to add more drives in the future.
I also have an IBM m1015 handy if i ever need it.

So here it is, any suggestions, recommendations and comments will be much appriciated. Thank you!
You can see the build here: http://pcpartpicker.com/user/eli_singer/saved/vtmnTW
And here's a rundown here also:

Intel Xeon E3-1220 V5 3.0GHz Quad-Core Processor
Scythe Kotetsu 79.0 CFM CPU Cooler
Supermicro MBD-X11SSH-LN4F Micro ATX LGA1151 Motherboard
Kingston ValueRAM 32GB (2 x 16GB) DDR4-2133 Memory
Fractal Design Define R4 w/Window (Black Pearl) ATX Mid Tower Case
SeaSonic 520W 80+ Platinum Certified Fully-Modular Fanless ATX Power Supply

Just to note- quiet computing is important to me hence the choices for the fanless PSU, the CPU cooler, and the case.

What say you?

Eli

 

[BigDave]

As long as the Kingston RAM is utilizing the tested chips (Samsung, Hynix, Micron)
and PartsPicker will take it back if it's not compatible, I see no issues there.
I did however go with the tested Samsung RAM for my X11SSM-F board, due to
Kingston's past transgressions with part numbers not being changed while using
different parts/components.

 

[Eli Singer]

Following this lovely post: https://forums.freenas.org/index.php?threads/so-you’ve-decided-to-buy-a-supermicro-x11-board.39549/
I'm thinking of switching to the X11SSl-CF for the SAS ports it has.

One thing i wanna make sure i get right- It says the X11SSl-CF has 8 SAS ports- does that mean i can run 32 SATA drives when connecting 8 SFF8643 cables to the 8 SAS ports?!?
Another thing is that by looking at pictures of the board it doesn't look like it has 8 ports but 4:

what am i missing?...

 

[Nick2253]

You are misunderstanding. A SAS port is not a breakout port. This board contains two SFF-8643 ports, each of which can be broken out to four drives, for a total of 8 drives. In other words, this motherboard can support 8+6=14 drives in total.

Based on this question, I'm assuming you haven't made your complete purchase yet. If it's not too late, I have a few comments about your original build:

• The CPU cooler is unnecessary. Unless you are getting an OEM CPU without a cooler, you should not need to buy one; the OEM cooler is more than sufficient. Even if your concern is quiet cooling, the stock CPU coolers from Intel are extremely quiet. Your money will be better spent buying quality case fans.
  
• Unless you can confirm in person which memory you are getting, I'd avoid Kingston. Samsung or Micron would be my top recommendation.

 

[Ericloewe]

In other words, this motherboard can support 8+6=14 drives in total.

Or many more, using SAS expanders.

But if you have an M1015 lying around, why would you bother paying more for a board with an SAS controller? Just stick with a cheaper X11SSM-F or one of the X11SSH-F variants.

 

[Eli Singer]

Thank you Nick for your reply.
I have an IBM m1015 so i thought that like the two ports on the m1015 can break out to 8 drives so does the listed 8 ports on this board would yield 32 drives...
Didn't realized that the 8 ports listed are actually 2 connectors that can breakout to 8 drives. That places it right there with the m1015 in terms of connectivity.
ALTHOUGH (and maybe i'm wrong here too)- the fact that this is a 12Gb port means i can double the amount of drives with expanders to 16 drives and still maintain 187.5MB transfer rate per drive- is that right?
If I'll go beyond that I'm risking bottlenecking the drives...

The point for getting the X11SSl-CF was for future upgrade- having the ability to have 32 drives was well worth the extra cost for the board. Too bad i was wrong...
But even though, the premium for the X11SSl-CF when compared to the X11SSM-F is well worth it if i can get 8 more drives out of the X11SSl-CF. The difference is much lower than adding another m1015.

I'm not buying just yet, have a few more things to figure out.
I'll avoid Kingston as you suggested. If i'm getting a Samsung or a Micron, is it really crucial that I'll look for the specific chips on the tested list on Supermicro's site? It looks like nitpicking...

 

[maglin]

No it's a SAS3 controller and thus the 12Gbs. Each SAS connector has 4 channels and each channel can connect to one drive. If you went another route you could get a server enclosure and run the drives off the backplane but then it would definitely be louder than you probably want. I think there is a backplane/expander option for the R5? Or maybe I'm just not thinking straight but I thought I remembered people talking about them.

 

[Eli Singer]

So what is the right calculation for this? How fast is each channel?

 

[maglin]

Each channel is faster than any spinning disk. SAS is more about total bandwidth. SAS3 is really only leveraged by either crazy large spinning disk arrays or SSD arrays.

In the end it will always be you NIC that is going to make HDD speed a moot point. Unless you go 10G which is still cost prohibitive. Just know that if you are going to use SAS breakout cables you can only attach 4 drives to each SAS connector. If you have a backplane or expander you can attach some crazy number of drive that most everyone on this board couldn't afford. I'll look up how many HDDs can be connected to a SAS.

Wikipedia says "SAS allows up to 65,535 devices", but I read the whole article and with edge expanders able to connect to 255 devices, and a fanout expander can connect to 255 edge expanders that is 65,025 devices. I'm unsure where the extra 510 devices come from.

Anyways a single SAS cable can run 65,535 drives. Latency might spike a bit over just 4 channels, but it's possible.

 

[Eli Singer]

I'm not asking about how many drives i can attach phisically. The question is how many drives can i put on 12GB divided to 4 channels and hooked up to expanders without bottlenecking the port?
And i do intend to go 10GBe not too far off. Seriously considering the X11SSH-CTF for that reason.

 

[Ericloewe]

I'm not sure you understand what's going on. The board has two ports, whose designation I have not yet memorized, similar in practice to the older SFF-8087 ports typical of SAS2 stuff. Each provides four SAS channels, each at 12Gb/s.

Even the fastest HDDs don't easily saturate a SATA 1.5Gb/s conmnection.

Wikipedia says "SAS allows up to 65,535 devices"

LSI HBAs are limited to somewhat more sane numbers, around 1024 SAS devices total for most.

 

[Nick2253]

I'm not buying just yet, have a few more things to figure out.
I'll avoid Kingston as you suggested. If i'm getting a Samsung or a Micron, is it really crucial that I'll look for the specific chips on the tested list on Supermicro's site? It looks like nitpicking...

It is nitpicking, and it isn't.

Basically, when SuperMicro tests memory, they put it through a rigorous set of batteries, and confirm that there aren't any bugs that might cause weird problems. In theory, since everyone is following "standards", this shouldn't be a problem. But there is wiggle room in each standard, and each manufacturer does things slightly differently, so you can get weird errors.

Usually, if one of a manufacturer's memory chips is found compatible, then you won't have problems with any of their chips. It's only when it's completely untested that you might have problems. And don't think the problems will always be obvious: it could be something weird, like partially functioning ECC, which you won't notice until it's too late.

The other benefit is that, when you use tested memory modules, SuperMicro will be happy to support whatever bugs you run in to. But if you aren't they may just point to the untested memory and say "yup, there's your problem", even if it isn't.

It sucks, and it sounds stupid, but at the end of the day, when it comes to server applications, like data storage, reliability trumps cost. This is why we recommend using the tested memory, and more specifically tend to be really pedantic when it comes to hardware choices.

 

[Eli Singer]

I'm not sure you understand what's going on. The board has two ports, whose designation I have not yet memorized, similar in practice to the older SFF-8087 ports typical of SAS2 stuff. Each provides four SAS channels, each at 12Gb/s.

Even the fastest HDDs don't easily saturate a SATA 1.5Gb/s conmnection.


LSI HBAs are limited to somewhat more sane numbers, around 1024 SAS devices total for most.

I think i'm still missing something...
If each CHANNEL is running at 12Gb, that means i can connect 10 drives per channel by using a 12 drives expander and still get 150MB/s from each drive?
That means 40 drives per 1 port that splits to 4 channels?
That means i can run 80 drives at their full speed with the 2 ports on the X11SSl-CF and expanders?!
That sounds insane...

 

[Ericloewe]

If each CHANNEL is running at 12Gb, that means i can connect 10 drives per channel by using a 12 drives expander and still get 150MB/s from each drive?

Theoretically, yes. In practice, expanders take four upstream ports and provide 20 or 32 downstream ports. You can chain expanders, though.

 

[Eli Singer]

Theoretically, yes. In practice, expanders take four upstream ports and provide 20 or 32 downstream ports. You can chain expanders, though.

So one expander will take the four upstream channels from one port with the SFF8643 cable and break it to at least 10 drives with no problem?
Will the intel RES2SV240 do the trick?
Is there other recommended expanders that might fit better to my scenario? If i understand correctly, i rather have each of the 4 channels going to an expander with 2-3 ports so i can connect 8-12 drives per channel, so is there a low coast solution for that?

BTW, me being a noob at this and all, i can't seem to locate from the pics on the web where does the upstream channel cables connect to the RES2SV240... How is that done exactly?

Thank you very much for you help!

Eli

 

[Ericloewe]

So one expander will take the four upstream channels from one port with the SFF8643 cable and break it to at least 10 drives with no problem?

Yes.

Will the intel RES2SV240 do the trick?

Theoretically, yes. In practice, people have had trouble mixing SAS3 controllers with SAS2 expanders. If you go ahead with that, you'll definitely want to ensure everything is using the latest firmware versions.
Also, SAS2 stuff uses SFF-8087, so you need cables that convert that to the new connector.

Is there other recommended expanders that might fit better to my scenario? If i understand correctly, i rather have each of the 4 channels going to an expander with 2-3 ports so i can connect 8-12 drives per channel, so is there a low coast solution for that?

All expanders are going to be functionally equivalent, with ports aggregated into groups of four.

BTW, me being a noob at this and all, i can't seem to locate from the pics on the web where does the upstream channel cables connect to the RES2SV240... How is that done exactly?

All ports are equal. So, wherever you like. You have 24 ports, four of them are going to be upstream.

 

[Eli Singer]

Yes.

Theoretically, yes. In practice, people have had trouble mixing SAS3 controllers with SAS2 expanders. If you go ahead with that, you'll definitely want to ensure everything is using the latest firmware versions.
Also, SAS2 stuff uses SFF-8087, so you need cables that convert that to the new connector.

All expanders are going to be functionally equivalent, with ports aggregated into groups of four.

All ports are equal. So, wherever you like. You have 24 ports, four of them are going to be upstream.

Let's see- So if I'll go with the RES2SV240 I'd need an sff-8643 to sff-8087 cable like this one: http://www.newegg.com/Product/Product.aspx?Item=N82E16816118210
It will go from the X11SSl-CF to the expander, from there a regular sff-8087 breakout cable will go to the drives like this one: http://www.bhphotovideo.com/c/produ...Lvy0s0CFYidGwod4UAMqQ&c3api=2572,113041717267

Is that true?

Is there a cheaper option than the $250 RES2SV240 for this? maybe a SAS 3 expander with less ports?... I don't need 6 ports on the expander, 4 or even 3 would do it...

 

[Ericloewe]

Let's see- So if I'll go with the RES2SV240 I'd need an sff-8643 to sff-8087 cable like this one: http://www.newegg.com/Product/Product.aspx?Item=N82E16816118210
It will go from the X11SSl-CF to the expander, from there a regular sff-8087 breakout cable will go to the drives like this one: http://www.bhphotovideo.com/c/produ...Lvy0s0CFYidGwod4UAMqQ&c3api=2572,113041717267

Right.

Is there a cheaper option than the $250 RES2SV240 for this? maybe a SAS 3 expander with less ports?... I don't need 6 ports on the expander, 4 or even 3 would do it...

I once saw 5 of the SAS3 model going for 300 bucks. Don't know if such a deal can be found these days.

As for fewer ports, no. LSI only makes 24 and 36 port expander chips, last I checked.

 

[Eli Singer]

Got it. Good info.
one more if i may, can i take the ports on the m1015 and connect them to a RES2SV240 and use it the same way?
Performance wise i could only get 5 drives per channel and not 10 to avoid bottleneck, but that still adds up to 40 drives for the m1015 having 2 ports, each splitting to 4 channels, each channel splits to 5 drives. Just wanna make sure I'm on top of this...

 

[Ericloewe]

one more if i may, can i take the ports on the m1015 and connect them to a RES2SV240 and use it the same way?

Of course. This is preferred, since it's known to work well.

Performance wise i could only get 5 drives per channel and not 10 to avoid bottleneck,

That's assuming a rather worst-case scenario that does not make sense to consider in most cases. I wouldn't mind stretching that a bit further, even. There's plenty of bandwidth to go around and I'd have no qualms with running 24 drives (a full 4U backplane) off four SAS channels. 36 wouldn't be crazy, either.