X10 or x11 for freeNAS newbie

[MikeeHig]

Here is the hardware I'm looking at for my FreeNAS build. I'd like a build that will last for a long time. It would be a home NAS/plex server for dvd/blu ray rips, a large music library, personal files and most importantly photos and home videos. Right now everything is on drives in my gaming computer and backed up on separate external drives. When I run out of room, I buy a new internal and external drive and rinse and repeat. That’s getting old.

Music and movies will be streamed throughout the house. Will use Plex media server to stream video to 3-4 TVs, home theater projecter and numerous phones and tablets. NAS will backup 3 home PCs. Server will be located in basement where it remains in the 60s year round.

Have built lots of computers for myself and others but this is first foray into server hardware and anything remotely close to freeNAS. Have been reading everything on this site for weeks and am ready to pull the trigger. Not sure whether to go with option 1 (tried and true and a favorite for freeNas folks but limited to 32 GB of memory) or newer equipment option 2 that will allow up to 64 GB of memory in future if needed.

Looking to use 6-8 drives initially RaidZ2 but probably have to add another Vdev later.

Parts
Case: Fractal Design Define R5 Black Silent ATX Midtower Computer Case
Newegg - $90 on sale (already have)

Motherboard:
Option 1) SUPERMICRO MBD-X10SL7-F-O
Newegg - $200 (currently on sale)

Option 2) Supermicro MBD-X11SSL-CF (kinda worried about this since not one person has reviewed it or asked a question on newegg or amazon and it’s not listed on pcpartpicker)
Newegg - $259


Processor:

Option 1) E3-1231 V3 3.4MHZ
Newegg - $255
Option 2) E3-1230 V5 3.4 MHZ
Newegg - $275

CPU Cooler: stock

Ram:
Option 1) 2x Crucial 16GB (2 x 8GB) 240-Pin DDR3 SDRAM ECC Unbuffered DDR3L 1600 (PC3L 12800) Server Memory Model CT2KIT102472BD160B(would add 16 GB later)
Newegg - $90*2 = $180

Option 2) 2x Crucial 16GB DDR4-2400 ECC UDIMM VLP (CT8374137)
Direct from Crucial.com - $98*2 = $196 (can’t find it anywhere else)

Couldn’t find any 260 pin SODIMMs at Newegg or Amazon that matched the tested models on supermicro’s website. Noticed some people in forum have used Samsung M391A2K43BB1-CPB 16GB DDR4-2133 ECC Un-Buffer LP Server Memory which Samsung’s website says is 288 pin DDR4. You already thought I was super NOOB, now I’m going to prove it by asking how both 260 pinn and 288 pin could be used in same slot???

Boot Drive: 2 x Sandisk Cruzer Fit 16 GB in mirror
$cheap

PSU: EVGA 650 GQ gold (already had but never used)

UPS: APC BE550G Back-UPS 550VA 8-outlet (got on super sale)

Data Drives: 8 x WD Reds 3TB
Newegg - $110 apiece

The difference is only $100 between option 1 and 2 but wanted to make sure not any issues with the skylake xeon cpus, X11SSL-CF and my lack of knowledge in memory area. Only real problem I saw with skylake and freeNAS was the USB 3 boot problems but I think I read that had been fixed.

The case (by adding another drive cage) and mainboard provide lots of future expansion for storage, ram, processing power, etc. To start I'll run 6-8 x 3TB raidz2. I can then go with another 6-8 drive array later.

Any issues with the thought process, hopefully explanation on the memory question (260 pin/288 pin) and/or opinions on which hardware option? Thanks in advance for any responses.

 

[Ericloewe]

Couldn’t find any 260 pin SODIMMs

Why are you even looking at SODIMMs?

Couldn’t find any 260 pin SODIMMs at Newegg or Amazon that matched the tested models on supermicro’s website. Noticed some people in forum have used Samsung M391A2K43BB1-CPB 16GB DDR4-2133 ECC Un-Buffer LP Server Memory which Samsung’s website says is 288 pin DDR4. You already thought I was super NOOB, now I’m going to prove it by asking how both 260 pinn and 288 pin could be used in same slot???

Forget everything about pin counts. Whatever you're reading has you confused and is absolutely irrelevant.
Is it a DDR4 UDIMM? It should work (and it will definitely fit). For best results, get something from the QVL.

Option 1) SUPERMICRO MBD-X10SL7-F-O
Newegg - $200 (currently on sale)

Option 2) Supermicro MBD-X11SSL-CF (kinda worried about this since not one person has reviewed it or asked a question on newegg or amazon and it’s not listed on pcpartpicker)
Newegg - $259

Here's a quick summary:

• X11 supports up to 64GB of RAM, vs. 32GB.
• LSI SAS 3008 on the X11SSL-CF is less mature than the SAS 2308 on the X10SL7-F, but otherwise equivalent (and theoretically faster).
• Either platform is supported well these days.

 

[MikeeHig]

Ok, sorry about the memory confusion on my part, but this is what is listed on the supermicro website (https://www.supermicro.com/products/motherboard/Xeon/C236_C232/X11SSL-CF.cfm) for the x11SSL-CF motherboard:

Memory Type: 2133/1866/1600MHz Non-ECC DDR4 SDRAM 72-bit, 260-pin gold-plated SODIMMs

The Samsung memory I listed in the post is on the QVL. I'll run with that. Thanks!

 

[Dice]

Welcome to the forums.
I'll commend you for doing a solid read up on your own.

- 650w PSU is fine for an 8 drive build, but you should consider getting in the 1kw range if you're considering adding a second vdev for a total of 16 drives.
- Consider getting a SSD for system.

In my opinion there is no reason to get a X10 E3 system any longer. X11/Skylake is the way to go.
However, I'm slightly concerned about the expected lifetime of the system granted the information about the amount of streams you'll be looking at doing, in particular if the number of devices increase (and the usecase increases too) with higher bit rates over time.
You might be completely fine if you've 'slightly exaggerated' your requirements of <simultaneous uses> (streaming to friends etc).. If it is the opposite, you'll <maybe> want to look into the next level machine already, to set yourself up for a mere CPU upgrade in the next 4-5 years (when they've cheaped out on e-bay from repurposed datacenter builds).

My take on this third option would be to get a single socket E5 motherboard and a E5-1620v4. RAM limitation is bumped from 64GB to 512GB and you'll have access to the redoncoulsly powerful E5-26xx cpu's.
CPU: Intel Xeon E5-1620v3, 4-Core, 3.5/3.6GHz
MB: X10SRL-F (10x SATA ports onboard)

The price difference between Skylake X11 E3 system and this suggested E5 platform is (locally and foreign..) about 120-140USD.
The longevity and 'upgradeability' of the system is vastly superior to the E3 build. However, as with all good things it comes with a notable drawback. Power consumption I'd guesstimate at the very least a 2-2.5x bump in power consumption, idle.
In the end, it depends on how far you would like your build to reach in the future. And most importantly - which underlies this entire option - how big you want to go with streaming to 'numerous devices', and what you mean by that in strict terms. Some people stream to their set of friends over net..

In conclusion, the E5 route is not for everyone - but it definitely is an under-explored alternative when requirements build up, and projective future scenarios come into play.

 

[CraigD]

Motherboard:
Option 1) SUPERMICRO MBD-X10SL7-F-O
Newegg - $200 (currently on sale)

The case (by adding another drive cage) and mainboard provide lots of future expansion for storage, ram, processing power, etc. To start I'll run 6-8 x 3TB raidz2. I can then go with another 6-8 drive array later.

I went with this motherboard (14 SATA) and a 4 port SATA add-in card (2 port SATA will do for you)

The motherboard is limited to 32GB RAM but I "think" is enough and allows me two vdevs of 8xHDDs RAIDz2 and SSD mirror (8xHDDs still to buy)

Yes it is an older platform that is limited to older 1150 CPUs with a maximum of 32GB RAM however, It is proven to work well and is reliable

 

[Dice]

It is proven to work well and is reliable

So are the X11 systems, with the exception of 'not yet fully tried' 12Gbps SAS.

 

[MikeeHig]

Thanks to everyone for responding. I appreciate you taking the time!

- 650w PSU is fine for an 8 drive build, but you should consider getting in the 1kw range if you're considering adding a second vdev for a total of 16 drives.
- Consider getting a SSD for system.

Thanks, will do. Any suggestions for a small size SSD?

In my opinion there is no reason to get a X10 E3 system any longer. X11/Skylake is the way to go.
However, I'm slightly concerned about the expected lifetime of the system granted the information about the amount of streams you'll be looking at doing, in particular if the number of devices increase (and the usecase increases too) with higher bit rates over time.
You might be completely fine if you've 'slightly exaggerated' your requirements of <simultaneous uses> (streaming to friends etc).. If it is the opposite, you'll <maybe> want to look into the next level machine already, to set yourself up for a mere CPU upgrade in the next 4-5 years (when they've cheaped out on e-bay from repurposed datacenter builds).

Just noticed I did a horrible job of explaining the intended uses very well in regards to simultaneous users. I was referencing ALL uses of the server, not simultaneous uses. Probably would at the most do 2 transcodings at a time, maybe 3 a couple times of the year for visitors.

The price difference between Skylake X11 E3 system and this suggested E5 platform is (locally and foreign..) about 120-140USD.
The longevity and 'upgradeability' of the system is vastly superior to the E3 build. However, as with all good things it comes with a notable drawback. Power consumption I'd guesstimate at the very least a 2-2.5x bump in power consumption, idle.
In the end, it depends on how far you would like your build to reach in the future. And most importantly - which underlies this entire option - how big you want to go with streaming to 'numerous devices', and what you mean by that in strict terms. Some people stream to their set of friends over net..

Thanks for the well thought out 3rd option. At this point looking at either option 2 or 3 but may downgrade processor in 2nd option to i3 6100/6320 range for savings of ~ $150. The i3 6320 has a passmark of 6,075 enough for 3 transcodings. An i3 may serve my uses fine, but there's always that question, "What might I want to do in the future that the i3 won't allow me to do?"

I did have 2 additional questions:

1) I know everyone leaves their servers on 24/7 and maybe freeNAS is designed for it, but what are the downsides to turning it on when needed? It seems like a lot of wear and tear for something I may access a couple of times a week.

2) I thought I had pools, vdevs, etc figured out but had a few questions on the matter. I understand a pool is one or more vdevs which consist of one or more drives. When I first starting reading the forums, it seemed that for instance I had a 6 drives vdev in raidz2 that if I needed to add drives later it was better to add another 6 drive vdev in raidz2 or increase the size of the vdev1 drives one at time. I really didn't catch on that you could have another pool with different characteristics until recent reading. Could I add another pool with say 4 drives in the vdev? If so, how does this change the dynamic of accessing the information from another computer? Hope that makes sense.

 

[Dice]

"What might I want to do in the future that the i3 won't allow me to do?"

I, if anyone, can relate to the above.
I wanted to get a Xeon, but realized I could by no means justify trippeling the CPU cost for my usage.
In retrospect, I wish I had gone for the G4400 option. I've never, under any circumstance - been able to push beyond 35% CPU use. Typically, I see only loads above 20% during heavy redisposition of my archive. Which should be an odd event.
The weekly average is less than 10% (by looking at graphs) on my i3-6100.
Remember, there is always that option to return and upgrade the CPU.

Your questions:
1) Most consider 'access a few times a week' way enough to motivate a 24/7 operation. My own 'threshold' to justify 24/7 operation with regards to drive health, I'd leave the box on if I'd not touch any of its services more than a period of an hour or two, once every other week. In practice, that would be a remote site backup sync box that would do nothing else than carry duplicates of important files. Anything more frequent than that motivates following the 'do not sleep drives' policy, in my understanding.

2)

Could I add another pool with say 4 drives in the vdev? If so, how does this change the dynamic of accessing the information from another computer?

Yes. The significant drawback is that you'll need to configure shares to reach the other datasets on the additional pool. At a time when I ran double pools, I choose to have all 'FreeNAS goodies' on my 'reliable pool' and a 'flush/temp pool' of a mishmash of drives configured suboptimally, only accessed via a simple CIFS share.

In my opinion, a beautiful aspect of ZFS/FreeNAS is the way shares do not need to be reconfigured once additional space has been acquired and added to the main pool. For large media archives/plex services, I'd recommend focusing efforts on a single pool, with matching vdevs. Selecting the width of the vdev is a function many aspects, among the least 'newbie friendly' is to plan ahead in terms of expansion.

Choosing a width of 8-10 drives (really pushing recomendations and good practice) RAIDz2, grants greater space utilization efficiency intially, but considerable less appealing upgrade costs down the line...
Choosing a width of 6 drives RAIDZ2 is popular. At first glance it doesn't look like a lot of space efficiency but when considering the next upgrade will demand the procurement of 6 drives (and potentially a HBA), it suddenly is less interesting to have opted for a super wide 8-10drive RAIDz2....

It comes down to the most difficult aspect of such a huge investment, - aiming high enough to get value.
@cyberjock puts it really neatly in the newbie guide:
"4. Be careful what you settle for. Often when you settle for less than you deserve you get even less than you expected."
Let that advice sink in, for what it is worth. It is probably the best one-liner to summarize the plunge into FreeNAS.

 

[BigDave]

I, if anyone, can relate to the above.
I wanted to get a Xeon, but realized I could by no means justify trippeling the CPU cost for my usage.
In retrospect, I wish I had gone for the G4400 option. I've never, under any circumstance - been able to push beyond 35% CPU use. Typically, I see only loads above 20% during heavy redisposition of my archive. Which should be an odd event.
The weekly average is less than 10% (by looking at graphs) on my i3-6100.
Remember, there is always that option to return and upgrade the CPU.

I made this mistake with my first X9 motherboard and bought the Xeon.
The only good thing about the E3-1200 V2 cpus, is that they have (by some miracle) retained
quite a bit of their original value and hopefully I can recover some of that initial investment.

 

[Ericloewe]

I made this mistake with my first X9 motherboard and bought the Xeon.
The only good thing about the E3-1200 V2 cpus, is that they have (by some miracle) retained
quite a bit of their original value and hopefully I can recover some of that initial investment.

Well, that and the whole thing with nobody being quite sure if the Sandy/Ivy Bridge i3s support ECC.

 

[joeschmuck]

My Two Cents...

For your RAM, check out Superbiz on the internet. They sell bulk RAM typically at a better price than NewEgg. And 16GB RAM is more than enough to do basic things and run Plex. I wouldn't recommend 32GB or more of RAM unless you are going to be running a lot of jails or VMs on FreeNAS. If you can purchase two 8GB sticks or even one 16GB stick (depending on the MB you purchase), you will leave yourself open for future expansion.

Any suggestions for a small size SSD?

Any single cheap SSD will do, size doesn't matter this time. These are more reliable than the USB Flash Drives.