24-48+ HDD Server - Some questions

[AlexIsNas]

Hey guys,

I’m planning on building my first FreeNAS sever. So I would like to have some thoughts about my planned configuration.


First of all I would like to start with what I’ve got:

Drobo B800FS (Raid 6)
- 6 * 4TB WD Red HDD’s
- 2 * 3TB Seagate Baraccuda HDD’s

Drobo 5N (Raid 5)
- 5 * 3TB Seagate Baraccuda HDD’s

1 Cloud Server
- 16GB RAM
- i7-6700T
- 2 * 400GB old Sata drives (Seagate) (Raid 1)

many PC, tablets and laptops which would like to have access to my data.


Both Drobo’s store personal files, movies, photos, backups, programs (iso’s and exe), documents, etc.


What I would like to do:
Replace the 2 Drobo’s with one FreeNAS server. It should be capable for 3-4 1080p streams in my home network, maybe some 4K streams in the near future and while doing so still be able to access my personal data. If possible I would like to have a share for my cloud aswell. So that my cloud server is having it’s data stored on the FreeNAS.

For the FreeNAS server I would like to go with HDD arrays of 8 drives in ZFS 2. No spare drives...
One big pool with x * 8-drive-arrays and all shares like movies, tv-shows, picture, data, etc. are stored on the big pool.

So in the beginning I have to buy 1 more 3TB drive to complete the 8 drives 3TB array.
And 2 4TB drives to complete the 8 drives 4TB array.

This would give me this configuration:
8*4TB - ZFS 2
Space = (8-2)*4TB*0,931
Space = 22,344TB

8*3TB - ZFS 2
Space = (8-2)*3TB*0,931
Space = 16,758TB

Total = 39,102TB


I read that FreeNAS needs 1GB Ram for 1TB of Storage.

One question here: do I need 39GB RAM, 42GB RAM (without the 0,931 factor) or 56GB RAM (8x[3TB+4TB]).

The next future upgrade would be the investment of the next HDD-array with 8*4TB WD Red
Which would give me a total of 61,446TB.

In the end I would like to be able to add another 24 HDD’s (Total 48+), that the reason for my following hardware configuration.

Hardware:

Supermicro X10SRL-F
8 ECC RAM slots
7 PCIe slots
10 SATA ports
http://www.supermicro.nl/products/motherboard/xeon/c600/x10srl-f.cfm

Intel Xeon E5-1620v4
LGA2011-3
4x 3.5GHz / 8 Threads
140W TDP / max. DDR4 PC2400
https://ark.intel.com/products/92991/Intel-Xeon-Processor-E5-1620-v4-10M-Cache-3_50-GHz

32GB DDR4 PC2133 ECC Registered CL15 Samsung
Can I start with only one RAM stick installed, or do I need to have at least two installed?
Modul number: M393A4K40BB0-CPB
Listed on the super micro website :D
http://www.supermicro.nl/support/re...9&prid=84050&type=DDR4 1.2V&ecc=1&reg=1&fbd=0

Inter-Tech 4U 4129-N Case
https://www.inter-tech.de/en/products/ipc/server-cases/4u-4129-n

Inter-Tech Rails
https://www.inter-tech.de/en/products/ipc/19-accessories/2u-rail-650mm

24 HDD’s
Western Digital and Seagate

PSU
Seasonic X-850
https://seasonic.com/product/x-850/

24 HDD’s:
W_peak = 24*1,75A*12V = 504W
W_idle = 24*3,3W = 79,2W
W_R/W = 24*4,5W = 108W

W_motherboard = 50W (I haven’t found some numbers)
W_ram = 8x5W = 40W (Same here…)
W_Xeon = 140W (TDP as „guess number“)
W_HBA = 2*10W = 20W

W_peak= 504W + 50W + 40W + 140W + 20W = 754W

For the 48 HDD configuration I would simply install another Seasonic X-850W

HBA:
IBM ServeRaid M1015 - is this a fake one? How can I detect a fake one?
http://www.ebay.de/itm/IBM-ServeRai...e7df3f9&pid=100005&rk=1&rkt=6&sd=201890415718

Intel RES2SV240 24 Port - can I also go with one like this? I know you guys recommend the M1015 but I read that some people use the 24 Port for bigger configurations.
http://www.ebay.de/itm/Intel-RES2SV...957827?hash=item3ae0a42883:g:w5cAAOSwZQRYfzaw

Questions:
So what do you guys think about my configuration?
Will it handle my planned load?
How big is the reverse? (circa...)
What amount of RAM do I need? (see "full" question in the text above)
Can I start with only one RAM stick? (again in the text above^^)
Would you still recommend the M1015?
Will it work with my planned "one big pool and the x * 8-drive-arrays"-system?
I don't need spare drives, do I?

Thank you for your help,
Alex

 

[Ericloewe]

ZFS 2

You must mean RAIDZ2. I really don't know where people get this "ZFS 2" name from...

This would give me this configuration:
8*4TB - ZFS 2
Space = (8-2)*4TB*0,931
Space = 22,344TB

https://forums.freenas.org/index.php?threads/zfs-raid-size-and-reliability-calculator.28191/

You seem to be seriously overestimating available space. The difference between TB and pseudo-TB is something like 0.909, not 0.931.
@Bidule0hm's calculator will give you a very good estimate.

Inter-Tech 4U 4129-N Case
https://www.inter-tech.de/en/products/ipc/server-cases/4u-4129-n

Inter-Tech Rails
https://www.inter-tech.de/en/products/ipc/19-accessories/2u-rail-650mm

You should consider something like a used Supermicro chassis, the experience will probably be much better.

HBA:
IBM ServeRaid M1015 - is this a fake one? How can I detect a fake one?
http://www.ebay.de/itm/IBM-ServeRai...e7df3f9&pid=100005&rk=1&rkt=6&sd=201890415718

Comes from China == probably fake.

Intel RES2SV240 24 Port - can I also go with one like this? I know you guys recommend the M1015 but I read that some people use the 24 Port for bigger configurations.
http://www.ebay.de/itm/Intel-RES2SV...957827?hash=item3ae0a42883:g:w5cAAOSwZQRYfzaw

That's not an HBA, it's an expander. You'd probably want one, but if you go with the Supermicro chassis, you can get a backplane that already has the expander.

Will it handle my planned load?

If you want to transcode 4K, the Xeon E5-1650 v4 might be a better choice.

How big is the reverse? (circa...)

What do you mean?

What amount of RAM do I need? (see "full" question in the text above)

I read that FreeNAS needs 1GB Ram for 1TB of Storage.

One question here: do I need 39GB RAM, 42GB RAM (without the 0,931 factor) or 56GB RAM (8x[3TB+4TB]).

The next future upgrade would be the investment of the next HDD-array with 8*4TB WD Red
Which would give me a total of 61,446TB.

It's a purposely-vague rule of thumb. More RAM is better.

Would you still recommend the M1015?

It's one of many options.

Will it work with my planned "one big pool and the x * 8-drive-arrays"-system?

That's the whole point of ZFS.

I don't need spare drives, do I?

You'd better have a bare minimum of one drive burned in and ready to go for when (not if - when) a drive fails. At your scale, probably two.

 

[joeschmuck]

This would give me this configuration:
8*4TB - ZFS 2
Space = (8-2)*4TB*0,931
Space = 22,344TB

8*3TB - ZFS 2
Space = (8-2)*3TB*0,931
Space = 16,758TB

Total = 39,102TB

Your calculations are a bit optimistic but close enough for general planning. Do not forget about maintaining 20% free space for optimum pool performance so ensure you cut that out of your total capacity.

I read that FreeNAS needs 1GB Ram for 1TB of Storage.

One question here: do I need 39GB RAM, 42GB RAM (without the 0,931 factor) or 56GB RAM (8x[3TB+4TB]).

That is a rule of thumb, mostly for corporate speedy systems. My advice is to start with 32GB, it should be more than fine for the server you are planning.

I don't need spare drives, do I?

Not really, but if you have a failure you might want to shut down your system if you cannot lay your hands on a new drive within a few days. But of course you would maintina a backup of your important data just in case all hell breaks loose.

Can I start with only one RAM stick?

Refer to the user manual for the motherboard and find out. They generally tell you.


EDIT: Someone beat me to the punch line so I'll just post what I got.

 

[c32767a]

You've already got a bunch of responses so I only have a couple thoughts..

Replace the 2 Drobo’s with one FreeNAS server. It should be capable for 3-4 1080p streams in my home network, maybe some 4K streams in the near future and while doing so still be able to access my personal data. If possible I would like to have a share for my cloud aswell. So that my cloud server is having it’s data stored on the FreeNAS.

If you're not transcoding, an h.264 1080p stream read from disk is going to be around 40-70Mb/s. Even 3 or 4 4k streams shouldn't be an issue for the hardware you picked. And you should still have a lot of headroom left over.

Now, if you install something like Plex in a jail and plan to transcode, that's another issue.

I read that FreeNAS needs 1GB Ram for 1TB of Storage.
One question here: do I need 39GB RAM, 42GB RAM (without the 0,931 factor) or 56GB RAM (8x[3TB+4TB]).

We run 64G on our servers. The extra RAM is useful as cache. We size based on our data files, which are about 10G. With 64G of ram we typically have enough free ram to cache 3 full file writes.

The next future upgrade would be the investment of the next HDD-array with 8*4TB WD Red
Which would give me a total of 61,446TB.

In the end I would like to be able to add another 24 HDD’s (Total 48+), that the reason for my following hardware configuration.

Hardware:

Supermicro X10SRL-F
8 ECC RAM slots
7 PCIe slots
10 SATA ports
http://www.supermicro.nl/products/motherboard/xeon/c600/x10srl-f.cfm

Intel Xeon E5-1620v4
LGA2011-3
4x 3.5GHz / 8 Threads
140W TDP / max. DDR4 PC2400
https://ark.intel.com/products/92991/Intel-Xeon-Processor-E5-1620-v4-10M-Cache-3_50-GHz

32GB DDR4 PC2133 ECC Registered CL15 Samsung
Can I start with only one RAM stick installed, or do I need to have at least two installed?
Modul number: M393A4K40BB0-CPB
Listed on the super micro website :D
http://www.supermicro.nl/support/resources/memory/display.cfm?sz=32.0&mspd=2.133&mtyp=68&id=C4F64B47A0C3DADA42A2C14BAD9EBD99&prid=84050&type=DDR4 1.2V&ecc=1&reg=1&fbd=0

Inter-Tech 4U 4129-N Case
https://www.inter-tech.de/en/products/ipc/server-cases/4u-4129-n

Inter-Tech Rails
https://www.inter-tech.de/en/products/ipc/19-accessories/2u-rail-650mm

24 HDD’s
Western Digital and Seagate

PSU
Seasonic X-850
https://seasonic.com/product/x-850/

24 HDD’s:
W_peak = 24*1,75A*12V = 504W
W_idle = 24*3,3W = 79,2W
W_R/W = 24*4,5W = 108W

W_motherboard = 50W (I haven’t found some numbers)
W_ram = 8x5W = 40W (Same here…)
W_Xeon = 140W (TDP as „guess number“)
W_HBA = 2*10W = 20W

W_peak= 504W + 50W + 40W + 140W + 20W = 754W

For the 48 HDD configuration I would simply install another Seasonic X-850W

HBA:
IBM ServeRaid M1015 - is this a fake one? How can I detect a fake one?
http://www.ebay.de/itm/IBM-ServeRaid-M1015-SATA-SAS-HBA-Controller-RAID-6Gbps-PCIe-46M0861-LSI-9220/192161121837?_trksid=p2047675.c100005.m1851&_trkparms=aid=555018&algo=PL.SIM&ao=2&asc=43784&meid=d3af0e19611b449fb3621ba02e7df3f9&pid=100005&rk=1&rkt=6&sd=201890415718

Intel RES2SV240 24 Port - can I also go with one like this? I know you guys recommend the M1015 but I read that some people use the 24 Port for bigger configurations.
http://www.ebay.de/itm/Intel-RES2SV...957827?hash=item3ae0a42883:g:w5cAAOSwZQRYfzaw

Questions:
So what do you guys think about my configuration?
Will it handle my planned load?
How big is the reverse? (circa...)
What amount of RAM do I need? (see "full" question in the text above)
Can I start with only one RAM stick? (again in the text above^^)
Would you still recommend the M1015?
Will it work with my planned "one big pool and the x * 8-drive-arrays"-system?
I don't need spare drives, do I?

Thank you for your help,
Alex

Your config is pretty close to the primary configuration at $dayjob, except we use real Avago/LSI 9211-8i and Solarflare 10G cards on Supermicro X9-SRLf-f motherboards with 64G and E5-1620 v2. All our boxes are in protocase 48 bay chassis with their custom PSU. They have been rock solid for us, aside from the occasional HD failures. We usually leave 1 drive in the case, so we can resilver a failed drive immediately and then schedule downtime to pull the faulty hardware and replace.

We actually have the X10-SRL-F with a 1620v4 on the bench as a test machine, but there just isn't a reason to upgrade yet. The next big thing we'll be doing is moving from 3TB drives to a larger size(hopefully 8TB). If we build any more boxes we'll need to upgrade our parts list though.. Things are getting hard to find.. :)

The case you found looks remarkably similar to the RoseWill L4500. I have one at home that I built a GPU rig in. The case is spartan, but seems to work OK. You might want to check the height of your CPU cooler and make sure it fits.

Good luck, you should have plenty of horsepower in the NAS for your environment.

 

[AlexIsNas]

Thank you Eric and Joe.



I will start with Eric:

You must mean RAIDZ2. I really don't know where people get this "ZFS 2" name from...

Yes, I mean RAIDZ2. I guess in some youtube videos they said ZFS 2. I will keep that in mind.

https://forums.freenas.org/index.php?threads/zfs-raid-size-and-reliability-calculator.28191/

You seem to be seriously overestimating available space. The difference between TB and pseudo-TB is something like 0.909, not 0.931.
@Bidule0hm's calculator will give you a very good estimate.

Another thanks for this hint. The 0,931 were „fixed“ in my mind. In the „old“ days all my HDD’s had 931GB instead of 1TB. So I used this factor as a calculating base.

You should consider something like a used Supermicro chassis, the experience will probably be much better.

But the Supermicro chassis are expensive as ….

Comes from China == probably fake.

that’s what I thought…

That's not an HBA, it's an expander. You'd probably want one, but if you go with the Supermicro chassis, you can get a backplane that already has the expander.

I need a short explanation on that. I thought I can’t use expander cards/boards. Or can I, but I also have to use a HBA card and the expander is just a better cable management?

If you want to transcode 4K, the Xeon E5-1650 v4 might be a better choice.

I keep that in mind and will rethink my choice.

What do you mean?

I thought about the E5-„load“ with e.g. 4 1080 streams. Will it be at around 20, 50 or 80% of load? Just to say if it will be under or overpowered. Or what I hope, be just right.

It's a purposely-vague rule of thumb. More RAM is better.

Ok, I got that. But the rule of thumb is for each TB of space I have, or for each TB of space I could use?

You'd better have a bare minimum of one drive burned in and ready to go for when (not if - when) a drive fails. At your scale, probably two.

Can I add the spare drive in every VDEV or do I need a spare drive for every VDEV?

And now thanks to Joe:

Your calculations are a bit optimistic but close enough for general planning. Do not forget about maintaining 20% free space for optimum pool performance so ensure you cut that out of your total capacity.

Will I be able to fill more space when adding more ram?

Not really, but if you have a failure you might want to shut down your system if you cannot lay your hands on a new drive within a few days. But of course you would maintina a backup of your important data just in case all hell breaks loose.

Same question as above: Can I add the spare drive in every VDEV or do I need a spare drive for every VDEV?

Refer to the user manual for the motherboard and find out. They generally tell you.

I will and post my result.

Thanks again.

 

[c32767a]

Thank you Eric and Joe.

I thought about the E5-„load“ with e.g. 4 1080 streams. Will it be at around 20, 50 or 80% of load? Just to say if it will be under or overpowered. Or what I hope, be just right.

Again, if the system is just serving data to a remote PLEX server, i doubt you'd even see the CPU usage needle move.
If you're talking about running plex in a jail and having a transcoded stream to your plex player, then you will be loading your CPU. My home plex server runs in an ESX VM and comfortably does 4 1080P streams on 4 cores from a E5-2430 (2.5Ghz)
Plex is it's own beast though. The workload can change depending on the client, etc. I personally would recommend running plex on another device, rather than on your NAS hardware. I'm a purist. I think the NAS should do NAS things only.

Ok, I got that. But the rule of thumb is for each TB of space I have, or for each TB of space I could use?

This is a rule of thumb. You need at least 8GB. Beyond that, it's a cost/benefit thing. You will benefit from as much ram as you can afford.. Nothing is going to pop up and say "your storage pool can't grow because there's not enough RAM". (I'm ignoring deduplication. Don't run it.)

Can I add the spare drive in every VDEV or do I need a spare drive for every VDEV?

You are better off having one drive in your box and just using it to replace a failing drive when the situation occurs. There is no reason to dedicate a warm/hot spare to each vdev in your use case.

 

[Jailer]

I thought about the E5-„load“ with e.g. 4 1080 streams. Will it be at around 20, 50 or 80% of load?

It depends on how you plan on accessing the files. Are you going to run Plex? If so and your client devices support direct play then you will just be streaming data and the load will be minimal. If the clients require transcoding then the load will be significantly more based on the source material.

 

[AlexIsNas]

Now, if you install something like Plex in a jail and plan to transcode, that's another issue.

Tell me more about that. What should I have to use plex?

We run 64G on our servers. The extra RAM is useful as cache. We size based on our data files, which are about 10G. With 64G of ram we typically have enough free ram to cache 3 full file writes.

I would like to start with one 32GB RAM module. Because the cost for one module are at around 300€. I would like to do this investment in a few month.

configuration at $dayjob

What is day job? I don't get it...

We actually have the X10-SRL-F with a 1620v4 on the bench as a test machine, but there just isn't a reason to upgrade yet. The next big thing we'll be doing is moving from 3TB drives to a larger size(hopefully 8TB). If we build any more boxes we'll need to upgrade our parts list though.. Things are getting hard to find.. :)

So you would recommend my config?

The case you found looks remarkably similar to the RoseWill L4500. I have one at home that I built a GPU rig in. The case is spartan, but seems to work OK. You might want to check the height of your CPU cooler and make sure it fits.

All our boxes are in protocase 48 bay chassis with their custom PSU.

That's kind of what I thought about. I will build it my own. Remove the front part of the chassis and add a HDD's mounting plate to fix the HDD's in place.

Thank you for your thoughts so far.

 

[Jailer]

What is day job? I don't get it...

He means he deals with this type of stuff every day where he works.

 

[Dice]

4k appears to be a completely different beast. A quick google around the web gathered some experiences:
CPU's of passmark scores +13k still manage to stutter during playback...
CPU's of passmark ~9k "buffers terribly".
The goto solution appears to be - equip HTPCs and playback devices with more powerful GFX To support the playback on device, as opposed to have the NAS do the work. It simply isnt feasible with 4k.

Since FreeNAS doesnt do GFX addons, you're left "without" future proof in this regard. Setting things up to handle a couple of streams 1080i streams (like the 1620 would handle) is about what you can do in terms of CPU.
The extra boost to the 1650 ....won't make the difference.

 

[Ericloewe]

I need a short explanation on that. I thought I can’t use expander cards/boards. Or can I, but I also have to use a HBA card and the expander is just a better cable management?

https://forums.freenas.org/index.ph...-sas-sy-a-primer-on-basic-sas-and-sata.26145/

But the rule of thumb is for each TB of space I have, or for each TB of space I could use?

Neither. And both. It's deliberately vague.

Can I add the spare drive in every VDEV or do I need a spare drive for every VDEV?

You could use a hot-spare, but I meant that you'd keep the drives on the shelf.
Hot-spares are per pool.

Will I be able to fill more space when adding more ram?

More RAM will help, but that's the wrong way of looking at it. Performance will absolutely tank around 90% and very bad things happen if it fills up completely.

 

[c32767a]

4k appears to be a completely different beast. A quick google around the web gathered some experiences:
CPU's of passmark scores +13k still manage to stutter during playback...
CPU's of passmark ~9k "buffers terribly".
The goto solution appears to be - equip HTPCs and playback devices with more powerful GFX To support the playback on device, as opposed to have the NAS do the work. It simply isnt feasible with 4k.

Since FreeNAS doesnt do GFX addons, you're left "without" future proof in this regard. Setting things up to handle a couple of streams 1080i streams (like the 1620 would handle) is about what you can do in terms of CPU.
The extra boost to the 1650 ....won't make the difference.

4k video is bits on the wire just like 1080p. That's why I made the distinction between serving the data and playing/transcoding it. Serving and transporting the data is nothing.

But, like you say, playback can be a problem depending on the client thats' doing the playback. Another reason why I prefer to separate transcoding/playback from my NAS platform. I don't want to build a high performance machine that is on 24x7 but only used a fraction of that time. Better to put that capacity at the client where it's only powered when needed and has a shorter lifecycle.

 

[c32767a]

Tell me more about that. What should I have to use plex?

Plex is a whole discussion by itself. I personally would use 2 separate servers. a NAS for file service and storage and another box to run PLEX. It can then mount the storage from your NAS and serve your media to your playback devices.
I would suggest talking to the Plex folks about what kind of resources that would take because as Dice posted, if you plan to do 4k or lots of video transcoding, it might require a lot of CPU.

I would like to start with one 32GB RAM module. Because the cost for one module are at around 300€. I would like to do this investment in a few month.

Based on what you described, I think 32G would be fine.

What is day job? I don't get it...

Sorry, slang for "the place I work", without exactly admitting who that is.. :)

So you would recommend my config?

If I was building a NAS for a small to medium sized office, it would look pretty much like what you listed, allowing for parts available in the US vs Europe.

That's kind of what I thought about. I will build it my own. Remove the front part of the chassis and add a HDD's mounting plate to fix the HDD's in place.

Thank you for your thoughts so far.

the case is no frills, but should be fine for what you're planning, though I expect you know you 'll run out of drive bays before you hit 48 drives.

 

[joeschmuck]

Will I be able to fill more space when adding more ram?

It's not as easy as adding a new hard drive to expand your storage capacity and has nothing to do with the amount of RAM you have. Before jumping in I think you need to understand how a ZFS vdev works and what you can and can't do. This is very important because anyone who adds a single hard drive to the vdev/pool ends up putting all data at very high risk of loss and everyone ends up rebuilding the vdev/pool after making this tragic mistake. Read this by Noobsauce (yes that was his original forum name).

 

[danb35]

But the Supermicro chassis are expensive as ….

They can be had used much more reasonably, or at least they can in the U.S. My entire server (see my sig) was about US$1000 just over a year ago, and though that was probably a better-than-average deal at the time, there's lots of gear out there, and it can be had fairly reasonably, even if you have to pay a small fortune to ship it.

 

[c32767a]

They can be had used much more reasonably, or at least they can in the U.S. My entire server (see my sig) was about US$1000 just over a year ago, and though that was probably a better-than-average deal at the time, there's lots of gear out there, and it can be had fairly reasonably, even if you have to pay a small fortune to ship it.

I've often wondered about the fan noise on that chassis when it's populated.. Hovercraft?

 

[Dice]

I've often wondered about the fan noise on that chassis when it's populated.. Hovercraft?

Beyond hoover craft.
If noise is a concern what so ever - don't even consider.
There are some stories floating around the forums for #reference about the noise.
Ie, one I particularly remember is a guy installing the server in the crawl space beneath his house. - Still able to hear the machine in every part of the house.

 

[CraigD]

You in Europe an X-Case case maybe cheaper than Supermicro

https://www.xcase.co.uk/search?q=RM424

Have Fun

 

[AlexIsNas]

Ok, thanks to all of you.

As far as I understand my configuration is good. I will fallow with c32767a statement.

I think the NAS should do NAS things only.

I also read all the introductions you guys send me and I think FreeNAS is still the best way for me. I will go with 8 HDD's in RAIDZ2 as a VDEV and add VDEV's with 8 HDD's from time to time. I'll also keep one spare drive lying around.

Just one last question^^
The Supermicro Chassis are way to expensive for me (1800€ for a new one). Even a used one with 36 bays costs at the moment 800€.

I can get a new 24 bay inter-tech storage case for 500€. What do you guys think about this case?
https://www.inter-tech.de/products/ipc/storage-cases/4u-4424

When this case is full I will add another case.

 

[Dice]

500€ vs 800€ SM.

In the grand scheme - it is typically cheaper to get a used SM box, once all costs add up. Then you'll get the vastly superior quality for free.
SAS expander backplanes (reducing required HBA's) and dual top quality power supplies for less than 300€.

There are some points regarding this in the hardware recommendations resource.