New User, New Build

[rfielder]

Greetings all!

I am at the starting point of specing out the requirements for a FreeNAS server.

As far as I can see, the main requirements are ECC memory and horsepower. There are other things, such as Intel networking, of course. The hardware guides are very interesting, and look like they will invaluable.

The rest depends on how one will use the box.

To get ECC, it means a Xeon processor. That is good. I like that idea.

In the end, I expect to end up with a box that has 8 data drives, boot drive, acting as a NAS and a media server (likely Plex), and some other stuff as I see what can be done. For costing and sizing reasons, I will likely start with 4Gb drives, ending up with 24Gb free space.

First questions:

If you use your FreeNAS for more than a NAS, are there advantages to going to more powerful processors?

Which helps more - CPU speed, core count, multithreading, or multi CPU? Or which combination thereof? Or is the best policy to just break the bank and go for the greatest and latest?

I expect that 32Gb RAM would be enough to start. A lot of motherboards I have looked at top out at 64Gb. Is that a realistic max, or it is wise to invest in a motherboard that allows more than 64Gb? I know that this is completely depending on what you do with the system, so nobody can say what I will need - but if I keep this system running for more than 2 or 3 years, might there be any expectation that future opportunities would benefit from the extra elbow rooms? Yeah, this is really a speculation, but informed speculation is always a good thing.....

If I build a NAS with 8 drives, each 4Tb, can I later start upgrading them one drive at a time? E.G., replace one 4Tb drive with an 8Tb drive, and have FreeNAS accept it and increase the total size? I expect the answer to this is in the FreeNAS documentation, which I am slowly reading through. Just jumping the gun, since the answer may impact my initial purchase of drives.

Many thanks in advance for any replies and advice!

 

[Jessep]

You'll need to define your use case(s) beyond a NAS before anyone can advise you.

vdevs will only expand size when all drives have been replaced.

You wont end up with 24TB usable space. For that many drives and that drive size it's recommended to use RaidZ2.

(24TB - 8TB redundancy) * 80% max pool fill = ~12.8TB usable (11.6TiB)

What is your current space usage?
What is your expected annual growth rate?

• If your drives have a 5 year warranty then annual growth rate * 5 = expected size when you need to replace the drives

How many drives will your case/chassis hold?

• If you have a 12 drive chassis you can start with a 6 drive vdev and add a second 6 drive vdev later.

 

[rfielder]

You won't end up with 24TB usable space. For that many drives and that drive size it's recommended to use RaidZ2.

(24TB - 8TB redundancy) * 80% max pool fill = ~12.8TB usable (11.6TiB)

Sorry, I am missing something.

I said eight drives of 4Tb. That is 32Tb, not 24Tb.

My current usage is all over the place, 14Tb on the server plus a load of stuff dumped off to external hard drives that have been gathering dust until I could start this project. 24Tb free will be a good starting point, but at this point I am not sure it will be sufficient.

However, maybe going with 8Tb drives to start makes more sense. Maybe six drives, total of 48Tb, leaving 32Tb free?

Yeah, I am very much a newbie, so estimates on free space are based on RAID6. Does that translate correctly to the FreeNAS environment?

 

[Jessep]

Bah early morning math is always a mistake.

You are correct, I missed the 8X4TB = 32TB.

This is the most posted tool for calculating space on FreeNAS:
https://wintelguy.com/zfs-calc.pl

 

[rfielder]

I missed the last "* 80%" on my calculations.

Guess that means the 32b free is reduced by another 20%.

Then, one must always respect that ZFS does not want to be full.

10Tb drives are staring to look very attractive..... :)

 

[rfielder]

If your drives have a 5 year warranty then annual growth rate * 5 = expected size when you need to replace the drives

Sorry, but can you explain why the drive warranty is a factor?

 

[rfielder]

You'll need to define your use case(s) beyond a NAS before anyone can advise you.

Initially - Move the 11Tb of files off the Windows Home Server, so that system can be decommissioned and the 2Tb drives cleared and reused or sold.

Second - I have external drives, ranging from 5Tb to 3Tb, where I have been dumping files for some years. Bringing them in and seeing what is there is next.

Third - ongoing data and media storage. Minimal data, maybe 500Mb of photos that must be protected, user data, mostly media files.

Fourth - investigate Plex

Fifth - playing with other features of FreeNAS

 

[Jessep]

Just good practice to not run drives until they die, if you get good drives with 5 year warranty that's a good time frame to replace.

"Bathtub curve" Most die in the beginning and the end of the life cycle.

 

[rfielder]

Hardware selection:

This is what I am considering so far. Opinions and experiences are appreciated!

SUPERMICRO MBD-X11SSH-F-O - the documentation on this motherboard is a bit less than clear as to max memory. Some places list it as 64Gb max, others say that a firmware update will let it accept 32Gb memory to go to a total of 128Gb max.

Intel Xeon E3-1245 V6 Kaby Lake 3.7 GHz - cheapest one with hyperthreading. Likely not needed now, but I try to buy for future use if possible.

Either:
2 x Crucial 16GB 288-Pin DDR4 SDRAM ECC Registered DDR4 2666
-- OR --
2 x Crucial 32GB DDR4 2666 (PC4-21300) SDRAM Server Memory ECC Registered

Does it matter that these are "registered"?

I think these are coming through as links, since I am copying from my Newegg wish list. Hope that does not mess up anything....

 

[Jessep]

That board needs unbuffered not registered.

That's not a bad board and CPU for new, you may want to think about buy a used server instead.

Dell R730xd for instance, with enterprise iDRAC.

 

[rfielder]

Found unbuffered memory instead:

Crucial 16GB DDR4 SDRAM ECC Unbuffered DDR4 2666

 

[rfielder]

you may want to think about buy a used server instead.

Dell R730xd for instance, with enterprise iDRAC.

Why?

I am in Canada. Not sure that buying used gear here is the same as for you - assuming you are in the USA.

Although I am willing to consider your suggestion, I like buying and building stuff, so would need to know more before I start a search for used gear.

 

[rfielder]

Just good practice to not run drives until they die, if you get good drives with 5 year warranty that's a good time frame to replace.

"Bathtub curve" Most die in the beginning and the end of the life cycle.

That makes sense. Thanks!

One disadvantage - if you build it new, you are buying all the drives at once. In five (or three) years, they ALL run out of warranty. That means a huge sticker shock! :)

I have always run my drives until the system reports they are failing. This is one thing that Windows Home Server seemed to get right - I have never had a drive actually fail, there was always a warning so you could gracefully remove the drive without loss of data.

 

[cJZ]

Why?

I am in Canada. Not sure that buying used gear here is the same as for you - assuming you are in the USA.

Although I am willing to consider your suggestion, I like buying and building stuff, so would need to know more before I start a search for used gear.

There isn't much other than slim pickings up here to be honest, there are a few recyclers but none of them are knowledgeable about their gear... They mostly just cut the CPU pins and sell them for gold and ship the toxic stuff away to the Philippines... yeah, I said what I said! There is a seller on eBay out of Calgary who sells used enterprise gear at reasonable prices. Search for "calgarycomputerwholesale" and you'll find their store. Sadly for us Canadians, homelab gear is difficult to come by. If the price is decent, postage and/or customs will rape us.

Also, I run my drives until they need to be replaced. I don't just replace them because they've got thirty thousand power on hours. It doesn't make sense in a home / small business setting especially if important data is diligently backed up, off site of course.

 

[Heracles]

Hi,

One thing to consider is Raid-10 vs RaidZ-2. If you wish to do more with your FreeNAS and start hosting VM, iSCSI and more, you will need a maximum of IOPS and Raid-10 will give you way more than Raid-Z2.

To change the structure of a pool requires to empty it, destroy it and rebuild it. As such, it is essential to get it right the first time, even if you are about to use it only in a year or two.

32G of RAM is surely enough for FreeNAS and a single service like Plex. Up to 64, you will have enough for a few more services.

As for the CPU, what you need more will depend of the work you will do on the server. Single threaded process will need higher frequency while multiple process / multi-threaded tasks will use more core.

The planning phase is very important. When done right, you will save yourself a lot of trouble. When wrong or missing, you will hit the wall faster than expected.

Have fun designing your setup,

 

[rfielder]

One thing to consider is Raid-10 vs RaidZ-2. If you wish to do more with your FreeNAS and start hosting VM, iSCSI and more, you will need a maximum of IOPS and Raid-10 will give you way more than Raid-Z2.

RAIDZ2 for me, all the way. Really not needed for the media files, but there will be stuff that can not be lost - person stuff, photos, documents, etc that are very important to us.

The planning phase is very important. When done right, you will save yourself a lot of trouble. When wrong or missing, you will hit the wall faster than expected.

That has been a mainstay of my life for over 30 years. Take you time, think it through, get feedback from others, think it again, then repeat a few more times. It has helped me be successful in my projects (when I was in the position to have reasonable control), and even those projects that came in a bit late made up with increased productivity and higher moral with the staff.

I am taking the time here to communicate, and to read, and the think.

And, as they say, great minds think alike! :)

 

[pro lamer]

Disclaimer first: I'm not sure if pre-owned servers you find use Xeon-E3/Xeon-Entry or not (versus Xeon E5) but I'd like to refer to this:

Some places list it as 64Gb max, others say that a firmware update will let it accept 32Gb memory to go to a total of 128Gb max

Wow! I thought 128 was only for X11SCH and similar (Xeon entry and newer i3 generations)

Sent from my phone

 

[Newfoundland.Republic]

Why?

I am in Canada. Not sure that buying used gear here is the same as for you - assuming you are in the USA.

Although I am willing to consider your suggestion, I like buying and building stuff, so would need to know more before I start a search for used gear.

I picked up a Supermicro SYS-6027R-3RF4+on eBay for CDN$600 (depending on the exchange rate. Shipping is a bit of a bugger (expect around $125) and duties. Shipping was from the US.