Right Hardware for first build? and expand Storage

[nico3]

Hello Guy,
I'm Nico and I've been reading the forum for some time now. I didn't want to post here and ask the same question as other 10k people but I decided that before I spend the $ I should get the PROs opinion...
So sorry for the noob questions: I'm about to buy the hardware for my first FreeNAS build, the use is mainly data storage for our small business. We have around 16 devices on the network (both Win and macOS). The "heavy users" are our interior designers that work on CAD files (around 6GB each) directly on the server: open file without moving it to local and once they finish working they save a new copy of the file on the server.
Will this hardware be ok for my scope (mainly "large" files transfer)?

So this is the hardware list:

MB: Supermicro X10SL7-F
CPU: Intel Xeon E3-1220 v3 @ 3.1 GHz
RAM: Crucial 16Gb Kit - CT2KIT102472BD160B
HD: 4 x WD 3TB - WD30EFRX in RAIDZ2
PSU 450W are ok?

RAIDZ2 should give me 6Tb of usable space right? how difficult is it to expand the usable space with RAIDZ2?

I've checked the hardware guide, the manual and the guide from Cyberjock (REALLY helpful) but I'm still not 100% clear on how to expand a RAIDZ2, do I have to create a new VDEV? can a folder be shared between 2 or more VDEVs?

I'm based in Sydney and I've a budget of $1500 (AUD), with this config I'm at $1594 and I can't really go over this. I would love to get a Xeon 1230 and 32 Gb of ram but that is OB

Let me know if you would change something in this config.

Thank you for your help!

 

[Stux]

TheX10SL7 supports a lot more disks than you are using, and it has actually been obsoleted by the X11 platform.

The X11SSM-F will allow you to use 8 disks and a v4/5 CPU and up to 64GB of ram, vs only 32GB on the X10 platform.

I would suggest going with the X11SSM-F platform.

You can increase a RaidZ2 pool by replacing ALL drives in a vdev with larger drives, or adding another vdev to the pool. So in your example, add another RaidZ2 vdev to your pool.. say another 4 drives.

Alternatively, if you use mirrors, you can upgrade each mirror (a pair of drives) at leisure, or add a pair of drives at leisure to expand the pool, so say, you want more space, add a pair of 6TB drives next time.

Alternativly, when you do want to expand your pool next time, and you acquire another 4 drives, you can use some creative data juggling to end up with an 8-way RaidZ2 pool for a total of 18TB out of 24TB raw, vs 12TB if you continue with 50% parity like you have currently.

Its recommended to only use 80% of the usable storage. I normally advise to begin planning your storage upgarde when you get the 80% warning, and have it implemented by the time you get to the 90% warning, but in a commercial scenario its probably better sooner rather than later.

The fuller the pool, then the worse fragmentation becomes, and there is no defragment in ZFS.

Once a pool gets to 90% full ZFS enters an anti-performance mode where it prioritizes non-fragmenting the pool over performance, thus there is a performance cliff at 90% full.

You do not want to go to 100% full.

So, if you decide to go to 8-way RaidZ2 eventually, you would do this by partitioning your new drives in half, making an 8way raidz2 with those new drives, transferring the pool to the new drives (can be done incrementally to minimize downtime), and the replace the partitions with fullsize disks, to grow the half size pool to fullsize. Ask for more details later if you want.

If you get the X11 board, then you should start with a single 16GB DIMM rather than an 2x8 kit like you have.

What are you using as the boot drive?

PS: X11SSM-F is 53$ cheaper on amazon
https://www.amazon.com/Supermicro-M...=UTF8&qid=1495610589&sr=8-1&keywords=x11ssm-f

If you are concerned about adding more drives than 8 drives in the future, you can get the budget for an M1015/M1115 LSI HBA in the future, about 150AUD. Which will allow you to add another 8 drives and use a PCIe slot.

Because 8 drives is a nice amount, I would recommend using 2 USB Sandisk Cruiser Fit thumb drive in mirror as the boot device. They're like 12AUD on ebay, shipped.
http://www.ebay.com.au/itm/SanDisk-...hash=item1c6ddf9f17:m:mVj4EHPoYruaWvg730dCOPA

And finally, as you've probably heard, RAID is not Backup. You need a backup plan for the data.

 

[nico3]

Thanks Stux!
I based my hardware on the hardware guideline, your post is great news as the X11SSM-F is actually $100 cheaper than the X10 (but more difficult to find).
The X11 means that I'll to use the 1151 socket, so what CPU should I use? Same Xeon E3 1220, v5 or v6? a 1230 is overkilled for the moment?

Thank you for explaining how to expand the drives.
I saw the option of replacing the drives one by one with a larger version but that is a bit of waste isn't it?

What are the real advantages of mirroring over RAIDZ2? with RAIDZ2 I can have "more usable space" out of the same number of drives and I can lose up to 2 drives.
If a folder if on my "first" VDEV and I need more space for that folder (assuming that the VDEV is nearly full) do I have to move that folder to a new (larger) VDEV or just by adding the new VDEV the pool gets bigger and the folder gets more space?

I'm still a bit unclear on how VDEVs relate with files/directories (folder).

Just saw your edit, yes as boot I was thinking about 2 16Gb USB. and Yes I'm already thinking about a backup plan, I'll have at least 2 external drives off site and I was looking into some backup plugins for incremental cloud backups... but that will be implemented only after I setup the server.

 

[Stux]

Thanks Stux!
I based my hardware on the hardware guideline, your post is great news as the X11SSM-F is actually $100 cheaper than the X10 (but more difficult to find).
The X11 means that I'll to use the 1151 socket, so what CPU should I use? Same Xeon E3 1220, v5 or v6? a 1230 is overkilled for the moment?

Yes, the E3-1220v5 or v6. It'll be more performant, but basically the same characteristics as the 1220v3

Thank you for explaining how to expand the drives.
I saw the option of replacing the drives one by one with a larger version but that is a bit of waste isn't it?

yes

What are the real advantages of mirroring over RAIDZ2? with RAIDZ2 I can have "more usable space" out of the same number of drives and I can lose up to 2 drives.

Mirroring provides more IOPS, since the IOPS of a vdev is about the same as an individual drive in a vdev, and in a mirroring scenario you use 2 drives per vdev. RaidZ2 provides faster writes and more capacity when you have more than 4 drives in a vdev.

You can add/strip mirrors from a mirror vdev. You can't do that with RaidZ2.

You can add two drives at a time with a set of striped mirrors. You can't do that with RaidZ2

You can get extra capacity by replacing two drives with bigger drives in striped mirrors, you can't do that with RaidZ2.

RaidZ2 protects you from dual disk failure, with 2-way mirros you can lose the whole pool if 2 drives fail. You could go to 3-way mirrors but then you have 66% parity! and only 33% data!!! vs say, 25/75% with 8-way RaidZ2.

If a folder if on my "first" VDEV and I need more space for that folder (assuming that the VDEV is nearly full) do I have to move that folder to a new (larger) VDEV or just by adding the new VDEV the pool gets bigger and the folder gets more space?

You add the vdev to your pool. The pool magically gets bigger, and everything on the pool has more space, unless you have quotas/reservations etc in use.

I'm still a bit unclear on how VDEVs relate with files/directories (folder).

FreeNAS can host multiple pools. You would only really use different pools if one was say a super fast SSD pool, adn the other was a slower HDD pool, or perhaps as a sortof in-box backup.

Each pool is made of one or more vdevs.

Each vdev (virtual device) is made of one or more devices... ie... harddrives.

The vdev provides the redundancy.

Just saw your edit, yes as boot I was thinking about 2 16Gb USB. and Yes I'm already thinking about a backup plan, I'll have at least 2 external drives off site and I was looking into some backup plugins for incremental cloud backups... but that will be implemented only after I setup the server.

ok

 

[nico3]

Great! everything more clear now, thanks!

So with RAIDZ2 there's no easy way to expand the total space, (pool) right?
While in Mirror I can expand the size "easily" but there's a bit more risk as only 1 hard drive from the same Vdev can fail.
I guess this is acceptable if I always keep a spare drive and replace the faulty drive straight away...

What is your personal opinion on this?

 

[Stux]

Well, the easy way to expand a Raidz2 pool is to add another vdev.

My personal opinion is that I'd rather the space saving from relatively wide raidz2

Which is why I just added another 8 drives to my pool... and why I built a 24 bay chassis

 

[Kevin Horton]

Great! everything more clear now, thanks!

So with RAIDZ2 there's no easy way to expand the total space, (pool) right?
While in Mirror I can expand the size "easily" but there's a bit more risk as only 1 hard drive from the same Vdev can fail.
I guess this is acceptable if I always keep a spare drive and replace the faulty drive straight away...

What is your personal opinion on this?

It depends on how quickly you can replace the faulty drive and how lucky (or unlucky) you are. Do you ever travel such that you couldn't replace a failed drive within a few hours? Or, would you always shut down the NAS before leaving on a trip?

 

[nico3]

That's right... Now I'm not sure hahaha
My first ideas was to go for a RAIDZ2 and my main issue is that we will need to expand the size of our server in the future, but if I can just add a VDEV that's not big issue.
If I start with VDEV of 4 x 3TB drives in RAIDZ2 and I decide to expand the pool space in the future can I just add a VDEV of drives right? The problem with RAIDZ2 is that if I want to add drives I must use at least 4 drives...

Also the Supermicro X11SSM-F has a 4 weeks ETA, what are the main differences with the X11SSL-F? L stand for "low cost" right? what does that mean exactly?

Thanks again!!

 

[Stux]

there is another possibility. With command line shanigans you could end up with an 8way Raidz2 across all 8 drives in the future.

It involves making a second pool with 8 partitions on your 4 additional drives. Then you replicate your original pool to the new pool, and the you replace the partitions with full drives. Then you end up with 8way Raidz2. This assumes 3TB Drives.

Alternatively you just extend your pool with another Raidz2 vdev like you suggested. You still end up with half your pool used for parity this way.

 

[nico3]

Thanks a lot guys! much appreciated!
I'll order the hardware and start the setup ASAP!