ZFS setup question

Status
Not open for further replies.

bboy8012

Cadet
Joined
May 22, 2018
Messages
9
Hi All,

I'm looking to build a 45 drive FreeNAS media server and am looking for a recommendation of an optimal setup for the system. I plan on eventually filling all 45 bays full of 10TB drives. My first question is which is optimal and most storage for media storage 9x5 vdevs RaidZ2, 5x9 vdevs RaidZ1 or 3x15 vdevs RaidZ2. I plan to add 280GB Optane drives for SLOG/L2ARC. System specs are:
Chenbro 40700
Supermicro X9DRI-LN4F+
(2) Intel E-2670 v2
128GB ECC RAM
(1) Mellonax ConnectX3 40GB NIC
(1) 32GB SataDom for FreeNas

Thanks
 

Inxsible

Guru
Joined
Aug 14, 2017
Messages
1,123
most storage for media storage 9x5 vdevs RaidZ2, 5x9 vdevs RaidZ1 or 3x15 vdevs RaidZ2.
Usually it is advised to not go beyond 8 drives per vdev for performance's sake. Given the same number of drives, RAIDZ1 will always give you more storage space than RAIDZ2 at the cost of redundancy.
plan to add 280GB Optane drives for SLOG/L2ARC.
Are you sure you need one or the other or both?
(1) Mellonax ConnectX3 40GB NIC
Do you know if that is compatible with FreeNAS/FreeBSD in terms of drivers and such?
(1) 32GB SataDom for FreeNas
Why not simply use a SSD? Gives you much better wear leveling and is not as finicky as a SataDOM
 

IQless

Contributor
Joined
Feb 13, 2017
Messages
142
With 10TB drives, RAIDZ1 should not be considered. Go with at least RAIDZ2.
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
I agree with @IQless, RAIDZ2 is sensible considering resilver time.

I think you should look at the RAM before considering L2ARC with 450TB of raw storage, you're going to need to think about 256GB or even 512GB of RAM to get good performance over time. Remember that adding boat-loads of L2ARC is no good as it steals RAM to manage the tables (about 1GB of RAM for each 10GB of L2ARC).

SLOG/ZIL is only helpful if you want a lot of Synchronous writes (usually for ISCSI VM targets), so it's not clear from the above if you really need it.

More VDEVS in your pool = more performance. Perhaps you can consider 7x 6 drive VDEVS in RAIDZ2 with a few open slots (to use for burning in and during resilvering with the dying disk still in the pool) or hot-spares. I know this proposal loses you quite a lot of storage, but you will need to decide the trade-off between storage, performance and data protection/redundancy.

A little experimentation with this site: http://wintelguy.com/zfs-calc.pl

might help you make the decision.

With my recommendation, you will be able to use 195 TB

With 5 x 9 drive VDEVs in RAIDZ2, you could have 238TB at your disposal. (but probably not as good performance and no free drive slots for burn-in/resilvering)
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
As an additional note, the resilver time makes it interesting to see how many years you could expect to operate with one data-loss event. Replacement time is also a key factor (hence thinking about hot spares to reduce that to zero)

You can use the calculator provided here:
https://www.memset.com/tools/raid-calculator/

The calculator only runs up to 5TB drive size, but you can get some kind of idea of the impact (you can just cut the rebuild speed in half to get the 10TB numbers).
Read the DLO/y as (for example) "1 in 27" meaning you can expect to see one complete data loss event in 27 years if you operate the system you input.

You need to read RAIDZ as regular raid so RAIDZ2=RAID6 in the calculator (only for the purposes of this calculation are they the same thing... otherwise, completely not the same)
 
Last edited:

bboy8012

Cadet
Joined
May 22, 2018
Messages
9
Appreciate all the feedback, I am looking into replacing the motherboard currently on the system that has the capability of RAM up to 512G. Also I think it’s a valid point of RaidZ2 with the 10TB drives and vdevs.
 

kdragon75

Wizard
Joined
Aug 7, 2016
Messages
2,457

bboy8012

Cadet
Joined
May 22, 2018
Messages
9
These two things do not normally go hand in hand. Intel thanks you anyway.
How many hundreds of clients will you be streaming 4k to?

Lol, wont be, just my 2 kiddos and a few friends. Im mainly doing this to gain knowledge and hopefully never have to expand my storage again.
 

kdragon75

Wizard
Joined
Aug 7, 2016
Messages
2,457

kdragon75

Wizard
Joined
Aug 7, 2016
Messages
2,457
Yes I have a Arista 7050Q-16 that I picked up for a steal.
Oooo its even proper ethernet. No messing with infiniband junk!
 
Status
Not open for further replies.
Top