new set up questions

[ybcnyc]

Hi,

I wanna build all flash freenas.
I got dell R720 that will be paired with M1015 in IT mode. Ditching the onboard H700 since it's not recommended. Server will have around 96GB.

I plan on using 6 x EVO 850 1TB drives with two Sansik 60G SSD for boot (mirror). drives will be overprovisioned with 15% and then pool won't go over 50%. Currently have NAS with 1.2TB that will be used as backup.

This storage is for VMs around 10 for now with future expansion, for 3 proxmox hosts over ISCSI.

What raidz would be recommended. mirrored vdevs for best performance ? or raidzX for fault tolerance ?
With mirrored vdevs I'd get around 2.4TB with 1.2T(50%) utilization. That will suit my needs but is raidzX better as far as safety ?
Thanks

 

[joeschmuck]

That is a good question because SSD's are very fast in the first place. Normally it would be mirrored vdevs but I'm curious to see what an expert says about SSD's, and I'm sure this has been asked before but I haven't seen it. I would think a RAIDZ2 would be fine because they are SSD's and virtually no latency with respect to a traditional spinning drive, but just because I "would think" doesn't make it a fact.

 

[ybcnyc]

it basically comes down to resilvering times where with mechanical disks should it be a consideration when picking disk setup however with SSDs the resilvering times should be much shorter. http://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-not-raidz/

any advice is appreciated.

 

[Robert Trevellyan]

it basically comes down to resilvering times

What do you mean?

Striped mirrors are the usual recommendation for block storage because more vdevs = more IOPS, but SSD IOPS are so much higher than spinning disks that it probably isn't an issue.

 

[ybcnyc]

the article i linked basically has TL;DR

• don’t be greedy. 50% storage efficiency is plenty.
• for a given number of disks, a pool of mirrors will significantly outperform a RAIDZ stripe.
• a degraded pool of mirrors will severely outperform a degraded RAIDZ stripe.
• a degraded pool of mirrors will rebuild tremendously faster than a degraded RAIDZ stripe.
• a pool of mirrors is easier to manage, maintain, live with, and upgrade than a RAIDZ stripe.
• BACK. UP. YOUR POOL. REGULARLY. TAKE THIS SERIOUSLY.

but given all SSD pool some of the points will be much faster thus making raidz setup more ideal in an SSD setup?

 

[Rand]

I am no expert but two more considerations:
-you do have 10GBe planned right? Else pool speed (likely) will not matter as it hopefully surpasses 1GB with both setups
-have you looked into using the 850's for parallel vm usage? Have you identified your Promox vm workoad patterns?
Depending on the kind of workload you have they might be sweet as sugar or bad as vinegar - I dont think they will sustain massive amounts of random writes for a very long time for example.

 

[Robert Trevellyan]

the article i linked basically has TL;DR
...
but given all SSD pool some of the points will be much faster thus making raidz setup more ideal in an SSD setup?

I'm familiar with the article, and it makes a strong case for striped mirrors. However, I think we agree that the much higher IOPS of SSDs are likely to make the benefit of having more vdevs moot for most applications.

You should also keep in mind that a pool is lost if any constituent vdev is lost, which for striped two-way mirrors means that you can't afford to lose two disks from any one mirror. Also, you will always suffer data loss if a URE occurs while re-silvering a two-way mirror.

 

[ybcnyc]

Right now it'll be 3x1Gbit bond on san and hosts but 10GBe is planned in the future.

I am no expert but two more considerations:
-you do have 10GBe planned right? Else pool speed (likely) will not matter as it hopefully surpasses 1GB with both setups
-have you looked into using the 850's for parallel vm usage? Have you identified your Promox vm workoad patterns?
Depending on the kind of workload you have they might be sweet as sugar or bad as vinegar - I dont think they will sustain massive amounts of random writes for a very long time for example.

right now bonded 2x1GBit on SAN and each proxmox host. VMs aren't that busy except email, spam filter, and surveillance storage. i believe the SSD setup will improve IOPS which will make the VMs snappier. the startup takes a while for all of them to spin up.

I'm familiar with the article, and it makes a strong case for striped mirrors. However, I think we agree that the much higher IOPS of SSDs are likely to make the benefit of having more vdevs moot for most applications.

You should also keep in mind that a pool is lost if any constituent vdev is lost, which for striped two-way mirrors means that you can't afford to lose two disks from any one mirror. Also, you will always suffer data loss if a URE occurs while re-silvering a two-way mirror.

That's what I was thinking too. raidzX allows for more fault tolerance and the IOPS boost will still be signification from mechanical disks.

 

[gpsguy]

Before you commit to a solution, I'd test 3-way mirrors. If you need more IOPS that what RAIDz2 provides, but want that additional fault tolerance, it might be a viable solution for you. Yes, there's a lot of overhead, but FreeNAS is cheap compared to what you'd pay most of the SAN vendors.

 

[ybcnyc]

Before you commit to a solution, I'd test 3-way mirrors. If you need more IOPS that what RAIDz2 provides, but want that additional fault tolerance, it might be a viable solution for you. Yes, there's a lot of overhead, but FreeNAS is cheap compared to what you'd pay most of the SAN vendors.

this looks interesting and lots of overhead. do all three drives stay in the system? I'm looking it up and people say you do 3 way mirrors and take one drive out physically for offline storage ?

 

[Robert Trevellyan]

do all three drives stay in the system?

Yes, in order to retain redundancy while re-silvering.

people say you do 3 way mirrors and take one drive out physically for offline storage ?

People say all kinds of silly things.

 

[ybcnyc]

thanks. i'll order 3 more EVO 850 from another vendor (got 3 from egg and 3 from amazon) and go with the 3 way mirror.

 

[Rand]

I'd suggest to simply test it - 3 way mirrors will most likely need the most disks so you are free to test all other variants before comitting to a final build :)

 

[ybcnyc]

will do with 2 way mirrors and can always go to 3 way without disturbing the pool. started looking at 10gb and the switches are pretty expensive. i see a lot of threads where box and hyper-visors are directly connected with 10gb cards (can be done for under $100). With 3 H-Vs can this direct connection be done ? Doesn't need it some sort of switching ? Basically is it 1:1 or multiple servers can be hooked up to freenas this way. thanks

 

[Rand]

Haven't done it myself but I imagine it will be 3x 1:1 - should be doable if you have enough PCIe slots or can get one or tow double cards.

 

[ybcnyc]

not sure how proxmox cluster would handle the separate 1:1 connections to the same LUN. i guess i'd have to try but it needs "shared" storage for migrations and what not. 1:1 would mean not clustered hosts most likely.

 

[Sakuru]

For 10 VMs I would think 3-way mirrors would be excessive for SSDs. Personally, I would go with a Z2. I really don't see what striped mirrors give you over Z2 when you use SSDs.

 

[Rand]

not sure how proxmox cluster would handle the separate 1:1 connections to the same LUN. i guess i'd have to try but it needs "shared" storage for migrations and what not. 1:1 would mean not clustered hosts most likely.

Hm you might be able access the FreeNas Server via Hostname to access the LUNs identically on all Promox Cluster Nodes? Still 3 distinct IPs o/c unless you can use a virtual IP for iSCSI ...

 

[pirateghost]

Proxmox handles iscsi with multiple nodes just fine.

 

[ybcnyc]

For 10 VMs I would think 3-way mirrors would be excessive for SSDs. Personally, I would go with a Z2. I really don't see what striped mirrors give you over Z2 when you use SSDs.

yes i will try z2 and 2 way mirror. 2way mirror obviously a bit faster, resilver is quicker, no stripe to write across all disks.