New nas multiple purposes

[Dsky07]

Hi guys. I am getting closer to creating the setup I want but am currently testing all the hard drives as documented by qwertymodo and greco. The testing is taking almost a week with a bunch of faulty drives having to be replaced before I can go online.

But to get to the point is the arrangement of the ZFS pools. I planned on running 8x6TB drives. None of the recommendations recommend going with 8 drives. RAIDZ Z2 and Z3 all recommend different amount of drives. So I was thinking of going with a 7 drive Z2 pool and a hot spare. Does this seem reasonable in terms of fault tolerance and also reasonable performance for my multiple server needs?

I also have a m2 pcie drive which I could use to host the VMs instead on the ZFS pool if that helps.

Thanks again for all your help.

 

[Mirfster]

But to get to the point is the arrangement of the ZFS pools. I planned on running 8x6TB drives. None of the recommendations recommend going with 8 drives. RAIDZ Z2 and Z3 all recommend different amount of drives. So I was thinking of going with a 7 drive Z2 pool and a hot spare. Does this seem reasonable in terms of fault tolerance and also reasonable performance for my multiple server needs?

While there are some recommended numbers for different RaidZ vdevs, that is not really a "Hard Line". You could easily do a RaidZ2 out of 8 drives, or even a RaidZ3. Having a "spare" in a Server that is easily accessible is not important to me personally. I would rather be making use of the drive; especially if I already have 2 drive redundancy.

Quoted from Slideshow explaining VDev, zpool, ZIL and L2ARC for noobs!

¨If you are not using compression:
For performance reasons it is preferred that you use these conventions when creating RAIDZ1, RAIDZ2 and RAIDZ3 VDevs. (n is any whole number you want….1, 2, 3 etc.)
RAIDZ1 should have the total number of drives equal to 2n + 1. (ie 3, 5, 9, etc. drives for the VDev)
RAIDZ2 should have the total number of drives equal to 2n + 2. (ie 4, 6, 10, etc drives for the VDev)
RAIDZ3 should have the total number of drives equal to 2n + 3. (ie 5, 7, 11, etc drives for the VDev)
This is to ensure ZFS stripes fall on the 4k sector boundary of newer generation hard drives.
If you do not intend to transfer large quantities of data constantly this thumbrule can be disregarded. For instance, if you intend to use your server to stream DVD rips of your collection, this thumbrule can be ignored

 

[Robert Trevellyan]

Quoted ... If you are not using compression:

And if you are using compression, which is the default and the right choice in almost all cases, there are no magic numbers of drives "for performance reasons".

 

[Dsky07]

And if you are using compression, which is the default and the right choice in almost all cases, there are no magic numbers of drives "for performance reasons".

Thank you. I will just go with the default compression setting and keep the complications to a minimum.

I had an extra HBA card flashed to IT mode but considering I will be running this off a USB flash drive I will remove the HBA card and use onboard controller to save on energy usage. Will see how the passthrough works for virtualization before that is finalized of course.

The next concerns I have is that my idle energy usage is very low around 70-80 watt but my PSUs are all over 600 watt. They are gold and platinum rated but overkill. This does not include an AMD or Nvidia GPU I plan to install later to test a VM gaming machine which might change everything. So a bit conflicted there on the right Power Supply to use.

 

[jgreco]

Generally RAIDZ3 is smarter than RAIDZ2 and a hot spare, unless the speed loss of RAIDZ3 is critical.

 

[fta]

Your use case is begging for mirrored vdevs.

 

[Dsky07]

Thanks. Can you elaborate a bit? I am just a bit scared of losing the wrong combo of drives and lose everything. I am open to it though if it makes sense.

 

[fta]

Thanks. Can you elaborate a bit? I am just a bit scared of losing the wrong combo of drives and lose everything. I am open to it though if it makes sense.

You're going to be running a bunch of stuff simultaneously, including one or more VMs. That's asking for IOPs. RAIDZn only gives you the IOPs of a single disk. True, you won't be as redundant as with RAIDZ2+, but you can mitigate some of that risk with the following:

1) Mirrored resilvers go much faster than RAIDZn resilvers. Especially if you are filling up those big 6TB drives. Double especially if you're doing Z3.
2) You're backing up
3) Regularly reading your data so you have a better chance of catching failing drives early (I scrub weekly)
4) Having an extra sata port: this is handy because if you can keep the failing drive connected while you're resilvering to a new drive, if you do end up having a read error on the "good" disk of the mirrored vdev, ZFS may be able to get the sector it needs from the failing drive.

 

[fta]

Oh, and if you're doing VMs, you're going to want a good SLOG device. If you have sync=always turned on, it makes an enormous difference.

 

[Spearfoot]

This is an excellent tutorial on setting up an 'All-In-One' FreeNAS+VMware server:

https://b3n.org/freenas-9-3-on-vmware-esxi-6-0-guide/

 

[Mirfster]

Yeah, 'cept I disagree with a few of his suggestions:

• MTU 9000
• "Striped" SLOGs (even posted there asking him about that)

But am currently testing/running a variant that is similar to his.

 

[Spearfoot]

Yeah, 'cept I disagree with a few of his suggestions:

• MTU 9000
• "Striped" SLOGs (even posted there asking him about that)

But am currently testing/running a variant that is similar to his.

I agree! Tried MTU=9000 and the VMXNET3 driver without success when I built my system (about a year ago). Ended up switching back to the E1000 w/ standard MTU. I only recently added a SLOG device (Intel DC S3700); before that I ran with sync disabled on my NFS datastore. Livin' on the edge! :)

But overall, Ben's guide is pretty handy.

 

[Dsky07]

You're going to be running a bunch of stuff simultaneously, including one or more VMs. That's asking for IOPs. RAIDZn only gives you the IOPs of a single disk. True, you won't be as redundant as with RAIDZ2+, but you can mitigate some of that risk with the following:

1) Mirrored resilvers go much faster than RAIDZn resilvers. Especially if you are filling up those big 6TB drives. Double especially if you're doing Z3.
2) You're backing up
3) Regularly reading your data so you have a better chance of catching failing drives early (I scrub weekly)
4) Having an extra sata port: this is handy because if you can keep the failing drive connected while you're resilvering to a new drive, if you do end up having a read error on the "good" disk of the mirrored vdev, ZFS may be able to get the sector it needs from the failing drive.

I will look into all of this. The thing regarding a spare port is that if I remove the HBA I was going to pass through the onboard controller which gives me 8 drives and 4 usable in mirror without a spare port. If I keep the HBA then can I selectively pass through individual ports of the onboard controller to the freenas VM while using others for SLOG etc.?

I will re-read the updated all-in-one guide too as I have better understanding than when I skimmed through first couple times.

 

[Dsky07]

I agree! Tried MTU=9000 and the VMXNET3 driver without success when I built my system (about a year ago). Ended up switching back to the E1000 w/ standard MTU. I only recently added a SLOG device (Intel DC S3700); before that I ran with sync disabled on my NFS datastore. Livin' on the edge! :)

But overall, Ben's guide is pretty handy.

Thanks. I am probably gonna go with the true and tried version.
Regarding a SLOG device I got a Samsung PM951 256GB M.2 NGFF PCIe Gen3 x4. Think that would work? I might use it for my VMs installation/SLOG device partition.

 

[Spearfoot]

Thanks. I am probably gonna go with the true and tried version.
Regarding a SLOG device I got a Samsung PM951 256GB M.2 NGFF PCIe Gen3 x4. Think that would work? I might use it for my VMs installation/SLOG device partition.

Ummm... I'm not familiar with the Samsung PM951. What you want in a ZIL SLOG device is low latency / fast write capability.

I do know that you want a ZIL SLOG device to be a dedicated device, so if you're thinking of using the Samsung PM951 for multiple purposes, with multiple partitions (as you seem to imply) - that's not a good idea. Please forgive me if I misunderstood what you meant.

 

[jgreco]

There's no value to using a PM951 as a SLOG device. It lacks power loss protection. It's better to just disregard sync writes and understand the possibility of failure exists rather than have a device that you're trusting to do a job that it cannot.

 

[Dsky07]

Ummm... I'm not familiar with the Samsung PM951. What you want in a ZIL SLOG device is low latency / fast write capability.

I do know that you want a ZIL SLOG device to be a dedicated device, so if you're thinking of using the Samsung PM951 for multiple purposes, with multiple partitions (as you seem to imply) - that's not a good idea. Please forgive me if I misunderstood what you meant. :)

I did imply to possibly use for both since it is large. I guess that idea is out of the window.

 

[Dsky07]

There's no value to using a PM951 as a SLOG device. It lacks power loss protection. It's better to just disregard sync writes and understand the possibility of failure exists rather than have a device that you're trusting to do a job that it cannot.

I will have a ups for whole system. Would that help?

 

[jgreco]

I will have a ups for whole system. Would that help?

Not really. It's just reducing the things that could cause damage.

 

[Dsky07]

Thanks jgreco. Which drive do you recommend for SLOG? Is Intel S3700 the only qualified?