HDD DRIVES BUILD ADVICE

[zbyte]

I'm just building my 1st home NAS and planning to use TrueNAS.

My build:

• Node 804 with stock 2x120mm and 1x140mm fans
• Ryzen 3700X with Noctua NH-U12S SE-AM4
• Asrock X570D4U-2L2T with added Noctua NF-A4x20 PWM for the X570
• Superflower 850W Leadex III 80 Plus Gold
• 64 Gb ECC RAM
• 1x Samsung 960 PRO M.2 1TB
• HBA Avago LSI 9400-8i
• 4x WD40EZRX 4TB Green (coming from another NAS which was used for about 2 - 3 years)
• 1x 12TB HGST (HUH721212ALE600)
• 2x 6TB HGST (HDN726060ALE610)
• 2x 4TB HGST (HDN724040ALE640)
• 4x 4TB DELL (ST4000NM0033) (Seagate rebranded) (if I can avoid using these it would be great, I have these on a Dell 720 which not used now but would like to use it in the future perhaps)

OS will be ESXi 7.0, TrueNAS will run on a VM out of the M.2 disk.
Another VM will run Plex server on M.2
Another VM will run SQL server on M.2
NOTE THIS BOARD HAS 2x 10GBe INTERFACES WHICH I'M PLANNING TO USE

Planning on using 8 HDD drives only for music, movies and files, so just NAS purpose no VM's, don't know how will I distribute the drives yet that's why I would like to ask here. Based on what I read on the post there are several options.
My current storage requirements are around 10 - 12 TB

1. I can use just one VDEV and ZFS pool with 8x 4TB drives in RAIDZ2 totaling 24 TB, this will be slow, although for NAS could be ok, still if I want to increase size or performance I will be limited in the future so I'm not sure about this.
2. I can created 2 RAIDZ2 VDEVs with 4 drives each providing 4x 4TB and 4x4TB totaling 16 TB of usable space, this will be a little faster and with options to more easily increase one VDEV on size in the future with larger drives but not so sure about it either.
3. There is an option of using mirrored VDEV with 2 drives each, this seems to me the most logical case it will be 4 VDEVs each of 2x 4TB providing 12 TB total, I can also use the 2x 6TB on one of the VDEVS giving me 14 TB total, if I need to increase space I just need to purchase 2 drives of larger size and upgrade one of the VDEVs, possibly the mirroring set will provide more performance
4. A final option would be purchasing 4x 8TB WD RED PLUS drives which is $800 or so and could provide some extra storage but at this point I don't think I need it.

I understand the mirrors only tolerates one drive failure not two drives vs the RAIDZ2, I also understand the mirrors will rebuild / resilver faster than the 4 drives VDEVs, I am planning to keep backups of all the data on larger drives.

Not planning to use ZIL or L2ARC at the moment unless the performance over 10 GBe is limited (which I'm kind of forecasting will happen after reading these excellent posts on this site).

Would love to hear your comments or recommendations regarding my build and especially which ZFS pool to use. Thank you in advance

 

[Etorix]

You may be short of one or two case fans to provide one or two intake fans and one exhaust fan in both the CPU side and the HDD side.

There are ten possible positions for 3.5" HDD: 8 hanging, 2 on the floor of the CPU side. So with mirrors (option 3), you can have up to 2*6TB + 4*(2*4TB) = 22 TB.
With 4-wide Z2 (option 2), which is indeed safer, you could do (12+2*6+4) and 4*4TB. 16 TB, but by upgrading one single drive it would grow to 20 TB.
Both these are better than option 1 if you plan to progressively grow the pool and cost less than option 4.

 

[jgreco]

OS will be ESXi 7.0, TrueNAS will run on a VM out of the M.2 disk.

Be sure to have read

https://www.truenas.com/community/t...ide-to-not-completely-losing-your-data.12714/

A lot of work has gone into identifying the usual pain points you may run up against.

I understand the mirrors only tolerates one drive failure not two drives vs the RAIDZ2,

Only correct for two-way mirrors. Three or more-way mirrors will tolerate N-1 failures.

Not planning to use ZIL or L2ARC

ZFS doesn't give you a choice about ZIL. You *always* have (and use) ZIL. It is a requirement for the copy-on-write system to avoid something akin to the RAID5 write hole.

See:

https://www.truenas.com/community/threads/some-insights-into-slog-zil-with-zfs-on-freenas.13633/

The usual error newcomers and/or people who've been misled by random Internet posts, videos, etc. with misinformation about this "ZIL" thing make, is that they mean they're not planning on a SLOG device to give them a *fast* ZIL. That's fine of course. You will just have a slow ZIL, not no ZIL, and you *will* be using it.

We do not suggest L2ARC until your FreeNAS system (not the host, the actual VM) has a minimum of 64GB RAM. It can maybe be beneficial with somewhat less RAM in certain situations, but ZFS really needs a large ARC to be able to do a good job of analyzing which blocks are best for eviction to L2ARC. With only 64GB in your host and lofty goals of Plex and SQL running alongside FreeNAS, you are not likely to be committing sufficient RAM to FreeNAS, and so L2ARC is unlikely to work well.

 

[zbyte]

You may be short of one or two case fans

Yeah possibly I will be adding one more fan on the HDD side, front intake, the CPU side with the Noctua and the one intake one exhaust is fine

With 4-wide Z2 (option 2), which is indeed safer, you could do (12+2*6+4) and 4*4TB. 16 TB, but by upgrading one single drive it would grow to 20 TB.
Both these are better than option 1 if you plan to progressively grow the pool and cost less than option 4.

The HBA has only capacity for 8 drives, I can't use the SATA onboard ports (which was my 1st attempt) as ESXi won't work properly passing thru the SATA on this specific board. I am also limited in slots to use cards and amount of lanes when I populate these slots.

I have to keep my options at 8 or less for now. Maybe next iteration of ESXi will support the onboard SATA in which case I could add VDEVs

I don't understand the part where upgrading "one single drive" would give me more space, which scenario would that be?

Thank you for the answers

 

[jgreco]

An HBA can definitely support more than 8 drives. Look for an inexpensive used SAS expander. They can be placed elsewhere in the chassis as long as there is space.

See the SAS primer for details.

https://www.truenas.com/community/t...-sas-sy-a-primer-on-basic-sas-and-sata.26145/

 

[zbyte]

Thank you @jgreco for providing feedback

Be sure to have read

Yes I read it completely, thank you

Three or more-way mirrors will tolerate N-1 failures.

Good to know, I can't apply it with 8 drives, if I could use 9 drives then would be practical for me.

ZFS doesn't give you a choice about ZIL. You *always* have (and use) ZIL. It is a requirement for the copy-on-write system to avoid something akin to the RAID5 write hole.

I'm sorry, I did not mean not using it, poor choice of words, I was under the impression I could provide enough RAM (32 GB) or so for TrueNAS and ZIL will be stored in RAM but not on dedicated storage for this. Let me read your link, it might better be on enterprise storage with power loss protection than RAM.

We do not suggest L2ARC until your FreeNAS system (not the host, the actual VM) has a minimum of 64GB RAM. It can maybe be beneficial with somewhat less RAM in certain situations, but ZFS really needs a large ARC to be able to do a good job of analyzing which blocks are best for eviction to L2ARC. With only 64GB in your host and lofty goals of Plex and SQL running alongside FreeNAS, you are not likely to be committing sufficient RAM to FreeNAS, and so L2ARC is unlikely to work well.

Better plan for 128 GB then from the beginning

 

[zbyte]

can definitely support more than 8 drives. Look for an inexpensive used SAS expander

Something like the RES2SV240 you mentioned on your post? can leave the card "securely dangling" somewhere inside the chassis providing power to it with a Molex connector, one of the cables from the LSI HBA will go into the Intel Expander and then breakout cables to the disks
Something like this

https://ipsassets.unraid.net/uploads/monthly_2017_11/5a1c0955774a3_2017-08-2413_30_07.jpg.45000eea88b0f7f379fc5a77c22cf54c.jpg

 

[Etorix]

I don't understand the part where upgrading "one single drive" would give me more space, which scenario would that be?

A RAIDZ2 vdev with 12 TB, 6 TB, 6 TB and 4 TB would be limited to the size of the smallest drive, i.e. 4*4 TB, giving 8 TB of storage (which should not be filled more than 75-80%, else you can forget about any kind of performance). If you then replace the 4 TB HDD by a 6 TB or larger drive, the vdev will automatically expand to 4*6 TB, giving 12 TB of storage.
The 12 TB HDD would still be vastly underused, but since you have it and want some capacity, putting it in a vdev with the second largest drives you have is the best way to use (some of) its capacity.

 

[zbyte]

A RAIDZ2 vdev with 12 TB, 6 TB, 6 TB and 4 TB would be limited to the size

Thanks I get it, didn't thought on that option initially mixing drives of different sizes.

From the comments I gather that possibly a 2x 4drives (8 total drives) RAIDZ2 vDev's would be recommended over the 4x 2way mirrors.

Following the SAS expander advice I could add a total of 10 drives allowing for 2x 5drives vDevs 24 TB total at 50% capacity, typically I'd like to stay away from slow rebuilds / resilvering in case of drive failure.

I understand you could expand a vDev once created, for example a 4 drive vDev expanded to 5 drives, that could allow me to start with 8 drives and add 2 more in the near future.

thank you for your feedback.

 

[Etorix]

I understand you could expand a vDev once created, for example a 4 drive vDev expanded to 5 drives, that could allow me to start with 8 drives and add 2 more in the near future.

No, you got this wrong. The geometry of a RAIDZ vdev is set at creation and cannot be changed later. A 4-wide RAIDZ2 vdev cannot take additional drives and can never grow to 5-wide, 6-wide, etc.
You can however connect a new drive and replace an exiting drive with the new one. The old drive may temporarily remain in the vdev during resilver (this is the safest and fastest method), but once resilver is finished the old drive will go offline; the vdev will not be wider.
Mirrors can expand from 2-way to 3-way, 4-way… but this does not increase capacity, only redundancy.

To increase capacity, you have to add new vdevs (preferably of the same type as the previous ones, so mirrors in a pool of mirrors, RAIDZ2 in a pool of RAIDZ2; not necessarily of the same width as the previous vdevs, but the pool will only be as performant as the least performing vdev and as safe as the least redundant vdev), or replace all members of a vdev by larger drives.

If you go for mirrors (most perfoming for many clients), you can begin with 8 drives in 4 mirrors and later move to 10 drives in 5 pairs. If you go for RAIDZ2 (safest, and probably of sufficient performance for home use), the next step from 8 drives in 2 vdevs is 12 drives in 3 vdevs.

Out of curiosity, why did you chose to virtualize everything in ESXi rather than running Plex and SQL as jails/VM under TrueNAS? There would be no issue with passing drives controllers.

 

[zbyte]

No, you got this wrong. The geometry of a RAIDZ vdev is set at creation and cannot be changed later

Thank you for correcting me, I read an old post from 2019 that apparently expansion of the vDev was expected to be released but you just confirmed this hasn't happened yet.

Out of curiosity, why did you chose to virtualize everything in ESXi rather than running Plex and SQL as jails/VM under TrueNAS? There would be no issue with passing drives controllers.

Yeah I know, ESXi is a requirement, I recently dismantled my servers from a datacenter and I have about 8 servers, mainly Dells 420's, 710's and such, moved from my house to a townhome so running the servers it is not really an option for me because of noise mainly, decided to build a little more efficient NAS with less noise, I need to run firewalls virtual appliances, one option is cloud of course but many of them and their reporting counterparts comes natively in VMware format .ova hence my dilemma. If that wouldn't be a requirement I would have run jails under TrueNAS and wouldn't need to virtualize it.

 

[zbyte]

I have a question about mixing SAS and SATA drives on the main controller, I would keep all SATA but it seems good drives can be purchased cheaper being these SAS. I have read about the subject here but I don't really grasp the entire concept. The electrical parameters are different for SAS and SATA and seems like mixing SAS and SATA on the same connector might bring some problems.
So I have this card AOC-S3008L-L8E, Supermicro 9300-8i like controller, with Supermicros IT firmware.
My intentions were to get 4x 12TB SAS HGST drives and connect these 4 SAS drives to a connector on the HBA, the second connector of the HBA would have 4x SATA drives OR one Intel E91267-203 expander with 4 to 6 SATA drives.
Would all these scenarios work? I read the smartctl reporting is kind of incomplete with SAS drives?
Thank you in advance

EDIT: May I add my intention is to create a 4 wide RAIDZ2 with the SAS drives and 4 to 6 wide RAIDZ2 with the SATA drives and of course a pool with both vDevs

 

[Etorix]

@jgreco wrote a ressource:
Don't be afraid to be SAS-sy ... a primer on basic SAS and SATA

 

[Arwen]

@zbyte It should not mater which connector or port a SAS or SATA drive is connected to a SAS controller. Basically the SAS standard detects SATA drives on a port, and turns the port into a SATA port. But other ports on the same cable or connector can be either SAS or SATA.

Just be mindful of the cable length limits. SATA drives have reduced cable lengths when compared to SAS drives. Even when SATA drives are wired to a SAS controller, (because that port is now a SATA port).

Last, when using a SAS expander, (which works very well, compared to SATA port multipliers, which are NOT recommended), any drive port can be either SAS or SATA. And all the host ports are SAS, (with any SATA drives' data being tunneled over SAS).

 

[zbyte]

Don't be afraid to be SAS-sy ... a primer on basic SAS and SATA

Thank you, I read that one before posting, however per your link I re read it again and can't find what I'm looking for on that thread, I understand the signaling part and the connector part SAS vs SATA on that thread but not the exact application with TrueNAS.

Apparently it doesn't matter and I can even use on one SAS HBA 3 drives SAS with 1 drive SATA and possibly even mix these 4 drives on a vDev but it is the TrueNAS specific support for the mix what I'm looking for to understand before implement and know if this is recommended, I am not even asking if it is possible, it could well been for all purposes but is it recommended or not?

@zbyte It should not mater which connector or port a SAS or SATA drive is connected to a SAS controller. Basically the SAS standard detects SATA drives on a port, and turns the port into a SATA port. But other ports on the same cable or connector can be either SAS or SATA.

So it really doesn't matter I can just mix as SATA will be tunneled over (or under for semantics :) SAS.

Can I mix SAS and SATA on the same vDev? would the TrueNAS reporting smartctl work just the same for S.M.A.R.T. tests and info?

Thanks

 

[Arwen]

...
So it really doesn't matter I can just mix as SATA will be tunneled over (or under for semantics :) SAS.

Can I mix SAS and SATA on the same vDev? would the TrueNAS reporting smartctl work just the same for S.M.A.R.T. tests and info?

Thanks

Yes. Technically unless you are using a SAS expander, SATA drives on a SAS host port, aren't tunneled. The SAS controller simply turns the host port into a SATA port. Automatically.

Mixing SAS & SATA on the same vDev will work fine as far as ZFS is concerned. As long as they are the same rough sizes, you don't loose much space. It is preferable that all the drives in a vDev have the same rotational speed. And if not, (like my current NAS), you understand that each vDev will perform as fast as the slowest disk.

I don't know about SMART tests on SAS. But SCSI, (which is the protocol used by Serial Attached SCSI), has had drive statistics and such for close to 35 years.

 

[zbyte]

The SAS controller simply turns the host port into a SATA port. Automatically.

Thanks, I guess conceptually the hardest part to understand is that on the same HBA port you could have SATA and SAS coexisting, so now I know that it doesn't matter and I could use and mix SAS and SATA as long as the breakout cables and connectors are supported.

Thanks

 

[jgreco]

Thanks, I guess conceptually the hardest part to understand is that on the same HBA port you could have SATA and SAS coexisting, so now I know that it doesn't matter and I could use and mix SAS and SATA as long as the breakout cables and connectors are supported.

Thanks

If by "HBA port" you mean an SFF-8087 connector, you've misunderstood what an SFF-8087 is. It is an electrical connector that carries four separate channels or ports or whatever you would like to call an SAS lane. It is not any sort of "port" itself, except insofar as people often find it convenient to use four parallel SAS lanes to connect up backplanes.

 

[zbyte]

If by "HBA port" you mean an SFF-8087 connector, you've misunderstood what an SFF-8087 is. It is an electrical connector that carries four separate channels or ports or whatever you would like to call an SAS lane. It is not any sort of "port" itself, except insofar as people often find it convenient to use four parallel SAS lanes to connect up backplanes.

Thank you, yes I kind of misunderstood but now it is clear, 4 SAS lanes on just one physical connector (SFF-8087), the connector is nothing else than an aggregated connector with 4 lanes in it, these SAS lanes could have attached SAS drives (I guess you could say natively) or SATA drives on a SAS lane in which case the SAS lane automatically will handle the SATA drive.

 

[jgreco]

That is correct.