New Build - HDD Setup Discussion

Ryan Hansen

Cadet
Joined
Feb 23, 2017
Messages
4
Hi All,

New to the thread, read lots so far - much more to read yet!

I've finally put together my 1st TrueNAS build, just finished installing (bare metal)TrueNAS Scale today!

My Build: (Very likely overbuilt - but I didn't want to limit the system, or age it too soon either)
Case - Fractal Node 304
CPU - AMD Ryzen 9 5900X 12-Core Processor
CPU Cooler - Noctua NH-U14S, Premium CPU Cooler with NF-A15 140mm Fan
Motherboard - Gigabyte A520i AC
Graphics Card - Nvidia Tesla P4 with a 3d printed fan mod
RAM - (2x) Crucial 16GB DDR4 2666MHz PC4-21300 2RX8 CL19 ECC UDIMM Memory CT16G4WFD8266
PSU - 430W EVGA
HDD - (4x) Western Digital Red Pro 12TB NAS Hard Drive, WD121KFBX
Future HDDs - (2x) 22TB WD Red Pro as back-up drives for the above 4 drives.
Boot Drives - (2x) 256GB Samsung MUF-256AB/APC Fit Plus USB Drives - plugged into the USB 3.0 ports. These drives apparently are Samsung V-NAND - hopefully that helps in the reliability side of things?

I have a few questions regarding the configuration of everything now - I'd rather not learn the hard way. So here is what my thoughts are - please stop me if there are better ways of configuring this.

I will be using this as a headless TrueNAS computer for the following:

PLEX Server, Radarr, Sonarr, Home Assistant, NextCloud, Pi-hole, AdGuard(maybe), Jelly Fin, Tdarr, Lidarr, SABNZBD (can't see it in the app section however - but being Linux based, I'm sure I can figure something out), and possibly others as the system grows. I don't forsee virtualization happening (not even sure my CPU supports it). This will be for under 6 people. I'd like to set up my movie and music library - back-up and store all my photos and work documents (from OneDrive and Google Drive).

My thoughts for how I'd set up the HDDs is striped mirror (2x) 12tb drives, then again stripe mirror the other two 12tb HDDs. From here, I haven't figured out if I should leave them as two seperate pools, or try to combine them? As for back-up - I'm looking at purchasing two 22TB drives and backing up to those. I haven't figured out how back-ups work in TrueNAS yet - so I'm open to comments on how to set this up correctly.

I still have my m.2 slot available in my motherboard, not sure what I'll use that for yet - maybe upgrade to faster LAN (currently 1gbe) or sata expansion module.

I heard mixed reviews about SLOG and L2ARC. But from what I've read so far, I should steer clear of these and don't dedicate my m.2 to SLOG or L2ARC.

Just going through the TrueNAS settings and searching each setting configuration trying to understand what each one does and the desired setting (this will take a while). Then there is

I'm excited to figure out my system and research as much as I can - thought I'd ask the experts to weigh in on their thoughts as I go along.

Cheers!
 

Ryan Hansen

Cadet
Joined
Feb 23, 2017
Messages
4
Also, I understand my motherboard has realtek lan, although frowned upon in the forum as there have been issues with realtek historically. I'm currently not seeing any issues with my connection - is this due to the new 'FreeNAS Scale' (linux based) providing additional compatibility - or is this an issue that will eventually happen? I could remove my M.2 A+E wifi card and replace is with a M.2 A+E Intel chipset NIC from Amazon - just not sure it's worth it just yet?
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
is this due to the new 'FreeNAS Scale' (linux based) providing additional compatibility

No. In general, Linux does not provide "additional" compatibility. What happens is that hobbyist users have wider access to a wider range of knockoff and obscure gear, and they try to make Linux work with it. That might sound like "additional compatibility" but in many cases it works out to undermining what is already there in the Linux system in order to make their thing work, while damaging performance with other stuff. There are a number of examples of off brand ethernet controllers that work better on FreeBSD than on Linux.


or is this an issue that will eventually happen?

I've given you the paragraph answer above to help frame my answer to this. What may happen, possibly, it's hard to predict the future, is that at some point some other Shenzhen back alley knockoff silicon stamper may create a Windows compatible knockoff of a venerable Realtek design, and in doing so, it may not be quite perfect. One day, some Linux driver geek might be asked to "make it work" and may make driver changes that inadvertently impact your card while trying to make the new thing work out.

In my opinion, the days of this happening for a Realtek chipset is unlikely. People have moved on from the 1G stuff (which is terrible and tragic, but this isn't being seriously evolved further) to the 2.5G stuff (where substantial hackery in the future is absolutely a possibility).

So you have two options as I see it:

1) Keep running the TrueNAS firmware you are running today. This will keep working the way it works today, unless your Realtek chip lights on fire or something like that. But no driver change equals "should keep running the same way".

2) Wait for something bad to happen. Why spend money on an Intel upgrade if you're happy with the way it is working? Some people want or need an SUV to get around. If you are fine in a VW Bug, though, by all means, keep it up until the thing maybe fails someday. By that time maybe there will be better options available for purchase.
 

Davvo

MVP
Joined
Jul 12, 2022
Messages
3,147
My thoughts for how I'd set up the HDDs is striped mirror (2x) 12tb drives, then again stripe mirror the other two 12tb HDDs. From here, I haven't figured out if I should leave them as two seperate pools, or try to combine them? As for back-up - I'm looking at purchasing two 22TB drives and backing up to those. I haven't figured out how back-ups work in TrueNAS yet - so I'm open to comments on how to set this up correctly.
2 vdevs in a pool or two pools it depends on what you want to do; backup could be as simple as zfs replication.
Also do note that TN does not support WiFi.
 
Last edited:

Etorix

Wizard
Joined
Dec 30, 2020
Messages
2,110
My Build: (Very likely overbuilt - but I didn't want to limit the system, or age it too soon either)
Case - Fractal Node 304
CPU - AMD Ryzen 9 5900X 12-Core Processor
CPU Cooler - Noctua NH-U14S, Premium CPU Cooler with NF-A15 140mm Fan
Motherboard - Gigabyte A520i AC
Graphics Card - Nvidia Tesla P4 with a 3d printed fan mod
RAM - (2x) Crucial 16GB DDR4 2666MHz PC4-21300 2RX8 CL19 ECC UDIMM Memory CT16G4WFD8266
PSU - 430W EVGA
HDD - (4x) Western Digital Red Pro 12TB NAS Hard Drive, WD121KFBX
Future HDDs - (2x) 22TB WD Red Pro as back-up drives for the above 4 drives.
Boot Drives - (2x) 256GB Samsung MUF-256AB/APC Fit Plus USB Drives - plugged into the USB 3.0 ports. These drives apparently are Samsung V-NAND - hopefully that helps in the reliability side of things?
Oversized CPU, suboptimal motherboard (not server, Realtek NIC, 4 SATA ports while you want 8 or more for the case, one single PCIe slot!), undersized PSU for future growth (650-750 W)

PLEX Server, Radarr, Sonarr, Home Assistant, NextCloud, Pi-hole, AdGuard(maybe), Jelly Fin, Tdarr, Lidarr, SABNZBD (can't see it in the app section however - but being Linux based, I'm sure I can figure something out),
SCALE is not "a Linux distro", and apps are run by Kubernetes, not regular Docker (which will, or has already been removed).

My thoughts for how I'd set up the HDDs is striped mirror (2x) 12tb drives, then again stripe mirror the other two 12tb HDDs. From here, I haven't figured out if I should leave them as two seperate pools, or try to combine them?
Wrong terminology for striped here. You stripe vdevs to make a pool. Disks in a mirror are… mirrored.
I see little reason to make two distinct pools with the same geometry over one stripe of two mirrors (=one pool). But a 4-wide raidz2 would be safer, though with less IOPS and less flexible to extend.

I still have my m.2 slot available in my motherboard, not sure what I'll use that for yet - maybe upgrade to faster LAN (currently 1gbe) or sata expansion module.
That would require an adapter… Can you send back the motherboard and get something more suitable? With 8 SATA ports, two or more PCIe slots which are x4 or more, and an Intel NIC.

I heard mixed reviews about SLOG and L2ARC. But from what I've read so far, I should steer clear of these and don't dedicate my m.2 to SLOG or L2ARC.
Correct. Likely no use case for a SLOG; not enough RAM for L2ARC.
 

Ryan Hansen

Cadet
Joined
Feb 23, 2017
Messages
4
Oversized CPU, suboptimal motherboard (not server, Realtek NIC, 4 SATA ports while you want 8 or more for the case, one single PCIe slot!), undersized PSU for future growth (650-750 W)


SCALE is not "a Linux distro", and apps are run by Kubernetes, not regular Docker (which will, or has already been removed).


Wrong terminology for striped here. You stripe vdevs to make a pool. Disks in a mirror are… mirrored.
I see little reason to make two distinct pools with the same geometry over one stripe of two mirrors (=one pool). But a 4-wide raidz2 would be safer, though with less IOPS and less flexible to extend.


That would require an adapter… Can you send back the motherboard and get something more suitable? With 8 SATA ports, two or more PCIe slots which are x4 or more, and an Intel NIC.


Correct. Likely no use case for a SLOG; not enough RAM for L2ARC.

Thanks everyone for the feedback thus far - quick questions regarding IOPS - when/how does one decide IOPS vs Raidz2 - I do not plan on extending at this point in time (or in the foreseeable future).

I've read in this forum that people 'wished' they didn't go Raidz2 and switched to striped - I just don't want to get it wrong on my first go and have to do it again!

Probably comes down to each persons use case and tolerance to redundancy/speed/etc. But surely, there's some expert advice for my above stated use case scenario for this build.

Thanks!
 

Etorix

Wizard
Joined
Dec 30, 2020
Messages
2,110
It's really down to your own priorities.
For capacity, go 8-wide raidz2 from the start. 75% space efficiency. Replace all drives by larger ones to extend.
For flexibility, go for mirrors: Can add, or remove, vdevs at any time; alternatively, it takes only to replace two drives at a time to extend without adding drives. But resiliency is not that great (you'd need THREE-way mirrors, which you can do in this case with a drive on the floor of the CPU chamber) and space efficiency is only 50% (33% with 3-way).
In between, 4-wide (or 5-wide) raidz2 gives the highest resiliency and moderate flexibility (can add one more vdev, can replace drives in set of four) with the same 50% (60%) space efficiency.
 
Last edited:

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
I've read in this forum that people 'wished' they didn't go Raidz2 and switched to striped - I just don't want to get it wrong on my first go and have to do it again!

So basically the way this works out is how you're going to use this.

You seem to have a bunch of Usenet and torrenting apps listed.

If your media collection was simply a bunch of ISO's where you purchased DVD media, ran Handbrake on the DVD, and wrote it out to a library collection, then were continuously serving it up 24/7 via RAIDZ to six different endpoint users, that would be highly compatible with RAIDZ. By purchasing media, there's an effective limit to just how much ingest flow you have, and how much fragmentation would appear in the pool. It doesn't really matter that you've continuous read activity because DVD and Handbrake work together to limit the practical bit rate. Maybe 5 megabits? So you aren't hitting the pool particularly hard. Every several seconds, the pool has to execute a seek-and-read for each reading client, and there is little ongoing fragmentation activity because presumably you only puchased DVD's that you really want to be in your media library, and your library grows organically with stuff you really want. RAIDZ is really good at that.

If, on the other hand, you just go on an uncontrolled download-a-thon of every full bit-rate binary in every Usenet group your provider carries, or you try to download everything you can find on torrents, your pool may rapidly fill with crap, which means that you will also need to remove stuff at some point, which will cause fragmentation. Because you're ingesting crap, maybe at speeds of 200Mbps or even 1Gbps, you presumably have to process, sort, and assemble those bits, which favors mirrors. Many people do this "jail work" on a dedicated SSD pool to make this more practical. But either way, if you are going with UHD Blu-Rays, your much higher bit rate (up to 120Mbit/s) makes it harder for RAIDZ to sustain traffic to several simultaneous clients on the backend while also handling the write/ingest of traffic flooding in from the Internet. This favors mirrors, at least for the software tools, possibly for the entire media collection.
 

MrGuvernment

Patron
Joined
Jun 15, 2017
Messages
267
Boot Drives - (2x) 256GB Samsung MUF-256AB/APC Fit Plus USB Drives - plugged into the USB 3.0 ports. These drives apparently are Samsung V-NAND - hopefully that helps in the reliability side of things?

Do not install TrueNAS on USB drives.
 

Davvo

MVP
Joined
Jul 12, 2022
Messages
3,147
Even if they are V-NAND (practically a SSD)?
High endurance USB sticks are good. You still have to worry about them being external and as such potentilly disconnecting.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
Even if they are V-NAND (practically a SSD)?
Well, that's an interesting question. The straight answer to your question is that the type of NAND is the least of your concerns. However, it is legitimate to use some USB devices for boot in some situations:
  • Which USB devices? Well, think SSD rather than pen drive. Where that specific Samsung SSD falls is something I'll leave for someone else to research. You definitely need something that has a proper SSD controller (typically NVMe these days, though I think there are some dual-mode USB/Thunderbolt controllers that don't need a separate NVMe controller to speak to the NAND).
  • If you have M.2 slots, it's hard to beat the convenience and neatness of a cheap (think WD Blue) NVMe SSD thrown in there. Some USB devices can approach this sort of setup, but the cost tends to be much higher.
  • If you have a choice between adding an HBA or using a fancy USB SSD for boot, the USB SSD starts sounding a lot better through lower power consumption, similar cost (USB device vs. HBA + SATA SSD) and just fewer cables flying around.
 
Top