TrueNAS Scale: vDev Design for PLEX, HA, NC

DigitalMinimalist

Contributor
Joined
Jul 24, 2022
Messages
159
Dear all,

I'm currently running Proxmox with Plex and HA (home Assistant) - never got NC (Nextcloud) running properly - on some old hardware (Z77, 2600K). It's time for a new built from scratch :)

I'm planning to switch to TrueNAS Scale OS and run NC, HA, PLEX from there!

First: easy topics
  • Case
    • minimum mATX mainboard
    • ATX power supply fits
    • I want to build a SFF Server and I think the Intertech IM-1 Pocket with 20L would do the job
  • Hard drives, quantity, model numbers, and RAID configuration, including boot drives:
    • OS: NVME boot drive (1TB Samsung 970 EVO - PCIe x3.0)
    • Cache useful? (I have a second 1TB Samsung 970 EVO - PCIe x3.0)
    • Storage Pool: 3x 16-18TB in ZFS pool
  • Hard disk controllers:
    • Mainboard SATA ports should be sufficient
  • Network cards:
    • PCIe 10GbE - probably X520-DA2
Second: more difficult topics:
  • RAM quantity:
    • 2x16GB, or 4x 16GB
    • DDR4 ECC, or DDR5 (is on-die ECC sufficient for ZFS?)
  • Motherboard make and model
    • mATX motherboard
    • AM4 (or wait for AM5 if DDR5 is beneficial) , or Intel socket 1700 (if Intel: W680 chipset to gain ECC support)
    • 4x RAM would be nice
    • 2x NVME
  • CPU make and model
    • 6c/12T with iGPU and ECC support
      • AMD Pro 5650G
      • Intel 12400/12500 (with W680 mainboard) - Quicksync might be useful if I would like to use Plex Hardware transcoding

Key questions from my end:
  • Which RAM path should I follow to have proper ECC for ZFS - DDR4, or DDR5?
  • Are there any hardware restrictions regarding TrueNAS scale OS (e.g. PCIe NIC chip)?

Thank you
Alexander
 

DigitalMinimalist

Contributor
Joined
Jul 24, 2022
Messages
159
After some further research

ECC = on-die ECC AND side-band ECC
Right now DDR5 with side-band ECC is announced, but not available yet

If you want to run ZFS properly:
DDR4 ECC, or wait for DDR5 ECC
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
Some thoughts regarding ECC:

The "on-die ECC", among other names that have been used to refer to a characteristic of DDR5, should really not be seen as a positive feature in of itself. It's really just an enabler for higher densities / lower cost via higher yields. Might it reduce error rates relative to DDR4 without ECC? Certainly possible, but that's not the design goal and you get zero visibility into any errors.
Right now DDR5 with side-band ECC is announced, but not available yet
Assuming they haven't done anything weird to that part of the spec, ECC DIMMs will just have 1/4 more capacity to be used by the host memory controller as it sees fit (i.e. to store ECC-encoded words into memory instead of the plaintext words) - it's more than the 1/8 of DDR4 due to having two 32-bit channels instead of a single 64-bit channel per DIMM. My point is that ECC DIMMs will be released in sync with platforms that support DDR5, as there's nothing particularly difficult about them. The major volume stuff will only be released in 2023, but maybe there might be some low-end server news from Intel in late 2022.
 

DigitalMinimalist

Contributor
Joined
Jul 24, 2022
Messages
159
I`m starting to consider to swap my Desktop Workstation with Threadripper 1920X & Gigabyte Designare EX & 64GB RAM to my new Homeserver instead of selling cheaply. No SFF and no ECC (if I don't buy new RAM) and probably high power consumption.

On the other hand it's probably just smart to delay my TrueNAS built to 2023 and hope for buyable DDR5 ECC and low end AM5 CPUs with iGPU & ECC support...
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
Or just buy a current Intel platform today (I'd say "or AMD", but they're light on less-than-big-server platforms, whereas Intel still has the low-end stuff available). There's always "more betterer" around the corner, so delaying based on that is often not a great option unless you specifically need a certain feature.
 

DigitalMinimalist

Contributor
Joined
Jul 24, 2022
Messages
159
Thank you for sharing your thoughts.

My wish for SFF (mATX) and low power consumption are the challenge here.
Originally, I wanted to go with Jonsbo N1 case (mITX), but the premium to get from 20L with mATX to 10L mITX is significant
 

DigitalMinimalist

Contributor
Joined
Jul 24, 2022
Messages
159
Some further readings later:

Platform
Closest to my requirement: Asrock Rack X470D4U (X570D4U if I want to run both NVMEs at full speed) with Ryzen 5650G and 32/64GB UDIMM ECC DDR4

DDR5
If I want DDR5, I have to wait until DDR5 ECC is available, as on-die ECC is not solving my data integrity needs

vDev Design
I learned that vDev Design is critical and I haven't thought enough about it.
I tend to go to 4x16TB as a striped mirror - space & integrity is still sufficient, but faster speeds are possible and tehre is the possibility to expand later with additional 2x16TB HDDs
OS: SSD for sure, but what size is needed for TrueScale? 128GB, or 512GB 2.5 SSD?
VMs: Mirrored 1TB NVME (Samsung 970 Evo)? I think VMs need the fastest IOPS, or?

Am I on the right track here?
 

DigitalMinimalist

Contributor
Joined
Jul 24, 2022
Messages
159
Some questions have answered themself, as I won an online auction yesterday for a server rack:

Platform 1
Asrock Rack X470D4U, 2700X, Noctua Cooler, 2x16GB ECC DDR4

Platform 2
2011-3 Mainboard, E5-2630L v3, 4x 8GB DDR4

Storage:
  • 2x HBA Controller
  • 3x SSD 4TB Samsung 860
  • 5x SSD 500GB Samsung 860
  • 2x SSD 500GB Samsung 850
  • 2x HDD 4TB WD Red Pro
  • 2x SSD 120GB SSD
  • 1x NVME 250GB Crucial
  • 1x NVME 500GB Samsung 970 EVO Plus

2 power Supplies (one 80+ Titanium)

Therefore I will shift my attention to the vDev Design with all these drives for platform 1:

My noob idea - please critique, propose better options:
  • OS: 1x 120GB SSD (Motherboard SATA)
  • VM: 2x NVME 1TB Samsung 970 Evo - mirrored (1x PCIe 3.0 x2 + PCIe 2.0 x4 - hopefully both are close to 2000 MB/s of bandwidth)
  • SLOG: SSD? (Motherboard SATA)
  • Storage (Motherboard SATA):
  • 4x 16TB CMR HDD:
  • 2x 16TB Mirrored and both mirrors striped - basically RAID10 equivalent
4TB SSDs go into my workstation to supply silent and fast storage for photo editing (instead of building fast NAS storage) -> Storage will hold photo library backup and movie library

1 have additional storage which I could utilize:
  • 1x SSD 128GB Crucial
  • 1x SSD 512GB Crucial
  • 2x NVME 1TB Samsung 970 Evo
  • 1x NVME 2TB Intel 660p (QLC)
  • 2x HDD 4TB WD Red
  • 2x HDD 2TB Samsung (very old)



How could a good vDev Design look like in terms of storage space, power consumption, data integrity and speed?
 

DigitalMinimalist

Contributor
Joined
Jul 24, 2022
Messages
159
Updated vDev Design - just ordered two (used) PM983
  • OS: 1x 250GB NVME SSD (I just have a few small SSDs left)
  • Asus Hyper M.2 V2 (4x PCIe 3.0 x4 via bifurcation)
    • VM: 2x NVME 1TB Samsung 970 Evo - mirrored
    • special vDev: 2x NVME 960GB Samsung PM983 (PLP) - mirrored
  • Storage (Motherboard SATA):
    • 4x 16TB CMR HDD:
      • 2x 16TB Mirrored and both mirrors striped - basically RAID10 equivalent
  • SLOG?
    • I have one NVME slot left on the Motherboard - no SLOG, or a 970 EVO 500GB? (if it dies - replace)
Thoughts about overall vDev Design & SLOG question?

Other Hardware updates:
  • (used) Intel X520-DA2 (OEM) ordered
  • Icy Dock 5x 3.5 backplane
  • Searching Case: searching for an older small mATX case with 3x5.25 drive bays (ATX size no problem)
  • Searching for additional 2x16, or 2x32 GB Samsung ECC UDIMM M391A2K43BB1-CTD
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
Thoughts about overall vDev Design & SLOG question?
Fine, overall, but I question the need for the SLOG. What will you be doing that would produce sync writes?
 

DigitalMinimalist

Contributor
Joined
Jul 24, 2022
Messages
159
Fine, overall, but I question the need for the SLOG. What will you be doing that would produce sync writes?
I think that I don’t need one - I just have an NVME drive and slot available :)
L2ARC: better increase RAM
Mirrored boot drive: not needed if I snapshot config regularly
SLOG is the only potential use case I can imagine (or leave blank)
 

DigitalMinimalist

Contributor
Joined
Jul 24, 2022
Messages
159
Additional noob question:

If I have 32TB “rust” storage space and 960GB special vDev:
How can I calculate or estimate the size limit for special vDev?
64kb sounds like a standard recommendation…
 

NugentS

MVP
Joined
Apr 16, 2020
Messages
2,945
With the special metadata - you need to remember that the metadata size really ought to be less than the record size otherwise all your files end up on the special vdev which will rapidly run out of room.

So first look at the record size for each dataset, picking something appropriate.
for example - Movies and videos = 1Mb with metadata of 4Kib or less
For VM's they should be lower and closer together.
Another example - I have a book library so use Record Size 128K, metadata 64KiB. A lot of my book library is on the special vdev. It helps keep the library snappy (well snappier)

I used: https://forum.level1techs.com/t/zfs-metadata-special-device-z/159954 when I had the bulk of my data on pool, worked out what the metadat size ought to be and then had to churn eacgh dataset to rewrite the data.
 

DigitalMinimalist

Contributor
Joined
Jul 24, 2022
Messages
159
Thank you for your input.

My main storage needs in terms of data type:
Movies: usually >500MB to 20GB
Photos: usually > 2MB to 50MB
Music: usually > 2MB to 50MB
Phone & tablet backups: I need to check
This will probably accountant 90% of storage needs

Metadata, documents should be on the SSDs and 960GB should be more than sufficient space.

I probably need to fiddle around with some testfiles during the initial setup phase.
 

NugentS

MVP
Joined
Apr 16, 2020
Messages
2,945
I use twin 800GB SSD's as a special vdev. It can go to 75% full I believe - but aim for 50%

Storing small files on the vdev is a bonus, but the most important part of the whole thing is storing the metadata on the special vdev. I only really store books and documents on the special.

BTW - the same effect (metadata on fast vdev) can be achieve with an L2ARC (Metadata only) if you are not concerned about storing small files on the vdev. L2ARC (metadata only) has an advantage that it stores a copy of the metadata - so you can use a single disk - its not pool critical. If the disk fails then ZFS shrugs and goes back to using the metadata on the rust. This doesn't store small files, but is safer.
 

DigitalMinimalist

Contributor
Joined
Jul 24, 2022
Messages
159
Thank you - I wasn't aware of the L2ARC option with metadata - sounds like a good option
I'm probably going to also store small files, as I already bought (in transit) 2x PM983 with 960GB and 960GB (even with max 75%) is way too much for just metadata.

Another question for vDev Design: Mirror vs. striped mirror vs. RAIDZ2
I was 90% sure with the striped mirror setup with 4x16TB and enough storage space for the coming years (at 14TB right now) as it increases speed, acceptable capacity efficiency and possibility to extend (additional mirror set) - but with energy prices going up and 18TB HDDs as the sweetspot, I started to consider 2x18TB as mirror as an option
Same "risk profile" than stripped mirror, but less speed.

I learned from you guys that 3x large HDD in RAIDZ1 isn't a good option, because if resilvering takes long and the zpool is at high risk during that time - understood
I understood that simple mirrors should be less "stressfull" for the HDDs vs. RAIDZ1/2/.. in case of resilvering - is there any estimation about how long it approx takes if you add a new HDD (h per TB?) until the new Mirror is "safe" again? Scenario - as mentioned above- 18TB

I understand how important it is to define the vDev Design according to my personal risk profile and speed needs as it is very difficult/impossible to change afterwards.

Too many opinions to choose from for me :)
 

DigitalMinimalist

Contributor
Joined
Jul 24, 2022
Messages
159
I just received my 2 used PM983 drives:

TBW: 545TB & 1260TB, which means one of the two drives is really close to the guaranteed TBW of 1360TB... shit
I really should have asked upfront for the TBW usage...

I won't use the second for sure - Would you use a a drive for special vDev with 545 of 1360TB written? If yes, i would like to try to get a "new" used PM983, or get rid of both and buy new?

Best alternative seems to be Transcend MTE220S 1TB NEW with 2.2PB, but no power loss protection
 
Last edited:

NugentS

MVP
Joined
Apr 16, 2020
Messages
2,945
Use it as an L2ARC (metadata only). Doesn't matter if it fails then. Dump the 1260TBW one (or put it in a USB to NVMe box and use it occaisonally for non NAS related things). Or if you bought it off ebay try and return it as knackered (in particular if the ad says in full working health)
 

DigitalMinimalist

Contributor
Joined
Jul 24, 2022
Messages
159
Dumped the Samsung P983 and switched to Micron 7300 Pro 960GB.

I’m still scouting for a 10GBE SFP+ PCIe card.

X520-DA1/2 (and it’s clones) or Mellanox are the obvious candidates.

What about Intel X710-DA2, X520-SR2?
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
Top