Recommended Setup for All-SSD Server With This Specific Hardware?

[pr0927]

Hi all, posted to Reddit and didn't get much assistance. Figured I'd come here. I'm new to TrueNAS Scale, and I've recently built a new server and will be using it to replace my current main UnRAID server. I will later wipe and then repurpose that server for backup (Duplicati?) and Frigate NVR duties (currently a separate UnRAID server - trying to consolidate a bit) - also with TrueNAS Scale.

I currently self-host Plex (will be moving to Jellyfin), Nextcloud, FreshRSS, Vaultwarden, PhotoPrism, and Wallabag.

I also use VMs for Home Assistant and a Windows 11 system that has a passed-through RTX 3080 for living room gaming.

The UnRAID server has a 1TB cache M.2 NVME SSD, 4x 8TB WD Red Plus 3.5" HDDs, a 500GB M.2 NVME SSD (passed-through for the Windows 11 VM), and 64GB DDR4 RAM. CPU is a Ryzen 3700X.

Plex transcoding was handled by a GTX 1080.

For the new one - I wanted to go all-SSD, maintain the same services, and broaden my abilities for VMs. I acquired:

* 4x 8TB Intel DC P4510 NVME Drives (2.5" U.2 - NVME)
* 2x 1TB Seagate Firecuda 530 (M.2 - NVME)
* 2x 240GB Samsung PM893 (2.5" - SATA)
* 64GB DDR5 RDIMM ECC RAM

The dual SATA drives are the boot ones - mirrored. I will be putting the Intel drives into a RAIDZ2 pool for the main data pool.

Plex transcoding now to be handled by a Quadro P400 (lord knows I needed to reduce/shrink the amount of stuff/heat/power in there - 10GbE NIC was overheating). CPU is an EPYC 9124 (yes, I know).

I'm not certain what's the best use of the two M.2 drives - dedicated VM pool? Or dedicated pool for app config/install location? Mirrored? Striped? On the previous UnRAID server I used the 1TB cache for "appdata" and the Home Assistant VM (and as mentioned, the Windows 11 one had a dedicated 500GB SSD passed-through). Or one of those specialized L2ARC cache or special metadata pools?

Done buying equipment, so not looking for more drives - just trying to make the smartest decisions with what I've got.

Super appreciate what input could be thrown my way!

 

[pr0927]

Oh I should add - I've also thought about doing RAIDZ1 for the 4x 8TB Intel SSDs - my hesitation is paranoia of more than one failure - I have 2 parity drives in UnRAID.

But - I also have external drive backups of that data - a lot of them...(2 at my SIL's apartment as "off-site") and I'll be repurposing the existing main UnRAID server, as mentioned, with the 4x 8TB WD HDDs, as a backup server.

So perhaps RAIDZ2 then becomes unnecessary? RAIDZ1's increased storage access does seem very enticing (currently have roughly 70% used of 16TB array).

 

[Constantin]

It should be blazing fast. Those are good data drives as well. I'd use the two Samsungs for the VM in a mirrored pool, more than enough space for a few of them and plenty quick. I see no mention of the motherboard, case. What are you doing to mitigate your previously-noted heat issues?

 

[pr0927]

It should be blazing fast. Those are good data drives as well. I'd use the two Samsungs for the VM in a mirrored pool, more than enough space for a few of them and plenty quick. I see no mention of the motherboard, case. What are you doing to mitigate your previously-noted heat issues?

Appreciate the quick reply! Do you mean the Segates or the Samsungs? I already used the Samsungs (SATA 240GB) for the OS, mirrored - but the Seagates, NVME M.2, are the big question. I was leaning them for a VM pool, if that's what you meant - would it be wise to also use them for where apps get installed (the ix-applicationa folder, etc.)?

Motherboard being used is a Supermicro H13SSL-NT. Heat problems resolved - it was just the dual 10GbE NIC - moved a GPU to a different slot, got a half-height 4-port U.2 card (bifurcation supported on motherboard and working great) for better airflow, and strapped a USB fan to the back side to suck air away. That said, I did get a SFP+ NIC card anyway - they run much cooler than 10GbE stuff (and my switch has appropriate ports).

 

[Constantin]

Though used RJ45 10GbE cards have come down dramatically in price, SFP+ along with a DAC or optics are the way to go - less heat, lower cost, slightly more drama re transceiver selection.

If you have good off-site storage, then z1 array with the p4510s likely is ok, especially if these drives are mostly WORM, as it appears by your use case. I’m more paranoid (z3) but then I also have a lot of spinning rust, not SSDs.

I’d make a point of backing everything up. It’s easy to forget just how much time and effort goes into setting up a TrueNAS until your CPU goes kaploink and your next week flashes by in your mind. Save those configs and use ZFS send with ssh+netcat to make those backups as quick and comprehensive as possible.

 

[Constantin]

Firecuda, etc use for apps, etc seems ok, especially if they’re in a mirrored pool, ditto proposed boot pool.

 

[pr0927]

Though used RJ45 10GbE cards have come down dramatically in price, SFP+ along with a DAC or optics are the way to go - less heat, lower cost, slightly more drama re transceiver selection.

LMAO this cracked me up. Too true. But all's well, I got a X520-based NIC, using a DAC - working fine. Will probably just disable the two 10GbE LAN ports since they're generating heat anyway.

If you have good off-site storage, then z1 array with the p4510s likely is ok, especially if these drives are mostly WORM, as it appears by your use case. I’m more paranoid (z3) but then I also have a lot of spinning rust, not SSDs.

Yeah won't be quite WORM - downloads will be written to it, and modification to files on Nextcloud, etc. But definitely substantially more read than write. That said, and this is coming from someone who generally errs on the side of paranoia, the more I think about it, due to the factors mentioned earlier, the more comfortable I feel with RAIDZ1.

I’d make a point of backing everything up. It’s easy to forget just how much time and effort goes into setting up a TrueNAS until your CPU goes kaploink and your next week flashes by in your mind. Save those configs and use ZFS send with ssh+netcat to make those backups as quick and comprehensive as possible.

Yeah this is where I have a lot to learn - never setup anything like this before - just seen videos of people setting up resilio-sync and whatnot.

But much thanks, this has been helpful!

 

[Constantin]

No need for reselio sync or whatever.

First, get your Concorde of a TrueNas set up, including scrubs and snapshots. Set up your older system as a TrueNAS, then use replication (under tasks) which uses zfs send. That in turn requires a SSH setup between systems (easy to implement via system submenus).

Once you get set up for replication (ie ssh Tunnels work) go to replication tasks, select the SSH tunnel and what you want to replicate, how often, etc.

For max. performance, in replication task setup select “SSH + netcat” instead of just SSH. Then set your sending NAS Location to “Local”. Enable that task and watch stuff fly by at 200+MiB/s, quicker than any other program (and that’s with single VDEVs using HDDs for me)

I’d also make sure that if the backup is to an offsite location that the snapshot tasks on the Concorde TrueNAS system are not deleted until they have been sent. That way, should the backup target go down for an extended period, you will not lose crucial snapshots necessitating a do-over.

 

[Constantin]

I have been working on a true off-site for a while and like Time Machine backups, it really pays to do the first sync locally (at 10GbE or higher between NAS’). Subsequent transfers only capture changes and can be a lot slower (thank you, Comcast!).

 

[pr0927]

Oh interesting! Alright - definitely will be setting that up. Just need to figure out the basics first, gotta' get my essentials (Nextcloud, Vaultwarden, Jellyfin) working with Ingress and the domain I use for self-hosted stuff. Previous experience only with Nginx Proxy Manager, not with Traefik - so I have some learning to do (tried to help a friend setup a TrueNAS Scale setup with Nextcloud - couldn't get it to install, just got error messages).

Once I get that all down solid, I'll try to follow what you said - or follow-up with more questions!

Appreciate your time and explanations!