My ideas for a personal TrueNAS SCALE Server (Looking for feedback)

[stevenk]

Hello. I became interested in managing my data using ZFS. My data would be personal data, mostly video game-related files. If you collect various ROM and ISO sets, that can really take up a lot of storage. I'd just like to ask for some feedback for my pie-in-the-sky build idea that would be very expensive to implement. My ideas might not work at all. I'd appreciate any feedback, though.

I don't have the space where I live for anything like a rack server, so my idea was to build a PC dedicated for all this and chain together 16-bay Terra-Master Thunderbolt 3 enclosures. There is the first potential problem. I was told by Terra-Master support that daisy-chaining those enclosures would work in Windows and MacOS, but I have no idea if it would work in TrueNAS Scale. Even if there is enough support for it in the Linux kernel, I have no idea if that could work in TrueNAS.

My idea for a PC build would include a Gigabyte X570 AORUS XTREME motherboard because I saw that ECC RAM could be used with it. That motherboard has 4 M.2 slots.

My idea is to make a pool for each daisy-chained Terra-Master enclosure. If I have 16 22-terabyte drives in four enclosures using RAID-Z3, that would equate to a little over a petabyte of usable storage Would 128 gigabytes of RAM even work for that much storage? I have an Asus Thunderbolt 4 card that I would plan to use with this build.

Then, as I started learning more about the various VDEVs, I started thinking about layouts. What if I kept one M.2 drive as the boot drive, one each for L2ARC/ZIL, and one for SLOG?

I've read about the potential dangers of using the special VDEV for metadata. Would it be an absolutely awful idea to use an Asus Hyper M.2 card to use four M.2 drives for metadata because the Hyper M.2 card is a single point of failure? Would it even be possible or make sense?

Also, I have an idea to use SATA SSDs for easily reproducible data created using apps. I'm talking about data such as a Bitcoin full node. Taking a quick look, there doesn't seem to be an app for creating a Bitcoin full node either officially or through TrueCharts. I guess that I could make something work using the docker-compose app in TrueCharts, but I understand that that is completely unsupported.

Finally, is my question about VMs. If I'm using all available M.2 drives for boot and various VDEVs, would a SATA SSD be fast enough for various Windows and Linux VMs?

As for how this all backed up, I have a cloud solution in mind. I understand that it is not a proper 3-2-1 backup, but I would not be able to have two of these servers where I live.

Thank you so much.

 

[Arwen]

TrueNAS SCALE or Core does not support any Thunderbolt 3 enclosures. Only SATA, SAS & NVMe storage are supported for data. (Boot drives can be on USB if absolutely necessary.)

TrueNAS works better on server style system boards, not gaming boards. Also, over clocking is actively discouraged because a server runs for months on end, so extra heat or over-volt stress on the silicon is not desired.

Board support for ECC memory is one thing, you have to match it with a CPU that also support it.

Special Metadata vDevs are an advanced feature many users don't need. And can be problematic if not configured correctly. Whence a Special Metadata vDev is added to a pool, it is generally not removable. (It possibly could be removed on a Mirrored data pool...)


Both L2ARC & SLOG are for special cases, that new users generally don't need. Both have special AND different hardware requirements. Note that ZIL is the in-pool equivalent of the SLOG, (Separate ZFS Intent Log). Unrelated to L2ARC. All pools have a ZIL and its not possible to remove it, except by adding a SLOG. Both L2ARC & SLOG can be added or removed live, as needed or desired.

A L2ARC is not suggested until you come close to, or have, maxed out your memory. This is because L2ARC needs RAM for the indirect table / catalog of data on the L2ARC. Next, a L2ARC can have a bit of churn, (constantly changing data), so a high endurance SSD or NVMe is suggested.

The SLOG is only useful for sync writes, like from NFS or iSCSI. SLOGs require power loss protection, and high endurance of writes. A SLOG is basically a write only device, never really read until a NAS server crash.


Looks like you have a lot of reading to do both about ZFS & TrueNAS. You can check out the Resources tab at the top of all forum pages. Resources are community write information on specific subjects. You can avoid ones that seem unrelated, we have quite a few for special failure modes or specific hardware cases. Sorry I don't have a handy reading list for new comers.

 

[stevenk]

Thank you so much. Your reply is very helpful.

 

[danb35]

I don't have the space where I live for anything like a rack server, so my idea was to build a PC dedicated for all this and chain together 16-bay Terra-Master Thunderbolt 3 enclosures.

This doesn't make any sense to me--those enclosures appear to be about the same size as a rack-mount chassis. How can you have room for multiple units of these, plus a separate PC, but not room for a rack server? And how much storage do you expect to need here? Are you seriously contemplating a petabyte server?

Keep in mind that any such solution is going to require a good bit of cooling for the drives. That means airflow, and that in turn means noise. The more compact you try to make the system, the harder the fans are going to need to work, and thus the greater the noise levels.