Recommendations for Getting Started

[GioNainf]

I have little experience with ZFS or TruNAS (other than a previously failed attempt to do this), but I'd like to build a ZFS file server for home & two for work. I'm looking for any guidance you can provide.

I thought I start with a test rig, using what I currently have:
- Rosewill RSV-L4500 15 Bay Rackmount Case
- Dual Intel Xeon X5680, Motherboard, 24GB ECC Ram, Power supply
- 10x 16TBGB HDD (used), 2x Samsung 128TBGB 840 (used), numerous Optane P1600X M.2 (still in box)

What I think I still need:
- HBA or 2
- 5x 16TBGB HHD, maybe as a backup pool
- PCIe NVMe Carrier Card (if I use the optane)

For starters (testing), I think I'll setup
- 1x SSD for Op Sys
- Perhaps Optane for SLOG and/or SSD for L2ARC
- 10x 16TBGB HHD as 4 pair mirrored VDEV with 2 Hot Spares for primary storage (64TBGB total)
- 5x 16TBGB in RAIDz1 to experiment with ZFS replication as a backup pool (64TBGB Total)*
*All my data is currently on several 8TBGB externals using DrivePool/Snapraid, which I'll keep as a backup also.

I'll probably start with TruNAS Core. What I'd eventually like to be able to do run a couple Windows VMs and Linux VMs for various server software such as Blue Iris (not as a workstation). I'm not sure I can pull that off with this hardware, or if TruNAS Core is the right OS for this either. Maybe TruNAS Scale is a better option, or even a bare metal hypervisor with virtualized TrueNAS

Eventually (sooner rather than later), if all goes well, I intend to purchase additional, newer hardware but I wanted to test this out before spending anymore money

Is this a reasonable approach? Is the hardware satisfactory? Any suggestions?

Also, what happens if I physically remove the HHDs in a pool, and put them in another ZFS based file server? For example, If I physically transfer the 5x HHDs backup pool to another ZFS based machine (or blow away the OS on the existing machine), will I be able to read those drives in the new machine?

 

[Arwen]

First, I assume you mean 16TeraByte HDDs, not 16GiigaByte HDDs. (But, we have seem some odd things here...)

Unless you KNOW you need a SLOG, you don't need a SLOG. Period. One can always be added after, even live, without impact. (Or even removed.)

If you export your ZFS pool, you can generally import it into another computer that has ZFS. But, you need the minimum feature set on the pool, to be supported by the new computer. This is one reason why some of us make our ZFS pools with limited features, like Grub2 compatible, and only enable specific features as desired.

It is very helpful to read up on ZFS, (and TrueNAS). Maybe even make a simple VirtualBox VM for testing & playing. ZFS is quite a bit different than other file systems and RAID schemes. This can lead to design or usage mistakes, even ones that cause data loss.

While ZFS, (and TrueNAS), were both written for the Enterprise Data Center, they can both be used with SOHO and consumer computers just fine. Their are some trade offs, but one that is overlooked a lot, is power. ZFS was purposefully designed to keep the existing data consistent even with many, MANY, unexpected power offs. So, while a UPS is nice, it's not needed to keep existing ZFS data intact. Of course, data in progress of being written can be lost during a power loss. And a power loss can trigger marginal hardware to fail, even to the point of data loss. But, that is not a ZFS issue.


As for the rest, I don't know, so I've not commented.

 

[GioNainf]

Thank you, & yes I meant TB not GB

I remember reading about import / export, which is great if the disk transfer is a planned event. What happens if the MB fails unexpectedly? I understand that the TrueNAS config can be saved periodically. If I install another, different MB/CPU is it simply just a matter of reinstalling TrueNAS, restoring the config file & then our ZFS array/data is back online?

I've been studing ZFS off & on for years. At this point, I think I just need to get my hands dirty. I'm also not familar with building servers. I've built desktops, but not servers so I've had a learning curve for things like HBA, backplanes & the various different connector types (MCIO I guess is the new thing) that seem to evolve much more quickly than in the desktop.

Last night I ran into another problem: while my current power supply has plenty power, it only has 9 SATA power connectors & no molex connectors. If I can find them, I've thought about adding crimped Inline/Through connectors (as opposed to using splitters, which I understand may cause fires) but then I still have an issue getting power to the chassis fans (which are molex).

I may just drop trying to use this older hardware, & pickup a new chassis with backplane like the 45 drives home lab. I think that may prevent a lot of headaches.

 

[Arwen]

...
I remember reading about import / export, which is great if the disk transfer is a planned event. What happens if the MB fails unexpectedly? I understand that the TrueNAS config can be saved periodically. If I install another, different MB/CPU is it simply just a matter of reinstalling TrueNAS, restoring the config file & then our ZFS array/data is back online?
...

Yes, you can still import your ZFS pool after an un-planned outage of your NAS. It does require using the "force" flag, but that is harmless in the case we are discussing. The "force" flag is just to prevent accidental imports of another server's pool.

Yes, restoring the configuration is just that easy. I've not done it, but plenty in the forums have. And it is meant to be that easy for the Enterprise customers. The only annoying detail about restoring a TrueNAS after system board failure, is that the network chip likely changed PCIe address. Or even vendor & model. So that has to be fixed from the console. Unless you use an identical system board as a replacement.

...
I've been studing ZFS off & on for years. At this point, I think I just need to get my hands dirty. I'm also not familar with building servers. I've built desktops, but not servers so I've had a learning curve for things like HBA, backplanes & the various different connector types (MCIO I guess is the new thing) that seem to evolve much more quickly than in the desktop.

Last night I ran into another problem: while my current power supply has plenty power, it only has 9 SATA power connectors & no molex connectors. If I can find them, I've thought about adding crimped Inline/Through connectors (as opposed to using splitters, which I understand may cause fires) but then I still have an issue getting power to the chassis fans (which are molex).

I may just drop trying to use this older hardware, & pickup a new chassis with backplane like the 45 drives home lab. I think that may prevent a lot of headaches.

Think servers as;

• Needing perfect cooling, as they run 24x365
• Don't need audio, Firewire or lots of USB
• Unless you plan on video trans-coding, a graphics card is not needed except as a console
• Many server boards include IPMI / BMC for remote console access & server management, (remote power off / on)
• Not having any fancy lighting or fan lights, as that would both waste power and be distracting

Servers are like big trucks, where desktops are like cars or mini-vans and gamer rigs like sports cars.

 

[ChrisRJ]

I've been studing ZFS off & on for years. At this point, I think I just need to get my hands dirty.

I don't want to be impolite. But in my view your questions partially suggest otherwise. I am a big fan of hands-on and would definitely support the approach that @Arwen has suggested, i.e. setting up a VM and play around.

But once we talk about the hardware choices I would recommend additional reading before buying things. No dodgy YouTube stuff or other forums, but the great collection of resources that people have written here. A sub-set of that can be found in my signature ("Recommended readings"), but there is certainly more.

Again, no offense meant, just trying to help.