Hardware Audit for Editing NAS

[SpecTech]

Hi All, first-time poster. I have about 15 years of working with various RAID systems and recently deployed some 45drives systems, but this will be my first TruNAS Scale system. This is a lower-budget system designed for expandability but I am mostly interested in throughput for this post. Please see my intended build below.

I will start with 12 Exos x20 20TB HDD and 6 Micron 5400 Pro 1.92TB SSDs for L2 ARC. I run a building full of Macs and will rely primarily on SMB3 Multichannel as the transfer protocol. I plan on running a dual-port SFP28 Nvidia/Mellanox ConnectX 4 NIC in my server.

I am considering adding an additional HBA to run the secondary backplane where I intend on setting up the L2 ARC pool but wanted to post it here first. I am focused on fully saturating my network connections.

 

[Ericloewe]

The general idea is okay, but...

6 Micron 5400 Pro 1.92TB SSDs for L2 ARC

That does not make sense to me, it's a crapton of L2ARC (possibly too much), but it's slow as molasses. It's not clear that you'd benefit from L2ARC (what kind of workload? Streaming directly from the server? Downloading from the server and uploading the results?), but if you do benefit, you really should go with NVMe SSDs.

There are a few options for that in the chosen chassis:

• There might be some internal 2.5" positions you can populate with U.2/U.3 SSDs
• Standard PCIe card form factor SSDs
• Upgrade one of the backplanes to support U.2 disks (e.g. BPN-SAS3-846EL1-N8 front or BPN-SAS3-826EL1-N4 rear), which may be simple or not simple. Research the details before buying!
• M.2 adapter cards (not great)
• Replace the rear 2.5" drive bays with the NVMe model (but that's just two SSDs, which is limiting) and boot from the on-board M.2 slot

I plan on running a dual-port SFP28 Nvidia/Mellanox ConnectX 4 NIC in my server.

The Mellanoxes are a bit finicky, I hear. You can probably have a better experience with an Intel XXV710 or E810.

 

[SpecTech]

@Ericloewe Thank you for the advice on the Intel NIC, I mostly wanted the L2 ARC to be a Z2 array to tolerate drive failure well. Everything fails, that is what I like about the ZFS system. It trusts nothing. Replacing the rear backplane sounds like the most attractive option to me. I like everything to be pristine. Since L2 ARC is mostly for read I will look into 4 x Samsung MZWLL3T2HAJQ-00005 in a Z1 array with a supporting backplane. It would be cheaper with a speed boost.

 

[Ericloewe]

,I mostly wanted the L2 ARC to be a Z2 array

That's not a thing, L2ARC is always striped. In practical terms, (if my theory is correct, it's a bit of an edge case not often discussed) if a disk dies, you only lose the L2ARC blocks from that disk. Of course, that's fine because it's just a cache anyway.

 

[SpecTech]

@Ericloewe OK, good to know. So is L2 ARC block based instead of file based? I have several editors pulling multiple streams of 1080p ProRes per project. So it will only pull the blocks they have requested into L1 ARC (eventually spilling to L2 ARC) and not load the entire file being accessed? Which TrueNAS doc has a deep dive on this? Or is this more of a general ZFS deep dive?

 

[Ericloewe]

So is L2 ARC block based instead of file based?

Yes, very few things in ZFS deal with files, files are just an abstraction in the higher layers.

So it will only pull the blocks they have requested into L1 ARC (eventually spilling to L2 ARC) and not load the entire file being accessed?

There is some prefetch, but it's not super speculative. File read-ahead is supposed to work, to some extent, but there's been substantial work around the ARC recently and I'm not sure which bits are in which version.

Which TrueNAS doc has a deep dive on this? Or is this more of a general ZFS deep dive?

Yeah, this is strictly a ZFS topic, but I'm not aware of any good deep-dives. To a large extent, it's good enough that people don't worry about the specifics. But, as I said, there have been recent improvements to fix bugs that creeped in as other pieces changed around the ARC and make things more consistent.