Scale system - Hybrid of NVMe and HDD pools presented as a single share

[eastendtom]

Hi there,

I'm new to the TrueNAS project (looks very interesting!) and I’ve got a question before building my 1st test system. I run in-house technology for a film post production company, and need to read large image sequences very quickly. (4K TIFFs playing at 24 fps etc), therefore an all-flash solution to upgrade our legacy SAN storage is very interesting. Something like 40 TB NVMe to start with. However, I also need a volume of storage which will be much larger capacity (for slower reading material) but doesn’t need to be so fast read / write. Something like 250 TB (with ability to scale).

My question is this. At the mounted filesystem level I’d like a single share presented to Mac, Linux and Windows workstations, but to have some data 'homed' on the NVMe section, and some 'homed' on the HDDs. Is this possible please? Costing up some options at the moment, and if this seems feasible then I'll bite the time-bullet and get stuck in.

Thanks in advance!

 

[Arwen]

Based on your requirements, you would have 2 pools. One of NVMe devices and another of HDD. I would not think a single share is possible, as I think each has to have it's own shares.

Their is something about UnionFS or overlay mounts, but I don't recall the details. Perhaps someone else can chime in.

Getting specific performance characteristics out of a home built NAS can be tricky. One mistake and the performance target can be missed. Two mistakes, (or more), and performance can suck.

You might also look at a pre-built system from iXSystems. Possibly even with a support contract. (I have no connection to iXSystems other than using the free version of TrueNAS.)


One last comment. ZFS is basically radically different from all prior file systems, volume managers and RAID schemes. Lots of new TrueNAS users make design or usage mistakes because of those differences. Their is a lot of information in the Resources section of this forum, (top of any Forum page is a "Resources" link). I'd suggest poking around it. Plus reading these, (if you are not familiar with ZFS);

Introduction to ZFS

This is a short introduction to ZFS. It is really only intended to convey the bare minimum knowledge needed to start diving into ZFS and is in no way meant to cut Michael W. Lucas' and Allan Jude's book income. It is a bit of a spiritual...

www.truenas.com

Terminology and Abbreviations Primer

We realize that new users have a lot to learn when they come to FreeNAS. There's a certain amount of confusion added to discussions when users pick random/approximate terms to describe things. I've spent a lot of time quietly trying to...

www.truenas.com

Picking a ZFS Pool Layout to Optimize Performance

Part 1 of the blog post is here, covering basics, striped pools, and mirrored vdevs: https://www.ixsystems.com/blog/zfs-pool-performance-1/ Part 2 covers RAID-Z and some example workload scenarios...

www.truenas.com

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

 

[sretalla]

something about UnionFS or overlay mounts, but I don't recall the details. Perhaps someone else can chime in.

There was some discussion during SCALE development of using a Tiered storage approach with something like autotier or gluster to do that work, presenting a single entry point for both fast and slow storage and managing files between them in the background.

Limited success/joy in that direction and it has gone cold for now (https://www.truenas.com/community/t...tion-i-wish-to-run-autotier.94103/post-651313)

I also raised the tiering concept years back for CORE and did get some kind of response indicating it was being looked into (mixed with an indication it would have to be done in ZFS itself... I didn't agree with that assessment), but nothing ever came of it.

 

[HoneyBadger]

My question is this. At the mounted filesystem level I’d like a single share presented to Mac, Linux and Windows workstations, but to have some data 'homed' on the NVMe section, and some 'homed' on the HDDs. Is this possible please? Costing up some options at the moment, and if this seems feasible then I'll bite the time-bullet and get stuck in.

Morning Tom,

"Pinning" or "tiering" certain directories or folders by path isn't currently possible. I don't believe your use case would be solved by the pending inclusion of OverlayFS support, as the lower layer is designed for read-only support and you need to be able to add files to the "archive" tier.

Would having two separate shared folders be possible, as in \\Server\Fast and \\Server\Archive? This could be easily configured within TrueNAS itself.

 

[eastendtom]

Thank you @Arwen @sretalla and @HoneyBadger. I realise that home-brew high-spec NAS setups are going to be a challenge for a newbie, I'm already talking to the sales guys about an iXSystem, but am interested in experimenting first in some 'real world' tests which might be meaningful against a potential turn-key system.

On combining NVMe and HDD Pools, I understood that it's not possible currently to Tier as above. It would be nice, in my use case. SANs from other manufacturers have features like 'caching' of certain directories to faster storage, which might be a useful feature in TrueNAS, too.

For now the suggestion from @HoneyBadger for \\Server\Fast and \\Server\Archive would be a good starting point for testing. Thanks very much all!

 

[HoneyBadger]

SANs from other manufacturers have features like 'caching' of certain directories to faster storage, which might be a useful feature in TrueNAS, too.

ZFS will use RAM to cache the read workloads of any dataset (where the default value of "primarycache" hasn't been modified) so if read performance is your primary goal, a healthy amount of RAM would be a good place to focus - in a system where 40TB of NVMe storage will be in play, aiming for something as plentiful as 512GB wouldn't be out of the question, with 1TB being reasonable as well.

How long would a given "segment" of film be that's being represented by the raw/uncompressed 4K frames, and how many concurrent projects would be active? If there's enough RAM in the system to hold the entire "working set" then you'd effectively be reading those files at line-speed out of the TrueNAS machine's memory.

 

[ksimm1]

One option to consider:

1. Create a pool with the spinning rust as normal vdevs
2. Add the NVME drives as metadata vdevs
3. Create your datasets - for the datasets that need to be NVME-only, set the Recordsize = "Metadata (Special) Small Block Size"

Shares can then be presented as normal at dataset-level without regard for NVME vs spinning rust

 

[sretalla]

One option to consider:

1. Create a pool with the spinning rust as normal vdevs
2. Add the NVME drives as metadata vdevs
3. Create your datasets - for the datasets that need to be NVME-only, set the Recordsize = "Metadata (Special) Small Block Size"

Shares can then be presented as normal at dataset-level without regard for NVME vs spinning rust

Be aware that although this will "work", you will see no benefit from the special VDEV(s) if you have sufficient RAM and all of your files are video/media (large files will never hit the special VDEV).

 

[Etorix]

I run in-house technology for a film post production company, and need to read large image sequences very quickly. (4K TIFFs playing at 24 fps etc), therefore an all-flash solution to upgrade our legacy SAN storage is very interesting. Something like 40 TB NVMe to start with. However, I also need a volume of storage which will be much larger capacity (for slower reading material) but doesn’t need to be so fast read / write. Something like 250 TB (with ability to scale).

For professional use, with the budget to match the above description "40 TB NVMe to start with"/"250 TB (with ability to scale)", you'd do well to give iXSystems a call and get a quote based on their big systems.