video editing NAS on a budget in 2024

[akspecs]

hello all,

i would be eternally grateful if folks would be willing to share their thoughts and input on accomplishing the following task.

i'm in the process of sourcing parts and/or a pre-built storage server for video editing and storage for a small team.

the current workflow involves copying to/from and editing off of countless USB3 / thunderbolt DAS devices that hold many (up to 8) high capacity spinning rust drives configured in hardware raid (raid 5 !!!). i think it's time to move this team to a towards a more optimal and safe workflow that integrates TrueNAS CORE / SCALE.

generally there is one primary editor, though at times there can be as many as a few (2, maybe 3). i would like to set something up that can modestly accommodate up to 5 collaborators in the future. the typical take size should not be larger than a about a TB give or take (in <3...4GB pieces).

what we have now:
18 x 18TB seagate exos drives (new and under a 5 year warranty) that i plan to run in in a 6 wide raidz2 configuration (3 raidz2 vdevs), and expand storage as needed in the future in similar increments. i pushed the team to budget for this much storage due to their current 150 TB requirement.

these drives plan to live in a 36 bay supermicro chassis (which will be half populated for starters, allowing for room to grow as needed). the chassis and other hardware has NOT been ordered, yet.

of the following options, which ought to be sufficiently adequate?
these choices will come with a 36 bay chassis with dual 1280w platinum psu's ...

for $1200 + tax and shipping:
X10DRi+T4+
2x intel xeon E5-2683 V3 @ 2.0GHz (14 core, 28 total, 3.0GHz max turbo frequency) OR 2x Intel Xeon E5-2667 V3 3.2Ghz 8 Core (total 16 cores)
Controller: 1x AOC-S3008L-L8e HBA 12Gb/s IT mode
512GB DDR4 ECC REG (2133)

is this a good deal? i like that this board has integrated Intel X540 Quad Port 10GBase-T, and it appears affordable (though i may be wrong, let me know!). the goal would be to get as close to saturating 10 GbE, though i understand this may be a little difficult with a limited number of spinning rust disks at first without a dedicated flash pool. i reckon i can push the team to budget for a flash pool in the near future. i wonder how much of a wide / striped mirrored SSD pool is suggested for this.

alternatively we budget several hundred dollars more and get an X11 board as follows:
for $1769 + tax and shipping
X11DPH-T in a 36 supermicro chassis with an additional 2 x 2.5" drive bays in the rear
2x intel xeon 6140 @ 2.30GHz (18 core, 36 total, 3.7GHz max turbo frequency)
128GB DDR4 ECC REG (+$500 for 512GB RAM)
960 GB samsung m.2 22110 SSD PM963 PCIe nvme MZ-1LW9600 (subtract $100 to remove this drive)
NIC: * Integrated Dual LAN with 10GBase-T with Intel® X722 + X557

or, for approximately $450 i purchase a 36 bay supermicro chassis with the additional 2 x 2.5" caddies in the rear + 2x1280w psu and roll my own build.

for all those who made it this far, thank you so much. i look forward to your criticism and advice.

 

[sretalla]

10GBase-T

You mention that a lot... it's not great and you should avoid it in preference for Fiber or DACs... see here: https://www.truenas.com/community/threads/10-gig-networking-primer.25749/

the goal would be to get as close to saturating 10 GbE, though i understand this may be a little difficult with a limited number of spinning rust disks at first without a dedicated flash pool.

Don't be so negative... with a large amount of RAM (like the 512GB you mention in one of the options), you can count on ARC to have the file cached if it's used frequently, so you're reading out at RAM speeds... faster than 10Gbit.

 

[akspecs]

You mention that a lot... it's not great and you should avoid it in preference for Fiber or DACs... see here: https://www.truenas.com/community/threads/10-gig-networking-primer.25749/

agreed. the benefit of using a server board is that i can install SFP+ cards down the line once the existing office network will be upgraded with fiber / DAC. for now, i'm assuming that any office space that will be rented will more than likely be wired with cat 5a, 6, 6a, or something or other.

Don't be so negative... with a large amount of RAM (like the 512GB you mention in one of the options), you can count on ARC to have the file cached if it's used frequently, so you're reading out at RAM speeds... faster than 10Gbit.

that is the goal! mainly the ask here is if this will be overkill for starters, or not. i've also considered starting with 128GB or 256GB ECC RAM.

i'm also wondering if the use of registered ram vs unbuffered should be a problem here, or not.

 

[sretalla]

if the use of registered ram vs unbuffered should be a problem here, or not.

Don't mix them and make sure both CPU and motherboard support whichever type, but both are OK and "meet" the recommended ECC RAM point.

i've also considered starting with 128GB or 256GB ECC RAM.

Sure, if upgrading is easy enough, you can see where your sweet spot is... 18 x 18TB is a lot of storage, so if your working set isn't in the hundreds of GB, you may want a system with a lot more RAM.

more than likely be wired with cat 5a, 6, 6a,

You might need to think about how long those cables are if they are only 5a or even 6.

 

[briancmoses]

Don't be so negative... with a large amount of RAM (like the 512GB you mention in one of the options), you can count on ARC to have the file cached if it's used frequently, so you're reading out at RAM speeds... faster than 10Gbit.

Seconded! I'd also expect it'd be likely that the 10Gbps could get saturated under certain workloads (example: reads) for the proposed pool layout (3x 6x18TB raidz2 vdevs).

...2x intel xeon E5-2683 V3 @ 2.0GHz (14 core, 28 total, 3.0GHz max turbo frequency) OR 2x Intel Xeon E5-2667 V3 3.2Ghz 8 Core (total 16 cores)
... 2x intel xeon 6140 @ 2.30GHz (18 core, 36 total, 3.7GHz max turbo frequency)

Just playing devil's advocate, but are these CPUs appropriate for the anticipated workload? It's a lot of CPU cores on CPUs which are pretty aged. These older CPU cores are going to consume more power than their more modern counterparts. If two CPUs aren't required, then dropping a CPU will probably save you money over the long run. Similarly, moving up to a more modern CPU might be more expensive today--but end up saving you money in the long term.

is this a good deal?

Maybe! Value is pretty subjective and specific. At the very least you've got some good search terms to go look through the secondary markets for recent sales of similar hardware.

 

[akspecs]

Just playing devil's advocate, but are these CPUs appropriate for the anticipated workload? It's a lot of CPU cores on CPUs which are pretty aged. These older CPU cores are going to consume more power than their more modern counterparts. If two CPUs aren't required, then dropping a CPU will probably save you money over the long run. Similarly, moving up to a more modern CPU might be more expensive today--but end up saving you money in the long term.

good time for a reality check. what do folks recommend as a cut off for the oldest cpu i should consider, and at what base clock speed? it's my understanding samba benefits from higher clock speeds for single core performance. is skylake a good cutoff?

while for my personal use cases at home i find nfs shares to be sufficient and often preferable, i imagine SMB shares would be more appropiate for a multi-user office environment. i'm not going to count on the editors having an indepth understadning of unix + acl permissions.

the goal is to also host nextcloud or filebrowser for browser based file browsing and sharing (e.g. generating a url link to a file), and maybe syncthing as well. i'm not sure if the NAS /storage server should also host wireguard / openvpn.

but that should be it.

dual cpus, while probably not needed, offer more lanes and ram upgrade-ability into the future. this is also in anticipation of up to 5 users communicating with the storage server concurrently./

 

[Etorix]

Your limit is likely to be IOPS during random access in multiple files, since you'll have the total IOPS of… 3 spinning drives. Preferably 512 GB RAM, but I don't see a clear need for a PM983: L2ARC is NOT going to help with what is likely a multi-TB working set and you do NOT want a SLOG (sync writes OFF!!!).
For a handful of editors, likely using SMB, you want a CPU with high clocks over high cores.

 

[akspecs]

i expect the editor(s) will need more than 3 x 6 wide raidz2 vdevs in the coming months considering the trajectory of how quickly their storage has grown so far.

i won't be shy about justifying the costs of more RAM then.

 

[edgecode]

Wouldn't striped mirrors make more sense?

What's your working codec? Which NLE? Frame rate/frame size? Are you cutting OCM or is it dailies? Is it narrative, docs, reality, short form? Are finishing in-house? Color grading, VFX, deliverables?

 

[akspecs]

Wouldn't striped mirrors make more sense?

surely striped mirrors would offer more performance, though we have (relatively massively) growing storage needs of least 150TB, and again, this need is certainly expected to grow with time.

What's your working codec?

Working codes depend on the projects. Recently we used REDCODE with 6K and 2:4:1 aspect in MQ quality compression giving us about 134MB/sec data rates.

There are scenarios when we bump the quality to the HQ giving us to 192MB/sec.

For 8k in MQ compression we would have 238MB/s while HQ would be 340MB/sec.

Which NLE?

Davinci Resolve

Frame rate/frame size?

All of the above is at 24 frames per second. Well, usually 24P, b roll with slow motion at 60P/120P often we have no slow motion at all if it is an interview.

Are you cutting OCM or is it dailies?

No dailies.

Is it narrative, docs, reality, short form?

Docs, and lots of interviews.

Are finishing in-house? Color grading, VFX?

Finishing the color grading not in house, as well as VFX not in house.

deliverables?

Yes. Streaming networks and conventional distributors.

 

[edgecode]

If you are not finishing in-house you don't have to cut OCM's but I can see why you'd want to.

I have a similar setup to what you're going to build, just a generation older (X9). It's been neglected and never performed well. I upgraded it to the current TrueNAS, rebuild the pool, aggregated the 2 10GbE channels. It is still work-in-progress since the hard drives need replacement. I have 24 old SAS 4TB drives in striped mirrors. I can reliably play 4 streams of ProRes422HQ at 3840x2160 at 23.98 fps. I can play most flavors of RedRAW and ArriRAW. I can copy to and from the NAS at over 800 MB/sec. The problem is it's all that good only for the big continuous files. As soon as I hit play on the actual Resolve timeline it's becoming jerky, regardless of the source media.

I'm not losing hope but something is very off with the default configuration or TrueNAS in general. People have been editing off of the shared storage for very long time and it worked. SneakerNet is a huge efficiency hit and a pain in the ass.

 

[edgecode]

If you search there are a few threads on this forum looking at TrueNAS from the postproduction requirements angle but they seem to die off rather quickly. Our industry is actually pretty small and badly compartmentalized. Just check any spreadsheet app — there is no "timecode" data type in any of them. Still, I hope we can regularly share if not our knowledge but at least our frustration here.

 

[ChrisRJ]

While I am not in the video editing business, I think quite some aspects apply from other use-cases and verticals. It basically boils down to the thought that not all data are equal. You almost always have "active" and "inactive" data.

For active data performance (be it throughput and/or latency) is usually critical, while much less so for inactive data. Tiered storage (SSD => HDD => tape) is a typical implementation pattern to solve this in an economical way.

For the scenario at hand it means something along the lines of having SSDs mirors (option: connected to workstations over a dedicated 10+ Gbps link) for video that are being worked on; and RAIDZ2 vdevs for the archive. If money is irrelevant, you can of course go all-in with SSDs.

It might also be worth investigating the use of NFS over SMB/CIFS for performance reasons.

 

[edgecode]

I am sorry but "tiered storage" is what I keep hearing in one form or another. That's not applicable. There is no "inactive data" in our use case scenario. Well, there is but it is irrelevant. Old projects are backed up, archived to LTO, given to clients and discarded. It's a solved problem. No need to mention. All data we are talking about here is "active". Hundreds of multi-gigabyte files which need to be randomly accessed at the guaranteed speed from several workstation with absolutely unpredictable pattern. That's the nature of the beast.

And we haven't started talking about DPX/EXR sequences here.

As for NFS/SMB I'm sticking with AFP for as long as it is alive.

 

[akspecs]

People have been editing off of the shared storage for very long time and it worked. SneakerNet is a huge efficiency hit and a pain in the ass.

i believe it. shuffling through over a dozen DAS devices is a real PITA...

...As soon as I hit play on the actual Resolve timeline it's becoming jerky, regardless of the source media.

in which case, what's the alternative workflow if the subpar performance becomes unmanageable? is it truly unmanageable?

considering our past workflow under less than desirable conditions, ideally we would like an outcome where moving towards editing off of a centralized NAS is an overall net benefit. to be more precise, performance that meets or beats the current experience of editing off of over a dozen DAS devices would be considered a victory and well worth the investment.

I have a similar setup to what you're going to build, just a generation older (X9). It's been neglected and never performed well

based on this and the advice of @briancmoses, it sounds like it may be well worth budgeting more for an X11 generation build with a more powerful and efficient single socket motherboard and processor. we are still open to this, and are okay with moving away from the older dual socket X10 build, despite having to spend more.

 

[edgecode]

I'm trying to gather some stats but so far it looks a bit scary. Every thread similar to this one just dies off. Is it an indication people abandon TrueNAS for postproduction? What are they switching to? I'd like to know.

My use case is somewhat different from yours. We are an online shop so smooth playback isn't that crucial, just nice thing to have. In your case, actually, the amount of RAM might be the solution: editing a scene is usually about playing the same set of shots over and over. I noticed that when I go back an play the same part of the timeline again it's smooth.