Hardware choice for a 32+TB NAS used video editing 4-8K footage

[Agrarvolution]

Hey,

I've read a couple of previous discussion about similar hardware needs. I didn't really see my usecase, since I'm just a single freelance editor. Thus I didn't really feel confident on moving ahead and build a system with the specs that I think that should work. I've choosen the Truenas path to be more flexible than Synology and get a cheaper and hopefully saver solution than what QNAP offers.
But in the end, I just don't have quite enough experience for that.

I currently gather somewhere between 4-6TB of data every year. Of this data, most of it, just sits in storage, and is basically write once, read many/rarely.
Anecdotally I know that editing in Premiere Pro from a larger Synology NAS over 1GbE is way too slow. Basically scrubbing the timeline is a waiting game, even with 1080p footage.
I currently edit from SSDs because the responsiveness is so much better than with a single HDD.

I also want to split work from storage, since I've made a really bad experience last year , if that isn't the case.

Use case:
Single editor with Premiere Pro using (currently) 4-5K footage which will upgrade to max 8K in about a year
Several smaller users who just access data, but don't edit

Current hardware choice:
Case - Fractal Meshify 2 (*)
PSU - 850W BeQuiet Straight Power 11 (*)
RAM - 128GB 3600MHz non EEC (*)
CPU-Cooler - BeQuiet Dark Rock Pro 4 (*)
CPU - Intel i7 13700 (**)
Mainboard - Gigabyte Z790D DDR4 (**)
NIC - INTEL X710-DA2 (**)

*These are from my previous workstation and I'd really like to reuse these parts - especially the case.
**These are from the current offer I've got from my PC shop.

Drive choice:
Sound is a big issue for me, at least if the NAS wasn't used for 30min it shouldN't make noise.
I'm not sure at all if going full SSD is worth my money or if a HDD is a better choice.

I planned on making a RaidZ1 SSD array with either 5x8TB Samsung QVOs or 5x8TB Samsung PM893. I'm rather wary of QLC and the PM893 still feels too expensive.
My other choice would be to do a a 5x16/18TB HDD pool in RaidZ2 (with maybe L2ARC?) and pray that the clicking isn't too loud or that it can utilize drive sleep (since most likely they'll be accessed about 1-6h a day).
For both options I wanna leave space for one hot spare at least.

The goal is to saturate one 10GbE connection (It should be enough for 8K, but I can't say for sure yet since I only have some very limited sample files) and to keep the system costs at least below 750€ or 1000€ max.

I'd really appreciate your help. I don't know if any of my thought process is correct or if I mislead myself in a way.

 

[Ericloewe]

Keep the data you're actually working with on your workstation (good consumer SSDs are dirt cheap and will easily scale to handle your <1 TB per month), move everything in bulk over to your NAS when you're done. You don't need your NAS to also make toast. All things considered, you can probably even skip the upgrade to 10 GbE.

The goal is to saturate one 10GbE connection

You're not going to do that with a single RAIDZ vdev of HDDs. Even with SATA SSDs, I'd doubt it. And when you did get to the point of saturating 10 GbE, you're dealing with less bandwidth and a heck of a lot more latency than even a single mid-range NVMe SSD inside your workstation.

Case - Fractal Meshify 2 (*)
PSU - 850W BeQuiet Straight Power 11 (*)
RAM - 128GB 3600MHz non EEC (*)
CPU-Cooler - BeQuiet Dark Rock Pro 4 (*)
CPU - Intel i7 13700 (**)
Mainboard - Gigabyte Z790D DDR4 (**)
NIC - INTEL X710-DA2 (**)

Your livelihood depends on this working and being reliable, so you should really cut the consumer/gaming stuff and use ECC memory. The good news is that you're seriously overdoing the specs if you just do bulk storage for 60ish TB of video (i.e. large files).

Used Skylake stuff is pretty reasonable these days and gets you enough oomph to make this scenario work even if you want to expand a bit, since you can go up to 64 GB of DRAM with something like an X11SSM-F or X11SSH-F.

 

[Agrarvolution]

Thanks for your quick response and suggestions.

You actually gave me a new idea I haven't considered yet.

Keep the data you're actually working with on your workstation (good consumer SSDs are dirt cheap and will easily scale to handle your <1 TB per month), move everything in bulk over to your NAS when you're done. You don't need your NAS to also make toast. All things considered, you can probably even skip the upgrade to 10 GbE.

I already have a similar setup (HDD to SSD) and I'm well too aware of the pitfalls of doing that.
A solution that works similar to how OneDrive or Dropbox operate on PC would be perfect. You always see every folder and you sync the stuff you need from these and then it copies the changes automatically back to the server. And if you don't need it anymore you just remove the local copy.

I'm wary of manually copyied folders. I try to keep that as organized as possible, but I already missed copying some of them back to the backupped drives. And I would rather not deal with that headache anymore.

I tried seraching that on Google, but couldn't find anything concrete. So I think I formulate that question wrong. Are you aware of any software that can do that on top of Truenas?

You're not going to do that with a single RAIDZ vdev of HDDs. Even with SATA SSDs, I'd doubt it. And when you did get to the point of saturating 10 GbE, you're dealing with less bandwidth and a heck of a lot more latency than even a single mid-range NVMe SSD inside your workstation.

10GbE should be 1280MB/s right? Do you really loose that much performance just by doing a RaidZ1 array?
I forgot to add in the initial question, that I'm already sitting on 20TB of data. If I go the SSD route, I will most likely have to add another pool in 2 years anyways.

Your livelihood depends on this working and being reliable, so you should really cut the consumer/gaming stuff and use ECC memory.

How big of a concern is this really? If bitflips were rampant I should have already lost significant amounts of my data, which I haven't. I know that I have minor amounts of bitrot in my archives but not on a "file damaging" level.


Thanks again.

 

[Ericloewe]

10GbE should be 1280MB/s right? Do you really loose that much performance just by doing a RaidZ1 array?

Since you'd be doing a lot of random I/O, you're looking at the equivalent performance of a single disk for every RAIDZ vdev. So yeah, saturating the network is unlikely.

How big of a concern is this really?

You tell us. Will customers get pissed off if you lose their footage? Are you willing to re-do a project because something got corrupted towards the end?
My professional experience lies elsewhere, but given the marginal difference in cost and the vast difference in features, compatibility and reliability, I find not buying professional gear something of a crazy idea.

 

[Agrarvolution]

Since you'd be doing a lot of random I/O, you're looking at the equivalent performance of a single disk for every RAIDZ vdev. So yeah, saturating the network is unlikely.

Ah now I see where you are coming from. I wasn't aware that RaidZ didn't improve IOPS. It currently doesnt saturate the SSD either, with 2 exception - high bit rate and during a render.

You tell us. Will customers get pissed off if you lose their footage? Are you willing to re-do a project because something got corrupted towards the end?
My professional experience lies elsewhere, but given the marginal difference in cost and the vast difference in features, compatibility and reliability, I find not buying professional gear something of a crazy idea.

I plan on doing a backup of the NAS in a (hopefully good) automated way. I see my risk mainly in not having enough copies of the footage.

So basically tiering down my hardware specs and using TrueNAS + Nextcloud would serve my needs better than going all SSD?

Thanks again. :)

 

[Etorix]

I forgot to add in the initial question, that I'm already sitting on 20TB of data.

This amount of data is a strong indication that a raidz2/3 of HDDs is a much cheaper solution than SSDs. Of course, the more drives, the more noise, so move the NAS away from your ears and enjoy editing from SSD in your (hopefully quiet) workstation, using whatever automated backup solution fits your OS of choice.

As for 10 GbE, which will help with moving big files back and forth, I don't know what your hardware shop ask for the X710-DA2 but Solarflare 7122F cards go for $50 on eBay.
For storage use only on the NAS, you can save a lot on CPU+motherboard (a Raptor Lake i7 would be way overkill…). And then use the savings to consider ECC RAM after all (e.g. refurbished DDR4-2400 ECC UDIMM to go with a Core i3 and a X11SSH).

 

[Agrarvolution]

Yeah, I thought of a 12300 or max a 13600 when I asked for parts.
The X710-DA2 is 229€ without vat. Last time I asked there was a cheaper option, but it didn't seem to be available for him at the moment.

I assumed that if I go the SSD route I need more CPU power.

 

[Ericloewe]

Regardless of the other details, definitely don't buy anything that recent from Intel (and the same probably goes for AMD), the software stack is just not there and the motherboard side of things is an absolute trainwreck in terms of support for anything other than Windows.

 

[Agrarvolution]

Regardless of the other details, definitely don't buy anything that recent from Intel (and the same probably goes for AMD), the software stack is just not there and the motherboard side of things is an absolute trainwreck in terms of support for anything other than Windows.

These are the thing I'll probably never learn, since I just suffered through 2 years of 5950X early release pains.

 

[Agrarvolution]

Regardless of the other details, definitely don't buy anything that recent from Intel (and the same probably goes for AMD), the software stack is just not there and the motherboard side of things is an absolute trainwreck in terms of support for anything other than Windows.

Thank you for your suggestions. Would use use a HBA rather then the internal SATA for my usecase?

 

[Ericloewe]

Not unless you're using expanders and/or need more disks.

 

[Agrarvolution]

Ah ok. Thanks for your quick reply.
Another question that kept comming up a couple of times, when I looked into HDDs before - is the premium you pay for the Ironwolfs Pro worth it all? They are about 120-150€ more expensive then Seagate Exos X or Toshiba N300 (at 16+TB). Kinda seems like marketing bs to me.



On a side note: Should I keep you guys updated in this thread with what I built in the end for the search engine's sake?

 

[Ericloewe]

Another question that kept comming up a couple of times, when I looked into HDDs before - is the premium you pay for the Ironwolfs Pro worth it all? They are about 120-150€ more expensive then Seagate Exos X or Toshiba N300 (at 16+TB)

Yikes, that's getting into "buy one, get one free" territory. 150 bucks "premium" is obscene for an HDD, outside of niche scenarios (e.g. dual actuator drives).