8K Video Editing Nas Hardware Requirements

[vineesh]

Hi,

I am planning to build an 8K ready NAS server with at least 4000MB/s read speed. I haven't yet decided between Xeon and Epyc.

My plan for components

EPYC 7343 / XEON W-11865MRE
128GB DDR4 3200
WD SN850 2TB Nvme for L2ARC
WD SN850 2TB Nvme for SLOG

A QSFP28/QSFP+ NIC, Probably going for Chelsio (Pls suggest)

Exos x20 20TB X 24 drives.

6 machines will access the data from this NAS server.

Should I go for striped Vdev Config instead of raidz2? I need as much space as I can have.

 

[ChrisRJ]

Will the editing be done locally on the workstation, or directly off of the NAS?

 

[sretalla]

Should I go for striped Vdev Config instead of raidz2? I need as much space as I can have.

Maximum speed doesn't come for free... you can either pay with money (for the right disk configuration) or with (high) risk (by going with a striped pool).

6 machines will access the data from this NAS server.

If the idea is that all 6 machines will be editing 8K video at the same time from that server, you're in for disappointment. You'll need a lot better hardware than that (probably 4x the RAM and if you're doing sync writes, an optane for SLOG).

 

[jgreco]

128GB DDR4 3200
WD SN850 2TB Nvme for L2ARC

Well, that's either too much L2ARC or not enough memory. 10:1 is the absolute maximum suggested, and 5:1 is the typical starting point. L2ARC is not particularly useful for sequential workloads either, so loading up on real RAM is often better.

WD SN850 2TB Nvme for SLOG

This is nonsensical. What is the purpose of this? Either you need SLOG, in which case you should get a proper SLOG device, which an SN850 is not, or you don't need SLOG. Do you have a requirement for sync writes? Sync writes are always slower than async, so, if you want high performance, the most obvious choice is to avoid sync writes if you can.

hould I go for striped Vdev Config instead of raidz2? I need as much space as I can have.

Sure, as long as you have some other form of backup of the data. ZFS needs redundancy in order to protect your data, and striped provides no redundancy. Expect occasional downtimes due to disk failures.

Two RAIDZ1 vdevs would give you maximum space with minimal redundancy. Performance could be okay. However, the "RAID5 is dead" meme applies.

Two RAIDZ2 vdevs gives you good space utilization with reasonable redundancy. Performance could be okay.

Twelve mirror vdevs gives you moderate space with minimal redundancy, but maximal performance.

Eight three-way mirror vdevs gives you the least space but good redundancy, and good performance.

A QSFP28/QSFP+ NIC, Probably going for Chelsio (Pls suggest)

Okay, so, is that 10G, 25G, 40G, 50G, 100G, or what?

 

[sretalla]

is that 10G, 25G, 40G, 50G, 100G, or what?

I guess, this might be the clue:

at least 4000MB/s read speed.

 

[vineesh]

Will the editing be done locally on the workstation, or directly off of the NAS?

Directly from NAS, but in case of an emergency I will cache it into workstation NVME.

 

[vineesh]

C

If the idea is that all 6 machines will be editing 8K video at the same time from that server, you're in for disappointment. You'll need a lot better hardware than that (probably 4x the RAM and if you're doing sync writes, an optane for SLOG).

Copy, so what is your suggestion for the hardware and storage config.
Thank you

 

[vineesh]

Okay, so, is that 10G, 25G, 40G, 50G, 100G, or what?

QSFP+ and QSFP28 both are capable of 40Gbps. SFP28 will handle up to 100Gbps.

edit: QSFP28 instead of SFP28

 

[jgreco]

QSFP+ and SFP28 both are capable of 40Gbps. SFP28 will handle up to 100Gbps.

Incorrect, but thanks for the "explanation". SPF28 will not handle up to 100Gbps. It also will not handle 40Gbps. So two of those three things you said are wrong.

SFP28 will handle up to 25Gbps (typically including 1Gbps and 10Gbps). QSPF+ will handle 40Gbps (or, more precisely, 4x any SFP+ data rate). QSFP28 will handle up to 100Gbps (or, more precisely, 4x any SFP28 data rate).

So we circle around to the thing I was actually asking you, which is WHAT exactly are you planning to use for your ethernet?

 

[vineesh]

Well, that's either too much L2ARC or not enough memory. 10:1 is the absolute maximum suggested, and 5:1 is the typical starting point. L2ARC is not particularly useful for sequential workloads either, so loading up on real RAM is often better.

Then maybe I can go with 256GB RAM and 1 TB NVMe.

This is nonsensical. What is the purpose of this? Either you need SLOG, in which case you should get a proper SLOG device, which an SN850 is not, or you don't need SLOG. Do you have a requirement for sync writes? Sync writes are always slower than async, so, if you want high performance, the most obvious choice is to avoid sync writes if you can.

So I will avoid SLOG totally for the performance's sake.

Twelve mirror vdevs gives you moderate space with minimal redundancy, but maximal performance.

Can you please explain the configuration?

Two RAIDZ1 vdevs would give you maximum space with minimal redundancy. Performance could be okay. However, the "RAID5 is dead" meme applies.

I was thinking about using this but now I will wait for the responses.

 

[vineesh]

Incorrect, but thanks for the "explanation". SPF28 will not handle up to 100Gbps. It also will not handle 40Gbps. So two of those three things you said are wrong.

yeah my bad, I meant QSF28

So we circle around to the thing I was actually asking you, which is WHAT exactly are you planning to use for your ethernet?

A NAS with 4000 MB/s Read. I am using 6 Machines. Also, I know I can't get enough throughput for all the machines at the same time. And mostly all of the camera RAW footages will not exceed 800MB/s on a single stream. When the VFX plates come into play it is a different game. Then a single stream alone will be 2000MB/s

 

[ChrisRJ]

If I may be so bold as to suggest getting professional support into the game. In my view this is certainly an advanced scenario and in a commercial context. The price for a consultant who has experience will easily be outweighed by avoiding the risk to spend a lot of money on something that doesn't work as expected.

 

[vineesh]

If I may be so bold as to suggest getting professional support into the game. In my view this is certainly an advanced scenario and in a commercial context. The price for a consultant who has experience will easily be outweighed by avoiding the risk to spend a lot of money on something that doesn't work as expected.

makes total sense, but I'd really love to learn about this. Most of the IT consultants I approached were trying to sell me Linux machines with RAID configured. They were not at all interested in even discussing Truenas. For them, Truenas is for hobbyists and some end users.

 

[ChrisRJ]

For them, Truenas is for hobbyists and some end users.

Sorry to hear that you had such disappointing experiences. I would argue that anyone who positions TrueNAS as primarily for hobbyists is certainly not worth their money. ZFS, and in that sense TrueNAS as well, comes from the high-end enterprise market. So historically you would easily pay a 6/7-digit-figure amount of money for a ZFS-based storage system. That is as far away from hobbyists as you can get (except for mainframes probably).

 

[Samuel Tai]

@vineesh, some VARS build scalable ZFS systems for media production. See https://www.owcdigital.com/products/jupiter-callisto. This is actually based on the previous FreeNAS UI.

 

[jgreco]

For them, Truenas is for hobbyists and some end users.

Well, at least now you know which IT consultants have no effin' clue. Consider it an upside.

 

[vineesh]

Well, at least now you know which IT consultants have no effin' clue. Consider it an upside.

Yeah, I am collecting a few more info.

@vineesh, some VARS build scalable ZFS systems for media production. See https://www.owcdigital.com/products/jupiter-callisto. This is actually based on the previous FreeNAS UI.

The site itself says ethernet up to 10Gbps, but thanks for the suggestion.

 

[vineesh]

Eight three-way mirror vdevs gives you the least space but good redundancy, and good performance.

4 vdevs of 6 drive raidz2 for redundancy and performance balance is my new idea for the config. Any thoughts on this? Is there a suggestion for a particular CPU and board for this kind of workload?

Does truenas support RoCE standard on NICs?

 

[ChrisRJ]

Perhaps it's only me. But I think it would make sense to do some more formal sizing. So far the requirements have been described in some level of detail. But it would be helpful to quantify what editing 8k videos directly on the NAS means in terms of required network bandwidth and latency, as well as the same for disks. My gut feeling is that spinning HDDs may not be sufficient.

 

[vineesh]

So, it seems a NAS with HDDs will never achieve the kind of throughput I need.