Building a NAS for Video Storage and Editing - Seeking Opinions and Advice

[adamoconnor]

Hello everyone! While I'm not exactly new (or anywhere near an expert, either) at servers and basic NAS usage, I'm planning my first real endeavor into TrueNAS as all of my experience prior has been w/ WinServer or unRAID for a high-density storage server I am planning to build. My current environment consists of 2x Windows Server machines, one hosting Hyper-V appliances for several different server instances and the other mainly handles NAS tasks via a RAID 10 SSD volume and a RAID 5 HDD volume. Up until now, these have been working great for our use, but we have lately started to need high-capacity video storage as we have hired a media manager and they are managing to fill up the rest of our storage with raw video files.

That said, as a proof-of-concept machine and one that will certainly be upgraded with more serious hardware, I am looking at doing the following "budget-build":

3rd Gen i7 3770K
32GB DDR3
6x WD RED Plus 4TB
2x SATA SSD (type and size not yet considered as I am learning adding SSD may not be of benefit to me)
Above drives attached to an LSI 9207-8i
Intel X520-T2 for peer-to-peer between server and video editing rig
Intel I340-T4 for LAGG'd access to rest of 1GBE network (yes, my switch supports aggregation)

I understand that the above is partially consumer-grade, particularly in CPU/Mobo/RAM, and probably unsafe in some way, shape or form, but like I said, this is a proof-of-concept idea to make sure we're going to be barking up the right tree.

My original plan was to place the disks into two separate RAIDZ1 vdevs so that I can stripe the data across the two vdevs to hopefully gain some performance and retain some safety in case a drive fails. I'm quickly learning though, that this may not be the right way to go about this. For some reason, there seems to be a consensus that RAIDZ1 is evil and should never be used but rather I should be going for Z2. Is there any practical reason for this other than the minimum of two drive failure safety? It seems like I can expect to lose some performance by going this route as I would no longer be striping across two vdevs.

Ultimately, what I am looking to get out of this server is
1.) High density storage for a very reasonable $/TB
2.) Fast enough drive access to be able to saturate a 10GBE link ( I don't think I'll quite make it at only 6 drives, but based on some napkin math it seems I should be able to at say 8 or 10...)

The server is going to be more about bulk storage, and less about actually editing video off of it. I'm expecting our editor to draw several-gig files off of it at any moment, and then the next moment dump a whole 2TB onto it. The dumping of the 2TB onto the server is what concerns me, as if it takes 8 hours to do I'll be losing a lot of productivity.

So this is where the opinion/advice part comes into play.

Are my aspirations something even practically feasible?
Is there something I am saying/planning in the above that I should be beaten with a short stick over for even thinking of?
What is the meaning of life?

As far as network and drive array tuning go, I understand that I'll need to do some work on both in order to maximize my performance. I'm not necessarily asking for advice on that front as I realize it's very subjective to the specific build, but as I understand it now, I'll be needing to enable jumbo packets on my 10GBE network, and set up sector sizing on TrueNAS to match the drive's sector sizes. Please correct me if I'm wrong on those.

Thanks in advance for your advice.

Adam

 

[NugentS]

Raid 1 / Z1. Think it through. If a disk fails you now have no parity / resiliency. Thus ZFS cannot repair any issues that occur. Also, the most stressful time for the disks in a pool is during a resilver (which takes quite some time) and loss of another disk would be fatal for the pool. Thus Z2 or Z3 is considered much safer.
In a mirror, the resilver time is much much quicker so whilst resiliancy is per vdev you are at risk for less time.

Why the crappy hardware - if you are going to do this - do it on the correct hardware in the first place. CPU, Memory etc are a PITA to change. Disks are easy (even if the cost of the disks does mount up). Just put it in a decent case with lots of spare drive slots - even if at first you can't use them.

ZFS Z1/2/3 is quite good at writing sequential files so a Z2 I would have thought should be fine. Mirrors supply extra IOPS which isn't that important for you. Note however that the pool should be 50% used to maintain performance. Try to stay below 50% - so that means lots of extra disks in the first place. 6TB disks are probably not the best bang for the buck at the moment (depending on where you are) I did some work recently for a customer and we ended up with 10TB as the most efficient option.

Moving 2TB around quickly is always going to be tricky

 

[adamoconnor]

Thanks for your input.

Raid 1 / Z1. Think it through. If a disk fails you now have no parity / resiliency. Thus ZFS cannot repair any issues that occur. Also, the most stressful time for the disks in a pool is during a resilver (which takes quite some time) and loss of another disk would be fatal for the pool. Thus Z2 or Z3 is considered much safer.
In a mirror, the resilver time is much much quicker so whilst resiliancy is per vdev you are at risk for less time.

When you say mirror, can you elaborate? I have seen a few instances where if there were 6x 4TB drives, there would be 3x mirrored vdevs. Is this what you mean?

Why the crappy hardware - if you are going to do this - do it on the correct hardware in the first place. CPU, Memory etc are a PITA to change. Disks are easy (even if the cost of the disks does mount up). Just put it in a decent case with lots of spare drive slots - even if at first you can't use them.

The hardware as noted is going to be replaced in production. For now, I'm just trying to evaluate if this route is at all going to work. The case is going to be one of rosewill's inexpensive but seemingly good L4500U chassis. Even at that, when the time comes to actually implement this, I'm still on a budget, so depending on what kind of hardware you have access to, for all I know it's probably going to still be "crappy" hardware. Believe me, if I had the kind of coin to invest in EPYC or high core count Xeon, I would.

ZFS Z1/2/3 is quite good at writing sequential files so a Z2 I would have thought should be fine. Mirrors supply extra IOPS which isn't that important for you. Note however that the pool should be 50% used to maintain performance. Try to stay below 50% - so that means lots of extra disks in the first place. 6TB disks are probably not the best bang for the buck at the moment (depending on where you are) I did some work recently for a customer and we ended up with 10TB as the most efficient option.

Moving 2TB around quickly is always going to be tricky

I was actually looking at qty 6 of 4 tb disks. So far, the $/TB seems to be least at 4TB quantities - in WD Red flavor anyways. If we bring space efficiency into play though, the 10TB is what I would pick, too.

 

[NugentS]

Be very careful of WD Red Drives - at that size point they may be SMR which you do NOT want to buy. ZFS and SMR do NOT mix.
Mirror = drives, one copy of the other.
This can then be striped across multiple mirrors. each mirror is a vdev.

The point is that resiliency is within the vdev so:
10 disk - 5 mirrored vdevs striped = 5 Pairs of disks - each mirror can lose a single disk with no issues. But if you lose more than one disk in a mirror the pool is toast. Also its a 50% loss due to resiliency
10 Disks in Z3 you have lost 30%, can only lose 3 disks but they can be any 3 disks.

 

[adamoconnor]

Be very careful of WD Red Drives - at that size point they may be SMR which you do NOT want to buy. ZFS and SMR do NOT mix.
Mirror = drives, one copy of the other.
This can then be striped across multiple mirrors. each mirror is a vdev.

The point is that resiliency is within the vdev so:
10 disk - 5 mirrored vdevs striped = 5 Pairs of disks - each mirror can lose a single disk with no issues. But if you lose more than one disk in a mirror the pool is toast. Also its a 50% loss due to resiliency
10 Disks in Z3 you have lost 30%, can only lose 3 disks but they can be any 3 disks.

Ah, yes, I see SMR has caused a lot of headaches. These drives are CMR. From what I gather, if it were you, you'd probably be doing a Z2 or Z3, right?

 

[NugentS]

I would indeed be doing Z2 or Z3 and I would be using the bigger drives
I would plug a relatively cheap SSD into a SATA port and use that to boot from - maybe even a 2nd hand Intel DC 40GB or similar.
Your 2 SSD's - I think are pointless. But spend the money on RAM instead

 

[tylerhawes]

I work in film and own a facility. We have a TrueNAS server hosting data for editing, vfx, color grading. Some thoughts:

• Buy bigger sized disks. I can't imagine doing anything less than 8TB disks right now, and would look really hard at 12TB or 16TB. Whatever you think you need for space, triple it. Because you want to drive on the top half of the tank (keep your pools close to 50% full, 80% absolute most), and because as soon as you have more storage space you will find new ways to fill it up.
• You should either do RAIDZ2 or Mirrors. Z2 will give you more storage efficiency, but it's IO performance is poor. It's fine if you're doing almost all sequential read/write and larger files and only have one or two users at the same time. If you're just keeping a copy of video and audio files there, that's OK. But if you're going to work off the volumes directly, especially with more than a couple of users, than the low IO of Z2 becomes a problem. It's not just a question of if the drives can spool the data fast enough, but it's one of latency in general. When you're editing, you can really feel the latency because you hit play or stop or try to jump to another part in the clip, and there's just this slight delay. It is really annoying at first, but in a professional setting it adds up over time to lost productivity = lost $$$. In that use case, you want to switch to a stripe of multiple mirrored volumes, which is what we have. Your IO is MUCH better. Yes, you are now losing half your raw storage to parity. Just try to keep it in perspective of how amazingly cheap storage is now, and that you are buying performance.
• If you go Z2, it's N+2 drives, where I think N is recommended as a divisible by 2 integer number (2, 4, 6, 8... +2 = number of drives per Z2 pool). If you're only talking a few drives, like 6 or 8, that means you'll only have one Z2 volume (i.e. 4+2=6 or 6+2=8). If you have much more drives, you break it into two Z2 volumes and stripe a pool across them (i.e. 16 drives may be 2 x 8-drive volumes of 6+2 Z2).
• I think it's a good idea to have a hot spare drive loaded in the system.
• I am of the crowd that would never build a production system without ECC memory. If just for playing around, home lab, etc., where you don't mind losing your data, OK I guess. But if this is work-related, I can't imagine justifying cutting that corner.
• On saturated 10GbE, keep in mind that aside from having a really good NIC (Chelsio is what we use and many recommend, may find best deal used on eBay), the biggest challenge here is your CPU. SMB sharing is mostly single-threaded, and each user in video editing is generally opening up a single transfer. In that case, the whole transfer between TrueNAS and your editor is loaded mostly in a single thread on a single CPU core. Therefore, you need to have the fastest single-threaded performance CPU you can get. This is contrary to what is usually looked for in server and workstation CPUs, and more akin to what gamers look for. So you may not want the most cores you can get, but the latest generation and highest clock speeds, while keeping in mind need to support ECC memory if this is for professional use. i.e. if I had only 4 users in a small workgroup of editors, I'd rather have a 6-core 3.2GHz CPU than a 16-core 1.8GHz CPU, since I know each user will probably keep a core busy, and then a couple more for good measure / background tasks / etc.

Lastly, all of this material is out there I'm just collating some of my findings. I've only been using TrueNAS a couple years myself. But, with respect, your questions and suggested build suggest that you've read a little of what is here and on Google, but haven't really scoured the history of threads and read everything relevant. I think you should take a couple days to do that and you'll have a lot more confidence.

 

[adamoconnor]

Thanks for your greatly detailed reply, Tyler.

• Buy bigger sized disks. I can't imagine doing anything less than 8TB disks right now, and would look really hard at 12TB or 16TB. Whatever you think you need for space, triple it. Because you want to drive on the top half of the tank (keep your pools close to 50% full, 80% absolute most), and because as soon as you have more storage space you will find new ways to fill it up.

To be honest with you, as it stands, it's not really all that possible for us to go up in disk size right now. 4TB disks is already a big upgrade for us. I completely understand where you're coming from here, but the only way for us to truly be able to place more disks in an array to give us a fighting chance with read and write speeds is to go the cheaper-per-drive route to add more drives to the system from the get-go. That said, if there's another reason other than higher storage density that is going over my head, feel free to slap me in the face and let me know.

• You should either do RAIDZ2 or Mirrors. Z2 will give you more storage efficiency, but it's IO performance is poor. It's fine if you're doing almost all sequential read/write and larger files and only have one or two users at the same time. If you're just keeping a copy of video and audio files there, that's OK. But if you're going to work off the volumes directly, especially with more than a couple of users, than the low IO of Z2 becomes a problem. It's not just a question of if the drives can spool the data fast enough, but it's one of latency in general. When you're editing, you can really feel the latency because you hit play or stop or try to jump to another part in the clip, and there's just this slight delay. It is really annoying at first, but in a professional setting it adds up over time to lost productivity = lost $$$. In that use case, you want to switch to a stripe of multiple mirrored volumes, which is what we have. Your IO is MUCH better. Yes, you are now losing half your raw storage to parity. Just try to keep it in perspective of how amazingly cheap storage is now, and that you are buying performance.

The Mirror route is likely the one I am going to take. As it stands, our editor will be using this server as a bulk storage medium, not as a working medium except in a few very rare cases where they may grab a file off of there and drop it in their timeline so that they can reference an older video in a newer one. That said though, transfer times are important, and while there's (as I understand it anyways) likely to be a small difference between Z2 and Mirrors in a 6-8 drive setup, as time goes on and we add more drives into the system, I imagine there to be more speed delta favoring the mirror architecture vs the Z2.

Thanks also for the other notes you have provided. In regards to your storage setup, what is your processor/ram combo that you are using? I would have thought that in a testing environment, a 3.7GHz i7 w/ 8 threads would have been plenty to play with when it comes to these kinds of transfers. I do realize that the older generation of the processor does mean that 3.7GHz 8 years ago and 3.7 GHz today are two completely different things. All said though, I don't have $10,000 to throw at this whole solution. As it stands I can get away with maybe $1,500 so that means I'm relegated to old consumer hardware for testing and then roughly 8 year old Xeon v2 processors with DDR3 ECC in production.

Lastly, all of this material is out there I'm just collating some of my findings. I've only been using TrueNAS a couple years myself. But, with respect, your questions and suggested build suggest that you've read a little of what is here and on Google, but haven't really scoured the history of threads and read everything relevant. I think you should take a couple days to do that and you'll have a lot more confidence.

I'd equate the research I've gone through so far to "drinking from a firehose". I've been reading through the forums a fair bit but have forgotten most of it in information overload. This thread is a way for me to get some baseline ideas of where people in a similar situation are so that I can get my feet planted firmly. Of course, as components are bought and configured, I'll probably be spending a solid week reading threads here to get everything up to speed. The flexibility that TrueNAS has is in my opinion a bit of a double-edged sword as all of the great features each have their own learning curve. Of course if I wanted the easy way out, I'd buy a Synology. But, I like a challenge.

Thanks again for your thoroughness in your reply.