Need sanity check for my first FreeNAS setup

[coulix]

Hello there,

I am here to share my first FreeNAS setup and get some feedback on what I may be doing wrong.
I am a dev by trade with some affinities with hardware, currently the tech lead of a 30 people company.

This startup happens to be in social media, more specifically video production. I started with a 15 TB Synology NAS that reached the 90% capacity in 6 months. It is time to scale up.

Being a hardware enthusiast I went on building my own NAS case with (pause a second) non-enterprise hardware.

Basically an AM4 motherboard with a Ryzen 1700x, 32 GB of ram (EEC), 8 x 10TB, 1 SSD (used a pcie card to add +4 sata to the existing 8) and a 10 Gbe network card.

Storage volume is ZFS2 pool.

Regarding our usage, people will work on a video for a week, doing video editing from the NAS and then once published, the edited video, adobe premier project, raw videos import stays there untouched, archived.

Should I use two volumes, one 2TB volume (Raid 0?) with performance in mind and the rest of the drives in ZFS2?

Any other tips ?

 

[tvsjr]

First, I'd suggest reading up on the basics of ZFS. It's not ZFS2... it's RAID-Z2. As far as your pool configuration, it all depends on your performance requirements. One or two 720p videos being edited at once is entirely different from a dozen 8K videos being edited simultaneously. I would set it up and do some testing starting with an 8-disk RAID-Z2 array, see if you get the performance you need.

 

[coulix]

Sorry, bad etymology, I meant RAID-Z2 indeed.

Regarding videos, lets say, at max, by the end of the year 15 people editing 1080p @60fps plus a few 60 GB file transfer.

 

[tvsjr]

You're talking about 15 people editing on the NAS simultaneously? The IOPS limitations will likely kill you. You can try it, but you probably need to be looking at striped mirrors... or a small SSD pool for your "online" content with a RAID-Z2 for your archive.

 

[Nick2253]

Editing video content live from a NAS is a pretty demanding operation. 15 simultaneous edits is going to require a very high performance setup. If you are working with raw files, you'll probably be bandwidth limited. Though assuming you have enough bandwidth (which is easy to fix with 10Gb ethernet), like @tvsjr said, your limit is going to be IOPS. For ZFS, to improve IOPS, you need more vdevs - hence the striped mirrors recommendation.

I would recommend looking at SSDs. Your cost per GB will be higher, but you'll need significantly fewer drives to reach your required performance. It might make sense to have a "scratch" space in SSDs where you edit the files, and an archive space for completed projects.

A couple comments about your specific build:

• AMD processor - AMD processors work (there are members using AMD setups here), but they are definitely the road less traveled. Also, Ryzen support is very new in FreeBSD (only started in FreeBSD 11), so you're definitely on the cutting edge. If I were advising a client on this setup, I would strongly recommend going down the Intel road, especially considering this appliance is critical to the existence of the business.
• NIC - I'm not sure exactly what motherboard you're using, but I'm assuming it's got a Realtek NIC on it. Realtek NICs are known to have weird issues with FreeBSD, so I would recommend getting an Intel NIC PCIe card (or a 10G card if you are going 10G).
• PCIe SATA Card - I'm assuming this is a pretty basic consumer-grade add-on card. These chipsets are not very well tested, and can have weird behavior when put under the stress of a NAS, and I would recommend you look at a proper SAS card. Second-hand SAS cards can be had for pretty darn cheap.

 

[coulix]

Thank you Nick for your insights.

I added a 10gbe network card, Intel one.
The extra 4 x sata is a maxwell chip and seems to be supported.

I have an emotional link to AMD but it is true an intel with a pro board would have been safer.

 

[Nick2253]

I have an emotional link to AMD but it is true an intel with a pro board would have been safer.

I understand the emotional connection, but your employer is relying on you to provide objective recommendations and decisions.

I bring this up, because this is one of the most common issues I have with my team. Many of them are young, and have a lot of self-taught experience with hardware and software (something I specifically look for when hiring, and something I try to encourage them to continue while they work for us). With this self-taught experience tends to come an emotional connection to the choices they've made, often because of the time and hardship they've put into making it work for them. However, our job is to provide the best solution we can for the company, not the best solution for us. When all other factors are equal, I try to go with those solutions, because my team has experience with them. But in the real world, they are rarely equal, and often, the more expensive solution up front is ultimately more cost effective in the long run given the requirements of the business.