Fileserver for a small film studio - criticise my plan please!

[subsonik]

Hi!

I’ve been tasked with speccing up a fileserver for a small but rapidly expanding film studio. I have decided on a rough specification as listed below, I’d be interested to hear what you lovely people think please. Criticisms welcome!

Requirements are:

* Users will be editing RAW footage direct from the server, so it must be capable of 500MB/s per user, for 2-3 users at a time
* There must be room for expansion, in terms of adding more HDD storage later on as our archives grow
* It must be kept to a semi-reasonable price, i.e. below 35000 Norwegian kroner (around $3-4k) - not including disk price, as we’re buying those separately
* It must also be capable of doing some video encoding/transcoding - I’m not a fan of this idea because a fileserver should do the job of a fileserver and nothing more, but we’re working to a budget and can’t afford another dedicated encoding server right now
* 6TB of SSD scratch space, capable of 1500MB/s+ for editing over the 10GbE network
* 12TB of HDD archive space, project files will be moved from scratch to here when the project is completed

My decisions so far:

* We’re going with a custom built Supermicro server from a vendor called Nextron in Norway, Dell is another option but the options in the Norwegian Dell store are somewhat limited, it’s hard to find a Dell server with 16+ 3.5” drive bays
* OS will either be FreeNAS, FreeBSD with me configuring ZFS and everything else manually, or Linux+mdadm if ZFS doesn’t cut it with the speeds
* Supermicro chassis, capable of housing 16x 3.5” and will come with 3.5” to 2.5” drive bay converters
* 4x 2TB Samsung 850 EVOs in a RAID-Z1, 500MB/s sequential read/writes each, from what I understand, these drives in this RAID config should be capable of at least 1500MB/s sequential, other RAID configs are possible if not
* 4x 4TB HGST enterprise drives in RAID-Z1 or RAID-Z2 for archive
* Xeon E5-2620v4 2.10Ghz, should be plenty for a fileserver, along with some occasional video encoding
* 32GB ECC RAM
* 2x 80GB SSDs in RAID-1 for OS
* Intel X540 dual 10GbE NIC (compatible with the latest stable FreeNAS, which is FreeBSD 10.3, afaik?)
* Server will be connected to a Netgear XS708T 10Gbps switch via 2x Cat6A cables in a LACP LAG link aggregation group
* All connected to a CyberPower Online UPS
* We’ll eventually be purchasing an LTO-7 tape drive for archiving old project file when we run out of HDD archive space
* The server alone comes to around 34000 kroner ex VAT ($4000)

What do you guys/gals think? :)

Thanks!

 

[Jailer]

This sounds like a very serious and mission critical project. Have you considered contacting iX systems for a quote on a TrueNAS system?

 

[Ericloewe]

This sounds like a very serious and mission critical project. Have you considered contacting iX systems for a quote on a TrueNAS system?

This advice aside, a few points come to mind:

4x 2TB Samsung 850 EVOs in a RAID-Z1, 500MB/s sequential read/writes each, from what I understand, these drives in this RAID config should be capable of at least 1500MB/s sequential, other RAID configs are possible if not

Doesn't seem like a very good setup. I'd try two to three mirror vdevs with 850 Pros or similar.

* Users will be editing RAW footage direct from the server, so it must be capable of 500MB/s per user, for 2-3 users at a time

That's going to be painful with any system. The third user complicates things as it goes beyond 10GbE. Link aggregation for 10GbE is going to have the same issues as 1GbE link aggregation.

* 4x 4TB HGST enterprise drives in RAID-Z1 or RAID-Z2 for archive

If you need 12 TB, I'd suggest starting with 4x 8TB. It'll give you a bit more space, but 4x 6TB in RAIDZ2 wouldn't give you the 12TB you need.

* Xeon E5-2620v4 2.10Ghz, should be plenty for a fileserver, along with some occasional video encoding

A Xeon E5-1650 v4 is probably a better choice overall. 1620 if the budget is really tight.

* Intel X540 dual 10GbE NIC (compatible with the latest stable FreeNAS, which is FreeBSD 10.3, afaik?)

Should be.

* We’ll eventually be purchasing an LTO-7 tape drive for archiving old project file when we run out of HDD archive space

That is not trivial in FreeNAS 9.10 and is going to be even less so on FreeNAS 10. That said, tape isn't as cost-effective as it used to be and you don't get any of ZFS' protections.

 

[subsonik]

This sounds like a very serious and mission critical project. Have you considered contacting iX systems for a quote on a TrueNAS system?

I contacted iX-systems, who quoted me around €17,000 for a not too dissimilar spec to what I mentioned in my post (which was ridiculous), I asked them for something cheaper and they have yet to get back to me, which was a couple of weeks ago.

Doesn't seem like a very good setup. I'd try two to three mirror vdevs with 850 Pros or similar.

This would cost a little bit too much money for us. It's scratch space so it doesn't need a helluva lot of redundancy, just maybe allowing for one drive failure. Its contents will backed up the archive anyway.

That's going to be painful with any system. The third user complicates things as it goes beyond 10GbE. Link aggregation for 10GbE is going to have the same issues as 1GbE link aggregation.

What issues does 1GbE link aggregation have? As I understand it, I can setup round-robin aggregation so that the third user will be sharing with one of the other users and help to saturate that particular 10GbE link, which should provide both with 500MB/s each?

A Xeon E5-1650 v4 is probably a better choice overall. 1620 if the budget is really tight.

Thanks, I'll look into the E5-1650v4. :)

That is not trivial in FreeNAS 9.10 and is going to be even less so on FreeNAS 10. That said, tape isn't as cost-effective as it used to be and you don't get any of ZFS' protections.

I did a quick google search for the feasibility of LTO tapes under FreeNAS and it seems like they are indeed problematic, although those threads were from a few years ago. I guess I need to go back to the drawing board with this. :(

Thanks!

 

[Ericloewe]

What issues does 1GbE link aggregation have? As I understand it, I can setup round-robin aggregation so that the third user will be sharing with one of the other users and help to saturate that particular 10GbE link, which should provide both with 500MB/s each?

In practice, benefits are only significant starting around 10 users.

This would cost a little bit too much money for us. It's scratch space so it doesn't need a helluva lot of redundancy, just maybe allowing for one drive failure. Its contents will backed up the archive anyway.

I'm thinking mostly about their performance and long-term reliability of the drives, not really pool reliability. TLC drives tend to drop performance like crazy when their SLC cache runs out.

 

[anodos]

I believe the truenas price includes a nice software support contract. If you think that's expensive, you should shop around more. :D

What filesharing protocol?

 

[tvsjr]

Personally, I'd suggest that a NAS isn't the right answer for online editing. Most online editing (especially handling raw video... based on your throughput requirements, I assume you're working in 4K?) is done on large, fast, directly-attached disk arrays. Network based filers are used for archive storage or to share edited/rendered clips.

Build an array of sufficient performance for each editing system, then build a network filer for archiving and sharing.