NAS build for Video editing and motion graphics

[esbowman]

So, I've put together a list of parts and went over the handy guides here. Lots of info to think about. I'd like to post my list here and get a few eyes on it to make sure I'm headed in the right track. I was pricing a Synology box (1515+) and it was quickly hitting $2500 for only 16TB of storage, and that didn't even include 10GbE ports. Needless to say, now I'm here. :)

What I need:

Shared storage for two Windows machines. They will primarily be sharing 3d assets, textures, video footage, and various other assets related to creating motion graphics. I will be doing some video editing as well. I'm currently using two external RAIDS (one mirrored for backup) and have about 8TB of data that I need to transfer to the new NAS. I think I'd like to double or triple the current storage I have. I'm taking on more 4K work and VR videos as well, both of which require much more space. I'm considering using 6 drives in a RAIDZ2 so I have two drives for redundancy. If I use 8TB drives, I should get roughly 32TB of storage (16TB for redundancy). Finally, instead of buying an expensive 10GbE switch, I'm planning to use two 10GbE ports on the NAS to connect to my Windows machines, both of which have 10GbE. From what I gather that should get me about 700-800MB/s, which should be good for asset sharing, editing etc. Most of the time only one PC will be accessing the NAS at a time, sometimes two. I will also have 1 or 2 other computers (an old Mac Pro and newish MacBook Pro) accessing the NAS, but only for backups.

Here's the current list I've compiled:

Motherboard: Supermicro Mini-ITX SoC Xeon D-1521 4-Core, FCBGA 1667 Motherboard - X10SDV-4C-TLN2F-O
RAM: Crucial - 16GB (2 x 8GB) Registered DDR4-2133 Memory (Plan to install 32GB)
Case: Fractal Design Node 304
Power Supply: EVGA SuperNova 550 G3
Drives: Western Digital Red 8TB NAS Hard Drive 5400RPM 128MB Cache (Will buy 6 Drives)
Boot Drive: Samsung - 850 EVO 250GB M.2-2280 Solid State Drive (Will use this as the boot drive)
UPS: Have an existing UPS...CyberPower brand I believe. Might need to buy a new one.

Questions:
1) Would love to know thoughts about this build...what should I change? Things I should add?
2) I have an old external RAID that I plan to upgrade for mirroring the NAS. Is this a good idea? In a perfect money-filled world I would just build two NASs and mirror it.
3) Is it realistic to think I can work off the NAS and obtain a speed of somewhere between 700-800MB/s?
4) Regarding the purchasing of drives, I've heard you want to try to buy different batches. Like some from Newegg, some from Amazon, etc. to prevent buying all from a bad run. Is this worth sweating over?
5) Finally, I picked the Supermicro board because it had 2 built-in 10GbE ports. However, I noticed the following paragraphs from the Hardware PDF. Should I consider a different board?

Intel 10GbE NICs are said to suffer from a less-than-perfect FreeBSD driver that makes it difficult for them to achieve the speeds expected from 10GbE. Currently, Chelsio 10GbE NICs are the preferred choice for use with FreeNAS, and are the models shipped with iXsystems hardware.

Really looking forward to building this, and everyone's advice. Thanks so much!

 

[SweetAndLow]

1. Build looks fine
2. Having a backup is a great idea, always!
3. If you want to work off the Nas iops are more important. So striped mirrors would be the layout.
4. That batch stuff is nonsense. You just need to test and stress your drives before using them. Badblocks test is a great way to do this.
5. Intel NICs get your about 500MB/s after tuning. Freebsd 11 has some upgraded drivers so that might be better now but untested. 10gig in general is harder and most people who have it working are not your average Joe. They are usually professionals.

Sent from my Nexus 5X using Tapatalk

 

[MrToddsFriends]

RAM: Crucial - 16GB (2 x 8GB) Registered DDR4-2133 Memory (Plan to install 32GB)

For future expandability better start with 2 x 16GB modules when the plan is to install 32GB now or in the near future.

 

[iammykyl]

Please check,
"M.2 Socket (Shared with I-SATA0 when a SATA device is installed in M.2)" Does this mean you will only have 5 usable SATA port, or bandwidth will be effected?
You may have a reason to list/buy from Amazon, but you will save a little at Newegg.
MB. https://www.newegg.com/Product/Product.aspx?Item=N82E16813182973
Case. https://www.newegg.com/Product/Prod...ractal_Design_Node_304-_-11-352-027-_-Product
HD. https://www.newegg.com/Product/Prod...2235063&cm_re=WD80EFZX-_-22-235-063-_-Product
PSU. ( after rebate) https://www.newegg.com/Product/Prod...=EVGA_SuperNOVA_550_G3-_-17-438-095-_-Product
Review on the board, has a link as well to CPU test.
> https://www.servethehome.com/supermicro-x10sdv-4c-tln2f-review-xeon-d-1520/

 

[esbowman]

Thanks to everyone here. All great advice. I might look at other boards, now that it seems the Intel 10GbE NICs might be a bottleneck. I definitely want solid 10GbE performance right out of the box. I have answers and additional questions below. Love this community already. :)

2. Having a backup is a great idea, always!

Yeah I should have worded my question a bit better. I was specifically asking if it were ok to plan on mirroring the entire NAS to an external RAID (USB 2) that I already own?

3. If you want to work off the Nas iops are more important. So striped mirrors would be the layout.

Thanks for this, I'll do some researching on how exactly I need to set that up.

4. That batch stuff is nonsense. You just need to test and stress your drives before using them. Badblocks test is a great way to do this.

That's what I assumed. Thanks.

5. Intel NICs get your about 500MB/s after tuning. Freebsd 11 has some upgraded drivers so that might be better now but untested. 10gig in general is harder and most people who have it working are not your average Joe. They are usually professionals.

Bummer. I was kind of setttled on that motherboard. I'll keep looking I suppose.

For future expandability better start with 2 x 16GB modules when the plan is to install 32GB now or in the near future.

Thanks, but I plan to actually install 32GB at the beginning. I have 4 RAM slots on this board....if I go that route. Not sure now that I know the intel NICs are possibly a bottleneck.

"M.2 Socket (Shared with I-SATA0 when a SATA device is installed in M.2)" Does this mean you will only have 5 usable SATA port, or bandwidth will be effected?

Good catch. I'll check into that. If so, I'll just do two USB Flash Drives instead.

You may have a reason to list/buy from Amazon, but you will save a little at Newegg.

I was just trying to be a good newbie and do as the Forum suggested. :) They said Amazon links were the best thing to help the forum. haha. I actually plan to buy as much of this as I can from my local Microcenter.

Review on the board, has a link as well to CPU test.

Thanks much!

 

[SweetAndLow]

Thanks to everyone here. All great advice. I might look at other boards, now that it seems the Intel 10GbE NICs might be a bottleneck. I definitely want solid 10GbE performance right out of the box. I have answers and additional questions below. Love this community already. :)


Yeah I should have worded my question a bit better. I was specifically asking if it were ok to plan on mirroring the entire NAS to an external RAID (USB 2) that I already own?


Thanks for this, I'll do some researching on how exactly I need to set that up.


That's what I assumed. Thanks.


Bummer. I was kind of setttled on that motherboard. I'll keep looking I suppose.


Thanks, but I plan to actually install 32GB at the beginning. I have 4 RAM slots on this board....if I go that route. Not sure now that I know the intel NICs are possibly a bottleneck.


Good catch. I'll check into that. If so, I'll just do two USB Flash Drives instead.


I was just trying to be a good newbie and do as the Forum suggested. :) They said Amazon links were the best thing to help the forum. haha. I actually plan to buy as much of this as I can from my local Microcenter.


Thanks much!

No 10gig setup will work out of the box. It takes lots of tuning to get full line speed. U don't have any 10gig stuff but there are lots of posts here with people messing with 10gig.

Sent from my Nexus 5X using Tapatalk

 

[esbowman]

No 10gig setup will work out of the box. It takes lots of tuning to get full line speed. U don't have any 10gig stuff but there are lots of posts here with people messing with 10gig.

Sent from my Nexus 5X using Tapatalk

Kind of confused now. The motherboard has two 10GbE ports. I planned to use two ethernet cables connected from each computer directly to the NAS for the speeds I need. I realize it won't be "out of the box" and will require some tuning, but I had assumed that since my two PCs have 10GbE NICs and the motherboard does too, I would be good to go in terms of hardware. Am I missing something?

 

[SweetAndLow]

Kind of confused now. The motherboard has two 10GbE ports. I planned to use two ethernet cables connected from each computer directly to the NAS for the speeds I need. I realize it won't be "out of the box" and will require some tuning, but I had assumed that since my two PCs have 10GbE NICs and the motherboard does too, I would be good to go in terms of hardware. Am I missing something?

That should work. Are they rj45 plugs or sfp?

Sent from my Nexus 5X using Tapatalk

 

[esbowman]

That should work. Are they rj45 plugs or sfp?

Sent from my Nexus 5X using Tapatalk

They are RJ45

 

[esbowman]

All the terminology is confusing me. Maybe I should be saying 10GBASE-T LAN? I assumed 10GbE was the overall technology.

What I have is a built-in 10GbE RJ45 port in each computer, and plan to attach those two PCs via Cat6 to the FreeNAS which will have two 10GbE RJ45 ports as well. That said, is it still not a good idea to use the Supermicro board since it has Intel 10GBASE-T Lan?

One more note, I'm pretty much forced to go with Cat6 due to the multi GPU setups I have. I have an ATX Tower with 4GPUs and therefore have no extra PCI slots for adding a 10GbE card with SFP ports. My second PC is a miniITX build with 2 GPUs...again...no room for adding a card. The GPUs are used for GPU 3D rendering...if curious.

 

[iammykyl]

That should work. Are they rj45 plugs or sfp?

Sent from my Nexus 5X using Tapatalk

Agreed.
SFP, > http://www.pcmag.com/encyclopedia/term/64582/sfp
SFP port (1000BaseSX) will not give you a better throughput, maximum throughput is based entirely on your NIC, in your case 10GbE .
If you could use the fibre cable, it would give you much greater distance than the copper, sorry don't know the exact figures.

 

[alpaca]

I am dealing with 10gb inconsistency betweens reads and writes with the Xeon D onboard nics. I would highly recommend, if rock solid 10gb networking speeds are important, to go with a Chelsio T4 or T5 add in card. They have both SFP+ and RJ45 variants - driver compatibility with FreeNas/FreeBSD is awesome and can totally support line rate 10gb if the underlying/connecting hardware can push it. With the onboard Xeon D nics, I am getting great write speeds (near line rate) but slow read speeds (1/4 speed) on striped pool of SSDs (for testing). The network performance is a non-issue with Chelsio nics.

 

[Evi Vanoost]

SFP is unnecessary for distances under 100ft. CAT6a or 7 should be sufficient.

I would recommend mirrors over RAIDZ2 for latency and throughput reasons. Also more smaller and faster drives instead of fewer larger drives will help. You should be able to push line speed with SSD but you won't with the 5400rpm drives, even in mirror you'll get at most 3x the speed of a single drive which have latency of 5ms (or about half a frame of video) per seek.

I see you pushing maybe 300IOPS in that configuration which is really really low. A good SSD cache will help but it doesn't kick in for streaming workloads. Go at least with a number of high quality 2TB 7200RPM or SSD.

Vendors I worked with over a decade ago recommended 14 drives at 7200RPM in mirror to handle ~4 streams of 1080p30 video editing. 4K is more than 100MB/s.

 

[Frank Huff]

Make sure the M.2 board and SDD you purchase are the NVME type like the 960 pro. At 4k resolution, you will be capturing around 1GB/min., and if you want to store the files uncompressed for editing, you will need the initial transfer rate to catch the raw data (if you're capturing/storing in real time.)

You will kick yourself for not using all SSD's for your new raid system. Plus independent read and write cache drives. 7200's can't compete with SSD on real-time capture.

If you can't afford it now, wait a bit. prices are dropping every day. Remember, time is money.

 

[Ericloewe]

Actually, both NAND and DRAM are seeing significant shortages, with the DRAM situation worsening due to a recent outage at a Micron plant.

NAND production is set to increase, but demand is higher than ever. Flagship smartphones are going to use up immense amounts of flash, enterprise can't get enough flash... Even relatively big players like Nintendo are having some trouble sourcing enough flash for their products.

 

[danb35]

Plus independent read and write cache drives.

There's no such thing as a write cache in ZFS, and adding a SLOG will slow things down in most circumstances.

 

[SweetAndLow]

There's no such thing as a write cache in ZFS, and adding a SLOG will slow things down in most circumstances.

Agreed! Slog is just a place to write data incase there is a sudden power loss. Everything about the write process is the same without one but when it's added it does an extra write to the slog device which will only be used to replay the transactions after power is restored. It basically makes sync NFS workflows faster.

Sent from my Nexus 5X using Tapatalk

 

[Ericloewe]

adding a SLOG will slow things down in most circumstances.

Not really, unless there are some sync writes and the SLOG manages to be so crappy that it's slower than the on-pool ZIL. In most cases, adding an SLOG is an expensive exercise that reduces reliability and has no impact on performance.

 

[esbowman]

SFP is unnecessary for distances under 100ft. CAT6a or 7 should be sufficient.

I would recommend mirrors over RAIDZ2 for latency and throughput reasons. Also more smaller and faster drives instead of fewer larger drives will help. You should be able to push line speed with SSD but you won't with the 5400rpm drives, even in mirror you'll get at most 3x the speed of a single drive which have latency of 5ms (or about half a frame of video) per seek.

I see you pushing maybe 300IOPS in that configuration which is really really low. A good SSD cache will help but it doesn't kick in for streaming workloads. Go at least with a number of high quality 2TB 7200RPM or SSD.

Vendors I worked with over a decade ago recommended 14 drives at 7200RPM in mirror to handle ~4 streams of 1080p30 video editing. 4K is more than 100MB/s.

Thanks man. So are you saying vendors recommended using 7 drives with another 7 drives mirrored...total storage roughly 14TB if using 2TB drives? I need to get close to 20TB of storage if possible, which is why I picked the larger drives. Maybe I should consider a different case and board so I can throw 8 or more drives in there. The different RAID configurations have me a little confused.

Make sure the M.2 board and SDD you purchase are the NVME type like the 960 pro. At 4k resolution, you will be capturing around 1GB/min., and if you want to store the files uncompressed for editing, you will need the initial transfer rate to catch the raw data (if you're capturing/storing in real time.)

You will kick yourself for not using all SSD's for your new raid system. Plus independent read and write cache drives. 7200's can't compete with SSD on real-time capture.

If you can't afford it now, wait a bit. prices are dropping every day. Remember, time is money.

I will not be capturing...probably ever. I typically get the footage delivered on an external drive or card, and just copy it over. The video editing tasks are not as common as 3D work and motion graphics in After Effects. Regarding SSDs, I don't think I can swing the price at the moment. I'll take another look though.

I am dealing with 10gb inconsistency betweens reads and writes with the Xeon D onboard nics. I would highly recommend, if rock solid 10gb networking speeds are important, to go with a Chelsio T4 or T5 add in card. They have both SFP+ and RJ45 variants - driver compatibility with FreeNas/FreeBSD is awesome and can totally support line rate 10gb if the underlying/connecting hardware can push it. With the onboard Xeon D nics, I am getting great write speeds (near line rate) but slow read speeds (1/4 speed) on striped pool of SSDs (for testing). The network performance is a non-issue with Chelsio nics.

Thanks for this. So, that makes me wonder if I should just consider a different board, especially if I get the Chelsio nics anyway?