Repurpose 240TB of hard drives into video editing NAS.

[riggieri]

Hi Community

I have been looking FreeNas for sometime and am starting to adventure down the path of possibly building a production environment NAS for my company. Here is my use case:

I would like 3-4 workstations to be able to saturate a 10GBe link. I understand most likely they won't all saturate at the same time, but if they do I am ok if read speeds drop to about 400MB/s We are a small video post production boutique that handles 4K-8K video. Currently we have a mix of DAS for the big workstations and a Synology NAS that can saturate 4 1GBe links. But this is about 3-4 years old and am really starting to feel how slow 112/MBs is. We already have a Netgear XS716T in place.

We have a huge selection of drive to work with after we archive most of our film storage to LTO8. We have available to us for repurposing:

16 4TB HGST NAS Drives
8 6TB HGST NAS Drives
16 8TB HGST NAS Drives
4 4TB Seagate drives
4 Sandisk 1TB SSD
2 Samsung 500GB SSD

I think the Optimal storage tiering would look like this:

1. Large Storage pool that hosts most of our media. Mix of Red R3D, ARRI MXF, Apple ProRes, and other formats. Some DPX frame sequences as well. Parity is not super important, as everything housed on this server would also be backed up, but downtime is expensive, and can piss clients off. I was thinking multiple RAIDZ2 pools stripped. Possible even creating one pool for Online Storage and one for Near-Line Storage.

2. Small Storage Pool that just for writing to for the most part. We will write outputs to this drive only. 16TB should do, and honestly could just be RAID0. If we lost this data, we can always re output and anything that is on this drive that needs to be archived, will be.

3. Extremely Fast Cache and Project Pool. Would like to take 4-6 SSDs and have them in a pool together. FCPx and Premiere Project files, plus render and cache file from Davinci Resolve. Small files, high throughout, medium IOPS.

Hardware wise right now I have an old Dell t7400 Workstation to play around with as a test bed. Intel X5620 with 16gb ECC RAM. If I decide that going down this, I will build a Supermicro based server on a Intel C612. I am also looking at RAIDMachine enclosures.

I would love to hear any thoughts on this. I have not figured out to best use to repurpose the drives we have, and thats what I am struggling with since they are all different sizes. I also am debating this over a turnkey system, but I think I can see more value and performance over a turnkey system

 

[Chris Moore]

What country are you in? Location affects what will be available to you. The information is not just there for fun.

I was thinking multiple RAIDZ2 pools stripped.

That is the wrong terminology, but I think I get where you are going with that.

You should probably look at the following resources to help you understand the technology in FreeNAS and ZFS better because it is different than what you are accustomed to with Synology and most other things on the market:

Slideshow explaining VDev, zpool, ZIL and L2ARC
https://forums.freenas.org/index.ph...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/

Terminology and Abbreviations Primer
https://forums.freenas.org/index.php?threads/terminology-and-abbreviations-primer.28174/

I will build a Supermicro based server on a Intel C612.

You will waste a lot of money on DDR4 memory to do that. You can easily use a Supermicro board such as this:
https://www.ebay.com/itm/Supermicro...ort-E5-2600-v2-24x-Slot-DDR3-I-O/112742607987
With lower cost Registered DDR3 memory and save quite a lot of money.
I suppose it depends on if you want to buy new or used. If you insist on new hardware, it will cost a lot more.

I am also looking at RAIDMachine enclosures.

Again, the question is if you are looking at new or used. Do you want to fit all that into a single enclosure or do you want externally attached disk shelves?
I have a system where I work that is using six disk shelves, that hold 16 drives each, that are connected back to the head unit by SAS cables and the head unit only has a pair of drives in it to boot the OS from.
Or, you could use something like this and put all the drives in a single chassis:
https://www.ebay.com/itm/4U-48-Bay-...ver-Chassis-UP-to-192TB-Capacity/153150709807

There are many options.

I have selected several systems for my organization and I would be happy to make some suggestions for you, but I would need a few answers from you about your desires and your budget.

 

[riggieri]

Hey Chris

I was hoping you would respond. I apologize for not using the correct terminology but I did read through the slideshow. So I get it now. Multiple RAIDZ2 vDevs in a single pool. I don't think I will need to use SLOG, as we have more of a sequential workflow rather than random. Unless otherwise recommended, I don't think we need to use cacheing.

You right about the DDR3 vs DDR4. I guess I am going to need about 128GB of RAM or more so that will help with the cost. Intel C602 will be fine.

In terms of the enclosure, I guess I could put them all in one chassis. And that could work great. But I like the idea of externally connected SAS enclosures. Easier to add more storage if needed and provides some form of fault tolerance if a backplane goes down I suppose. I would have multiple controllers, so more bandwidth split over more PCIe lanes.

But the cost of that enclosure if pretty tempting. But with only one controller, over PCIe2 8x, I am maxed at 4 gbytes/sec. (Not true after further research)


I would love to hear some more recommendations from you. I think my sticking point is how to effectively create and pool the vDevs with the drives I currently have.

 

[riggieri]

Oh wait, I see that the Chenbro chassis has two backplanes, and I can have multiple controllers. That is defiantly making me more interested.

 

[Chris Moore]

Oh wait, I see that the Chenbro chassis has two backplanes, and I can have multiple controllers. That is defiantly making me more interested.

I use the same model Chenbro chassis for my main NAS at home. I have a single controller with one port connected to each backplane. It works well for me. You can always add extra disk shelves even when you start with something big like this Chenbro unit. Supermicro makes a disk shelves that hold 45, 60 and 90 drives. I almost linked to one earlier. Easy to find though.

I think my sticking point is how to effectively create and pool the vDevs with the drives I currently have.

Using the drives you have, I would put them in a big pool using eight drives per vdev like this:
vdev-0 = 8 x 4TB HGST @ RAIDz2 - should give about 15.8TiB usable
vdev-1 = 8 x 4TB HGST @ RAIDz2 - should give about 15.8TiB usable
vdev-2 = 8 x 6TB HGST @ RAIDz2 - should give about 23.7TiB usable
vdev-3 = 8 x 8TB HGST @ RAIDz2 - should give about 31.7TiB usable
vdev-4 = 8 x 8TB HGST @ RAIDz2 - should give about 31.7TiB usable
That is about 118.7TiB of usable (deducting the 20% for free space) storage for your mass storage needs. I suggest RAIDz2 to reduce the risk of down time with the understanding that you have a backup elsewhere.

Small Storage Pool that just for writing to for the most part. We will write outputs to this drive only. 16TB should do

The 4 x 4TB Seagate drives could go into a separate pool for this but I would suggest getting more 4TB drives and making this RAIDz1 or z2 (for safety) so you don't end up with lost work and down time if a single drive fails. You would need 8 drives in RAIDz2 to get your 16TB usable here.

Extremely Fast Cache and Project Pool. Would like to take 4-6 SSDs and have them in a pool together. FCPx and Premiere Project files, plus render and cache file from Davinci Resolve. Small files, high throughout, medium IOPS.

All the previous things would fill the 48 bay Chenbro chassis in the hot-swap bays, but there is a space next to the system board where you can mount a 3.5 drive and I would get an adapter like this:
https://www.ebay.com/itm/Aluminum-4...ive-Disk-Mounting-Bracket-Holder/152726570740
Mount the four 1TB SSDs in that bracket and connect that bracket to the 3.5 drive tray next to the system board. You can use that as yet another pool, configured as a stripe set, for the fast data.

You might want to go with an adapter like this to connect the server to the network:
https://www.ebay.com/itm/Chelsio-T5...erver-Offload-Adapter-W-PCI-E-x8/173416784703
I believe that the Chelsio cards are well supported.

 

[Chris Moore]

PS. The pool I described above should be quite fast for sequential reads and writes.
The reason I suggested a SFP+ interface to the switch instead of a 10GbaseT (RJ45) connection is because the SFP+ interface has historically had less latency.
Did you have any questions about any of this?

 

[riggieri]

Hey Chris

Thanks for the links.

Questions:

I wasn't sure but I guess it is ok to have different size vDevs in a pool. Does this effect performance?

For the controllers, is it worth maybe going with two controllers, one per backplane? My thinking is that the 8TB drives read at close to 200Mbyte/s on the outer edges of the track, so 16 of those can easily saturate a single 6G SAS connection correct, meanwhile 24 drives will easily saturate a single SAS link.

Any reason for the Chelsio card over an Intel? Should I look for 4 Ports in LACP vs 2? My switch only has two SFP ports, but could I mix and match between SFP and 10GbaseT?

 

[riggieri]

Chris

Would this system work for me? Would it have enough CPU power? Would need to obviously put more ram in it, and it is a little PCIe limited, but cost wise it is 699 vs 899 shipped.

https://www.ebay.com/itm/Supermicro...097719?hash=item23a7faf737:g:hQcAAOSwheFasUjg.

 

[Chris Moore]

I wasn't sure but I guess it is ok to have different size vDevs in a pool. Does this effect performance?

The different size is not really the issue. The difference that matters is the individual drive IO speed. For example a slow drive can make the whole vdev slow and a slow vdev can slow down the entire pool. To know exactly, you would need to measure the performance of each drive, but you can get an estimate by looking up the sustained transfer rate from the published drive specs. As an example, you wouldn't want to make a vdev using old 500GB drives that are only able to sustain 80 or 90MB/s data rate and add that to a pool that has a lot of other faster drives in it because that vdev of old, slow drives will drag down the performance of the pool. The drives you have are all reasonably close to one another in performance, so should average out nicely. As time goes by, you will want to replace the older drives first so you can have bigger, faster drives and when the whole vdev is made of bigger, faster drives, it will bring up the performance of the entire pool.

For the controllers, is it worth maybe going with two controllers, one per backplane? My thinking is that the 8TB drives read at close to 200Mbyte/s on the outer edges of the track, so 16 of those can easily saturate a single 6G SAS connection correct, meanwhile 24 drives will easily saturate a single SAS link.

On a SAS controller, such as this:
https://www.ebay.com/itm/HP-H220-6G...0-IT-Mode-for-ZFS-FreeNAS-unRAID/162862201664
Which is what I use in my system, each of the connectors is four SAS ports (four lanes) which is able to transfer 24 Gb/s. That is 2400 MB/s, and if your hard drive is delivering 200 MB/s, theoretically, 12 drives might be all you could put on a 4 lane port through a SAS expander without bottlenecking the performance. The thing is, not all the drives are talking at once because of command cuing and other factors, so you don't usually have any problem with that. Then there is the data rate of 10Gb Ethernet which is in the vicinity of 1200 MB/s. So, having two backplanes in the Chelsio chassis, each connected to a port with 2400 MB/s of potential, in this situation, the drives (even if bottlenecking) will be able to deliver data around 4 times faster than the network can accommodate it. You loose some speed to overhead, so it is never as fast as the math might make you think it is, still it is an example.
Unless I messed up the math ...

Any reason for the Chelsio card over an Intel?

The main reason I chose Chelsio is because that is the brand of card that iXsystems uses for the FreeNAS Mini.
https://www.amazon.com/FreeNAS-Dual-Port-Upgrade-Ports-Twinax/dp/B011APKCHE
I figure that if it is what they are using, it must be well supported. I use a MELLANOX 10GB CONNECTX2 - MNPA19-XTR in my NAS and it works good enough for home and I have a FreeNAS system at work that is using the Intel 10GbaseT card like this:
https://www.amazon.com/Intel-X540T2...535139278&sr=8-1&keywords=intel+10gbaset+card
It works well enough that I wouldn't say, don't do it, but I don't think it is worth any extra money.
I would say that two ports in LACP would be a good place to start, because of the number of clients you have, and it would give you the potential to put close to 2000MB/s of data out to the users. Ballpark, without digging too deep into the details.

 

[riggieri]

Hey Chris, Is the Westmere/LGA1366 platform going to be an issue for me? Or should I Got to LGA2011 and Sandy Bridge/IvyBridge?

 

[Chris Moore]

Chris

Would this system work for me? Would it have enough CPU power? Would need to obviously put more ram in it, and it is a little PCIe limited, but cost wise it is 699 vs 899 shipped.

https://www.ebay.com/itm/Supermicro...097719?hash=item23a7faf737:g:hQcAAOSwheFasUjg.

The thing about an older system board is that if it doesn't do the job fast enough, you can always throw it out and get a newer, faster system board.
The limiting factor on that board, more than anything else, is the the fact that it has PCIe 2.0 slots instead of PCI 3.0 slots. An x8 slot on a PCIe 2.0 board is rated for 4GB/s where the x8 slot on a PCIe 3.0 board is rated at 8 GB/s, if I recall correctly. Both the SAS controller and the 10Gb (small b vs big B) are going to be better off in PCIe 3.0 slots, but it may not impact performance. You can give it a shot and plan to upgrade if it isn't fast enough.

 

[joeschmuck]

@Chris Moore
I believe you are giving great advice, keep it going.

 

[riggieri]

Hey Everyone. Again Thanks for your reply and knowledge sharing.

After further investigation, research and planning I think I have settled on a new plan, that I would like to run by you. I am going to build two separate FreeNAS servers for different reasons.

SERVER 1 "Vault"
Setup to be expandable. The storage is for cold/warm long term storage. For media files file that most likely will not be used much and will only be written to in very large sequential transfers from Server 2. This is the first one I am building and figured it could give me a testing platform.

Build:
Chenbro 48 Bay Chassis
Supermicro X8TI-f
2x L5640
128GB DDR3 RAM
1x LSI 9210-8i
Intel x550 with a single link (need my SFP+ ports for Server 2)
120GB SSD boot


Initial vdev setup as listed

6x1tb in RAIDz2
6x2TB in RAIDz2
6x2TB in RAIDz2

Now to my understanding I can expand the storage on this server two ways. First way is add additional vdev with 6 spindles and add them to the zpool. Or I can swap individual drives in a vdev slowly and resliver each time. Once I replace all 6 drives, additional storage will be available to the pool.


SERVER 2 "Production"
Setup to be very fast. This storage is for day to day working. High Sequential throughput and low latency.

Build:
SuperMicro 6047R-E1R36N
2x Xeon 2680 2.7GHZ 8 core
128GB RAM
2x LSO 9210-8i
Chelsio T520
120GB SSD boot (Mirrored)


Initial vdev Setup

Pool 1 (Editorial) Mix of Read/Write
6x4tb in RAIDz1
6x4tb in RAIDz1
6x4tb in RAIDz1
6x4tb in RAIDz1

Pool2 (Color/Cache) Majority Read
6x6tb in RAIDz1
6x6tb in RAIDz1

Pool3 (Output) Majority Write
6x1TB Sandisk SSD Stripped
(Not sure how I will mount these but I sure I can find space)

If I find a need for a higher IOPS but low overall storage pool, I might install a 4xNVMe setup in a 16x slot as well.

Now I know RAIDz1 come with its risks, but I have been running hardware RAIDs for years and know how to test and implement. I understand there is more risk, but, I was thinking smaller pools reduce the risk and help speed up the sequential throughput and increase IOPS.

Usually when I build and test RAIDs, I run each individual disk through a reliability test, fill/verify the volume completely with both sequential and random writes using a Mac based program called Disktester. Once all drives pass, I build the raid, and then again run the same test. After that we move the drives into production. What I am saying is I will do proper Testing and Burnin and except the RaidZ1 risk. Scrubs are usually schedule every weekend. Also any files on this server reside in at least1, maybe 2-3 additional places (LTO, VAULT, SINGLE DRIVE)

So I guess my questions are:

I am guessing there might be some nice features I can gain from two servers. How fast is the copy form server to server? What else can I do with two?

I might go with one 40GB ethernet card directly connected to a single workstation if I initially able to get good 1.5-2GBs speed internally and can achieve line speed on my 10GB network. Can I mix network cards manufactures?

Does my expandable storage system work?

How does the 6x RAIDz1 plan sound?


Thanks for your time.

 

[Stux]

I would consider combining the Editorial/Color/Cache pools. More spindles and more vdevs makes both pools faster, and potentially you may then benefit from an L2ARC too.

The output pool... well... if you think SSD is needed... but if its sequential write... then it'd probably be fine to the other pool too.

FWIW, I liked @Chris Moore original suggestions.

 

[riggieri]

Hey Stux

Thanks for getting back to me. The reason we thought of separate pools was so that we could distribute reads and writes a little bit better. I want to make sure the Color/Cache drives can hit 1200MB/sec plus and I fear if it is all one pool, then if other editors are reading and writing to it, the sequential throughput will drop.

 

[Stux]

I would suggest doing some performance testing before picking a pool layout.

 

[Chris Moore]

Normally, more drives makes more IOPS / more performance. If you want to break the data up logically, you don't need separate pools. The reason for separate pools is if the way the drives are configured is different. For example, a pool of mirrors is one pool. A pool of RAIDz2 vdevs is another pool.

 

[Stux]

Normally, more drives makes more IOPS / more performance. If you want to break the data up logically, you don't need separate pools. The reason for separate pools is if the way the drives are configured is different. For example, a pool of mirrors is one pool. A pool of RAIDz2 vdevs is another pool.

Exactly, although it might in theory be that the pools could perforrm faster by separeating reads/writes... it could also be that overall they'll perform faster by having more IOPS (due to more vdevs) and more bandwidth due to more HDs... for both cases.

But I don't know, which is why I suggest benchmarking.

The SSDs should definately not be mixed in the same pool though.

 

[riggieri]

I will definitely be testing and reporting back. Will report my findings. I am thinking that segmenting use case by Zpool will work well. But we will see.


How does my expansion plans sound for my Vault Server. Is that logical and will it work?

 

[riggieri]

Ok. So I built a test server and that went well, and went ahead and started assembling my first of two servers. There is where I am sitting with the Vault Server.

Chenbro 48 Bay Chassis
Supermicro X8TI-f
2x L5640
96GB DDR3 RAM
1x LSI 9210-8i
Chelsio T420-BT (One Link Active)
Switch is Netgear XS716T

My initial vdev setup is

8x8TB Seagate ST8000DM004 (Yes I know they are SMR, but for WORM use case it is fine.)

I plan on adding in 8 Drive vdevs as I validate previous drives being pulled from production. I run a Mac OS X version of bad blocks.

Anyways, seems to be going well. Not ready to put the server into production, waiting until our new small server closet is built.


Couple perfomance questions.

With iperf, I can fully saturate the 10GBe link from two different clients, so I know the network is performing.

After playing with Tunables, mainly sysctl values, I was able to get about ~700-750MB/s in both directions from two different 10G clients on SMB/CIFS. I have a feeling I am hitting the single threaded limit here of the L5640.

Since this server is really only going to be holding archive information , and this was far cheaper than storing in the cloud, I am not worried about the current speed I am getting of SMB. But in the future, when I build a faster production server, I guess I need to be looking at single core performance for single client maximum performance. Is this true?

If I start loading data onto this pool, and then add more drives, How do I make sure the data gets spread over the vdevs?

Anything else I should set up? I need to set up monthly smart tests, but how do I set up notifications if a drive starts failing?