12x6TB, 1240v5, Lian Li A76

[gregerg]

Original Post:
So after seeing this build I decided I'd like to use this case. I've really wanted a Lian Li q26 for a while, but I think I'm content with this case. Also, given that expanding a vdev is not possible, I kind of want this to last me... foreverish, and I'm going crazy with the number of drives.

There a few drive related things that I'm a bit undecided on that I'd love some feedback on (the "undecided" components of the build). I'm also putting off any SLOG / L2ARC SSDs for the future if I see a big need for them. If you think I could benefit from using some kind of SSD cache, given my use cases please comment. Thanks!

Here's some of my use cases:

• Plex (up to 2ish users, 1 local, 1+ internet)
• Nvidia Shadowplay gameplay recordings on separate desktop writing straight this NAS build
• Editing personal audio / video media projects via separate local desktop

Edit:
I've found a combo deal on newegg for x11SSH-F-O + 1240v5, so I went with that. I also Bought a Lian Li A76 case, as that fits my desires to have a 12 HDD drive case with decent airflow and low noise. Plus it was available to purchase, unlike many of the cases I was looking at.

I'm currently considering whether to go with a 650 or a 750 PSU, and then I'll buy that and an HBA. I'm also going to hold off on a 10gig network card for a while, and then when I get to the point where I'm editing video I may upgrade.

Here's the build (edited a few times):

• CASE: Lian Li A76
  
• CPU: E3-1240 v5
• MOBO: Supermicro X11SSH-F-O
• RAM: Crucial 2x16GB DDR4-2133 Unbuffered ECC
• HDDs: 12x6TB WD Red
• PSU: Seasonic FOCUS Plus Gold 750W (undecided)
• HBA: SAS9211-8I (undecided)

 

[Ericloewe]

Looks good, in general.

HBA: SAS9211-8I (undecided)

Also look at the 9207s, which might be easier to acquire these days.

If you think I could benefit from using some kind of SSD cache, given my use cases please comment.

Very doubtful.

 

[gregerg]

Also look at the 9207s, which might be easier to acquire these days.

It looks like both are in stock on amazon

 

[Chris Moore]

Editing personal audio / video media projects via separate local desktop

Transfer speed to storage would be helped with 10Gb networking, SSD cache is not called for. Spend the money if you want, but you would be better served with a 12 drive pool in 2 vdevs of 6 drives each because more vdevs gives you faster throughput to your storage. Many small files might benifit from a cache drive but large files, such as video, are sequential transfers, not random, and cache would likely never be used or only used a very small amount. The build you linked to is pretty, but it does not appear suited to the desire you have for video editing. You will need 10Gb interfaces in the editing system and the storage system to provide the fastest access possible for moving large files across the network.

So after seeing this build I decided I'd like to use this case.

The build you pointed out is using WHS (Windows Home Server) 2011 and was done in 2013. I don't understand why you are using this as an example for a FreeNAS build.

11x8TB WD Red (undecided vs 11x6TB)

Why did you choose 11, because that is all you can fit into that case?

 

[gregerg]

The build you pointed out is using WHS (Windows Home Server) 2011 and was done in 2013. I don't understand why you are using this as an example for a FreeNAS build.

I was browsing google for inspiration using the Define r4 case, and hadn't really considered using a hot swap cage to fill up more space in that case. I'm aware it's not directly applicable to a FreeNAS build. Chose 11 because that's what fits in that case, yes.

 

[Anonymouse]

https://m.newegg.com/products/N82E16813183013/specs

https://m.newegg.com/products/N82E16819117787

https://ark.intel.com/products/97469/Intel-Xeon-Processor-E3-1240-v6-8M-Cache-3_70-GHz

Just a thought and I'm a peasant here but why are you choosing the motherboard, the processor, and the RAM? I get that the RAM goes with the motherboard and the processor can just do DDR4, but why not more? The maximum RAM is 64GB for the E3-1240 v6. Doesn't 144GB sound much nicer. 11x6TB is a tad much with just the default.

I was toying around with an E5 that holds about a TB of DDR3 and it costs 144usd. The motherboard I want is 90usd. The server case is 200usd. The RAM is about 300usd for about 64GB.

 

[Pheran]

I'm unclear why you are choosing the old Define R4 case instead of a Define R5 (there's a brand new Define R6 but but sadly it removed some drive bays). Also, why not get rid of the HBA and use an X11SSL-CF motherboard instead? If you want to shave off a few dollars, the Xeon E3 1240 v6 offers very little benefit over the E3 1230 V6. You might also consider fewer larger drives, perhaps 8x10TB, instead of trying to cram all those drives into the 5.25" bays.

 

[Chris Moore]

I'm unclear why you are choosing the old Define R4 case instead of a Define R5 (there's a brand new Define R6 but but sadly it removed some drive bays). Also, why not get rid of the HBA and use an X11SSL-CF motherboard instead? If you want to shave off a few dollars, the Xeon E3 1240 v6 offers very little benefit over the E3 1230 V6. You might also consider fewer larger drives, perhaps 8x10TB, instead of trying to cram all those drives into the 5.25" bays.

More spindles gives faster access to the data. The OP wants to be able to edit video on the system. Few large drives will be slower than many smaller drives. To get fast access to data, I just created a pool of mirrors using 16 drives (8 vdevs) with each drive only being 500GB and the whole pool only 4TB. The purpose being speed, not massive storage.

 

[Ericloewe]

Doesn't 144GB sound much nicer.

It also sounds like a much more expensive platform.

 

[gregerg]

More spindles gives faster access to the data. The OP wants to be able to edit video on the system. Few large drives will be slower than many smaller drives. To get fast access to data, I just created a pool of mirrors using 16 drives (8 vdevs) with each drive only being 500GB and the whole pool only 4TB. The purpose being speed, not massive storage.

Is there a forum / blog post regarding this that I can read? I'm curious what read / write speeds I could get for a setup that you are suggesting. 12 drives over 2 vdevs.

 

[gregerg]

I'm unclear why you are choosing the old Define R4 case instead of a Define R5 (there's a brand new Define R6 but but sadly it removed some drive bays).

No reason. How many drives can fit in the R6? I'm not seeing that number on their website.

 

[MrToddsFriends]

Is there a forum / blog post regarding this that I can read? I'm curious what read / write speeds I could get for a setup that you are suggesting. 12 drives over 2 vdevs.

The comparison in this blog posting shows (among other configurations) double write speed for "12x 4TB, 2 striped 6x raidz2" compared to "12x 4TB, raidz2":
https://calomel.org/zfs_raid_speed_capacity.html

Of course your mileage will vary, depending on your workload (file sizes, ...).

 

[gregerg]

The comparison in this blog posting shows (among other configurations) double write speed for "12x 4TB, 2 striped 6x raidz2" compared to "12x 4TB, raidz2":
https://calomel.org/zfs_raid_speed_capacity.html

Of course your mileage will vary, depending on your workload (file sizes, ...).

Interesting. Thanks. Would I really benefit from a 10gig local network solution given those read / write speeds?

 

[Chris Moore]

Would I really benefit from a 10gig local network solution given those read / write speeds?

If you are on a 1Gb network, your maximum speed to access the data on the NAS is around 112MB/s regardless of how fast the NAS is able to address the drives (or how many cache drives you install) because that is as fast as the network can go. That is slower than most modern mechanical hard disk drives. If you want to take advantage of the speed of having an array of drives, you must step up to 10Gb. There really is no other way. With 2 vdevs at 6 drives each, you should have a minimum of 300MB/s of sequential read/write capacity (probably more depending on the disks) and you won't ever see it if you don't have a 10Gb network.

 

[Chris Moore]

Interesting. Thanks. Would I really benefit from a 10gig local network solution given those read / write speeds?

I am going to link you to this video:
https://www.youtube.com/watch?v=GGd-DiDnLpw

Just keep in mind that he has a couple problems with his configuration that prevents him from getting any better performance. First, he has too little CPU resources, only two cores, to handle all the computational tasks that are needed for what his NAS is doing. Also, his CPU has an extra burden because he does not have enough RAM, his NAS only had 8GB unless he changed that from the build video. In addition to that, he doesn't have enough disks (only 6) to be able to fully saturate his 10Gb network. There may be other problems, but I didn't look too close.

 

[gregerg]

If you are on a 1Gb network, your maximum speed to access the data on the NAS is around 112MB/s regardless of how fast the NAS is able to address the drives (or how many cache drives you install) because that is as fast as the network can go. That is slower than most modern mechanical hard disk drives. If you want to take advantage of the speed of having an array of drives, you must step up to 10Gb. There really is no other way. With 2 vdevs at 6 drives each, you should have a minimum of 300MB/s of sequential read/write capacity (probably more depending on the disks) and you won't ever see it if you don't have a 10Gb network.

Do you think the Define R5 / R6 are good cases for 12 drives? Are there any towers you would recommend?

 

[adrianwi]

I've got 9 in an R5 and whilst you could probably add another and keep things reasonably cool (~30) I think you'd struggle with any more.

 

[IQless]

Do you think the Define R5 / R6 are good cases for 12 drives?

I think you will have a hard time getting 12 drives into anyone of these, I have both.
R6 can accommodate for 8x3,5", but it is only delivered with 6 drive trays. FD does not atm sell extra drive trays, so you will have to buy them of someone who bought an R6 and don't need them.

Strictly speaking I guess you can place 12 drives inn... given that you get your hands on two extra drive trays, and use 8x3.5" and 4x2,5".

But if you try this, and get it to work, I would love to see the result! :D

 

[Chris Moore]

Once I get to my office, I will find something for you. I built one with 12 drives about 3 years ago and it was a challenge to find a case. When I built another as a backup, I bought a rackmount system because I was able to get it for only $130 plus shipping, and it included power supplies.
Would you consider a rackmount?

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[Chris Moore]

Here's the build:

• CASE: Fractal Define R4

You didn't answer if you would consider a rack-mount case, so I found this option for you.
This chassis will accommodate 9 x 5.25" drives:
https://www.newegg.com/Product/Product.aspx?Item=N82E16811517030
Using these adapters you can expand those bays into 15 x 3.5" drive bays:
https://www.newegg.com/Product/Product.aspx?Item=N82E16817994155

Although the exact products are different, this is a very similar build to to the third FreeNAS system I built for home use.
Total of these parts is $343.51 before any discounts but I think you can get a lower price on a rack-mount chassis that would hold just as many drives, if you are willing to go that way.