BUILD Proposed 10x6TB Home Server Build

[n3IVI0]

So in my case, it would be 8+2, total useable space of 34.37, leaving the recommended 8.592 free, correct? How long would such a vdev take to resilver if a drive goes down? I only ask because I really don't want to go to two 6+2, because of the added expense, but also because I am not finding any good cases that can accommodate 12 drives.

 

[Bidule0hm]

Yes.

It depends on your rebuild speed and how full is your pool. For example at 100 MB/s and with the pool totally full (excepted 20% of course) it would take 100 hours. In reality your rebuild speed should be higher than that and your pool will not be full so it'll take less time.

 

[n3IVI0]

Is that average?

 

[Bidule0hm]

What is average? the rebuild time? if yes, no, it's the worst case scenario.

 

[Pheran]

Glad to be of service! I swear I'll be posting part 3 of my build soon; it's been a crazy few weeks. I'm still ironing out some details on my NAS but it is up and running and it's fantastic.

One comment on your build - that's an awful lot of money for a 660W platinum PSU when its efficiency may be compromised by your system being too low on its power curve. But I sympathize with your plight - it's difficult to find low-wattage PSUs with a lot of SATA power connectors.

 

[n3IVI0]

They do not exist. That was the lowest wattage Seasonic I could find. If I was willing to drop it back to Gold rating, I could get this 650W Seasonic. I'm trying to choose energy efficient components so the average cost in power is lower.

 

[ChriZ]

You can always use Molex to sata cables; no need to buy a bigger psu just because it has enough sata power connectors.

 

[NAStard]

Apparently the diff between Platinum and Plus Gold is 2% efficiency. (80 Plus)
The issue of power got me curious as I had never looked into it before. And it made me wonder if I should get a lower power rated PSU with higher efficiency because 8,760 hours per year will accumulate some cost based on the rate of lost energy. One day I will actually measure my system's power consumption with different PSUs as the system seeks a permanent home in a future enclosure. But until then, I found this article/post from Tom's Hardware, Power Supply Efficiency Myths, which seems to satisfy my curiosity... for now. ;)

 

[Robert Trevellyan]

Perhaps make more than one raidz vdev and mirror them?

You don't mirror vdevs with ZFS (unless you're thinking of some kind of OS-level replication task). A mirror is one type of vdev, RAIDZ1 is another type of vdev, etc. You stripe vdevs to make a larger pool and/or increase performance.

 

[n3IVI0]

Problem solved: SeaSonic SSR-450RM 450W 80 PLUS GOLD Certified Modular Active PFC Power Supply This only has six SATA power ports, so you buy a Seasonic compatible 6-pin to 4 SATA modular cable. Now I can have my efficient SeaSonic, not too powerful so as to keep from reducing efficiency at idle, AND I still get to have 10 SATA power ports without having to resort to Molex adapters. It's even the exact same cable, so it will match the others. <chomps down on cigar and grins> I love it when a plan comes together!

You stripe vdevs to make a larger pool and/or increase performance.

So, I would put my 10 drives into a single vdev, RAIDZ2, and then stripe to increase the size of the pool and performance? How would this affect speed and the size of the pool? I am assuming I still have two drive parity, correct?

 

[Robert Trevellyan]

Striping refers to the combining of more than one vdev into a single pool. If you put 10 drives into a single vdev, there's no striping to be done. You should probably read this guide.

 

[n3IVI0]

According to the guide, striping yields no redundancy, so a failure of one drive would murder the zpool. I think, in order to get maximum storage out of my drives, but still have some parity, I am going to put the 10 drives into one vdev, RAIDZ2. It will be backed up on Crashplan in case of catastrophic failure. I have never had a WD Red fail, not one, let alone two, so I think I'll be okay. Does that sound logical?

 

[Bidule0hm]

Stripping of one drive vdevs, yes, but stripping of RAID-Z2 vdevs isn't a problem. But as you chose to do one vdev it doesn't matter anyway ;)

 

[n3IVI0]

Alright, I've been reading and pondering, and pondering and reading, and I think I've got the solution. I am getting the feeling that a 10 disk vdev might be too wide, but I was hindered, I thought, by a lack of cases that could hold twelve drives. I thought wrong. Check out this bad boy: Fractal Design Define XL R2 Computer Case This thing is HUGE. It looks like the monolith from 2001. It has eight 3.5" slots, and 4 5.25" slots, so with the addition of a 4x3 SATA backplane, you get 12 drives. It's got the same silent features as the slightly smaller R5 case I was originally considering, but it's a bit more spacious inside. I can put an extra 140mm Fractal Design Silent R2 fan in the front to cool the bottom drive cage, there's another fan (included) on the bottom of the case, and one in the back (also included), for a total of four fans, but without compromising the silence. I ran my components through this PSU calculator, and it appears my 450W Seasonic PSU will still be plenty, even with a total of 12 WD Reds (if anybody can see any gotchas, let me know). I'm going to put the case on one of these CPU caddies to add some additional clearance underneath, and also to make it easier to move (it's rather heavy, esp. with 12 drives). Everything else can stay the same, only some cable extensions are needed to reach from the PSU to the motherboard (12" 24-Pin Power Extension Cable, 9 1/2" 8-pin P8 Cable, 24" Latching Round SATA Cable).

This will allow me to have a zpool of 12 total drives composed of 2 6+2 RAIDZ2 vdevs. It will reduce the width of my vdevs, increase the IOPS, and give me 4 parity drives (two per vdev). I still get 42 TB of storage, less 20% for a total of 34 TB. I think that ought to do the trick.

So, Case Build Rev. 3:

• Case: Fractal Design Define XL R2 (one additional Fractal Design Silent Series R2 140mm Fan, and the above listed extension cables and CPU caddie)
• Motherboard: Supermicro X10SL7-F
• CPU: Intel Xeon E3-1271 v3
• PSU: SeaSonic SSR-450RM 450W (one extra 6-pin to 4 SATA modular cable)
• RAM: Crucial 16GB (2 x 8GB) 240-Pin DDR3 SDRAM ECC Unbuffered DDR3 1600 (PC3 12800) Server Memory Model CT2KIT102472BD160B X 2 = 32GB
• HDD: 12 - WD Red WD60EFRX 6TB (include 4-in-3 SATA backplane, which gets its power from 2 Molex)
• Boot Media: SUPERMICRO SSD-DM016-PHI SATA DOM - 16GB

Is 32 GB of RAM sufficient for 12 drives? Is a 450W PSU adequate for this rig (everything is low power, so I THINK it's okay - just wanted to double check)? If not, the same G Series comes in 550 and 650 as well.

Everything look copacetic?

Thanks for the great advice!

 

[Robert Trevellyan]

Is 32 GB of RAM sufficient for 12 drives?

It may very well be enough. However, you'll be a bit stuck if it isn't, since the mobo doesn't support more.

I don't have much to say about the rest, other than I prefer the idea of two 6-drive RAIDZ2 vdevs over one 10-drive RAIDZ2 vdev, for the reasons you state.

 

[Bidule0hm]

I'm going to put the case on one of these CPU caddies to add some additional clearance underneath, and also to make it easier to move (it's rather heavy, esp. with 12 drives).

You shouldn't move a PC/server while running, it's very bad for the drives.

Everything else looks ok.

If you can avoid extension cables then do so. If you can't then check that they are good quality before using them ;)

 

[n3IVI0]

My main goal in putting it up on casters was to give the extra ground clearance to increase ventilation to the bottom fan. You are correct, though. I will only need to move it in order to clean the filters periodically, whereupon I would need to shut down the server anyway. The front wheels lock, so it will keep it stationary. The users of the case indicate that the PSU cables are too short. I would like to route the cables behind the motherboard, a feature of these cases, and will need the extra length. I will definitely research the cables well to make sure they are of the highest build quality. I appreciate the advice. Also your nifty RAIDZ calculator. :)

 

[Bidule0hm]

If you can mount the PSU with the fan on the top I recommend do to that. I never understood how mounting the PSU with the fan right on the floor is a good idea... Yeah, I know, in theory the PSU can suck cold air from the room instead of warm air of inside the case but all the others downsides (starting with dust...) far outweigh this I think (the air inside the case is usually only a few °C above ambiant so, yeah, not that useful...). It's a bit like the rubber mount for the hard drives, one advantage but 10 downsides on the other side...

Adding cable extensions just so you can route them behind the MB is especially what I had in mind when I wrote my previous post. It's less pretty to have the cables in the front of the MB right in the middle of the case but it's more reliable (the more connectors and cable length you have the less reliable the system is and the more losses you have) and even if I like a lot good cabling management I'll always prefer reliability over that for a server. But of course it's your server so you'll do as you want, no problem ;)

You're welcome ;)

 

[n3IVI0]

Is 32GB of RAM sufficient? https://forums.freenas.org/index.ph...fficient-for-32tb-of-storage-using-zfs.23959/ Supposedly, it is recommended to have 1GB per TB of storage, plus 1GB for FreeNas. That would mean I would need, for my 72TB array, 73TB of RAM. It is unclear also, does that 1GB per TB apply to the total storage of the drives, or total useable space?

An added wrinkle is that neither Newegg nor Amazon carry the X9SRH-7F board. The closest I can come is this X10SRH-CF. It has similar specs to the X9, but it has 10 SATA III ports and 8 SAS ports and uses DDR4 RAM. Any thoughts on this board?

 

[Bidule0hm]

The rule is for the total raw storage space but it is a bit flexible, especially if you have a lot of storage. 32 GB for 72 TB is maybe ok but maybe not, it depends on what you do with the server.