1.6 PB

[ChristianR]

Looking into building a backup storage for our production storage (EMC Isilon).
We will need 1.6 PB usable and be able to expand. Speed is less important, an alternative is using the cloud but expansive to restore a large amount of data.
Looking into using something like this with 10TB drives;

Dell PowerVault MD3060e
HPE D6020 Storage Enclosure

The storage enclosures controlled from one server running FreeNAS.
Spindown drivers could be nice when the data rarely will be accessed.

Could FreeNAS handle this?

 

[Johnny Fartpants]

I wouldn't use the Dell PowerVault MD3060e myself. I had one of these units for R&D and it never played nice with FreeNAS (not sure why) so I sent it back. I run loads of the these in production (about 10) and find them very good. I only run one Supermicro head per one 90 bay as I don't like the idea of hanging more than one per head for various reasons. With 12TB drives without compression you will get about 600TB of usable space.

 

[ChristianR]

Thanks for the info.
May I ask for some more spec about your setup?

 

[Johnny Fartpants]

Pretty much everything is in my sig but happy to answer any questions.

 

[ChristianR]

Why not multiple bays / head?
What models do you use for Supermicro head?
Why all that RAM?
Are your 10 bays joined in any way?

 

[Johnny Fartpants]

Why not multiple bays / head?

Its just a personal choice really. Nothing technical stopping you from doing it. I guess for me about 1/2 PB usable seemed the largest I was happy to go within one unit for the time being. Things like scrub times and re-silvers become a factor when building very large pools.

What models do you use for Supermicro head?

I commonly use something like this although the most important part is the HBA you use and I find the LSI 9300-8e works well.

Why all that RAM?

Ah you haven't been reading the forums have you. Some might even say you need more than 256GB but I find this works well for my use case (bulk filestore).

Are your 10 bays joined in any way?

Nope. Well probably not the way you might be thinking. I always buy two identical units and one backs up to the other (in different datacentres) using ZFS snapshot replication as these are primary filestore not backup like yours. Each unit has an individual namespace providing storage for several different groups of users.

 

[ChristianR]

Thanks,
Ah you haven't been reading the forums have you. Some might even say you need more than 256GB but I find this works well for my use case (bulk filestore). :)
Will do that now...

 

[Mlovelace]

Are your 10 bays joined in any way?

You should look at GlusterFS on top of ZFS, this would fit nicely into Gluster's wheelhouse.

 

[ChristianR]

You should look at GlusterFS on top of ZFS, this would fit nicely into Gluster's wheelhouse.

Have done that but my Linux skills are limited and I think with Gluster most of the config is by command line.

 

[ChristianR]

Have now got a quote of 2 x 946ED-R2KJBOD with 12TB HDD.
Johnny, You got 600TB with 12TB HDD / chassis. What kind of HDD protection do you have?

 

[Johnny Fartpants]

I normally do 15disk Z3 x 6 vdevs to make one pool and that replicates to another unit which is exactly the same but like I said the first unit is primary data. The biggest drives Im using atm are 10TB but I will go for 12TB for the next systems I purchase. Most of my pools are at about 50% capacity so approx. 250TB and currently scrubs take about 16 hours and resilvers about 10 hours. I monitor the drives SMART stats using scripts you can find on the forum. In practice what this means is that I replace about 95% of my drives before they actually fail so I don't get many surprises. You get used to the telltale signs particularly if you stick with the same brand/model. I would strongly recommend you do a burn-in when you first get your array and again lots of good info on the forum but in short I run badblocks followed by long smarts and that seems to weed out most of the dodgy ones. Finally make sure you have a good plan for disk identification as finding the dodgy drive in one let alone two 90 bays JBODs can be tricky.