Big project - Tips and references

[Mateus Bapista]

Hi Everyone,

I'm newbie in Frenas's world and I don't have expertise to buid a big project alone,
so I will list my initial configuration and I ask you help, please give me tips and references.

What can I do to make this project better and with more performance ?

Server
2x Xeon E5-2620 v3 2.40GHz
17x 32GB DDR4 PC4-17000 ECC Registered (544GB)
2x HBA 8 Ports external SAS3 12Gb/s
1x SATA DOM 16GB SATA 6Gb/s internal

JBOD
2x SAS Expander 8 Ports SAS3 12Gb/s
90x ST6000NM0034 Hard Drive 6TB SAS 12Gb/s 7200RPM 3.5in (540TB)

Networking
Netgear ProSafe M7100

This storage is to share big videos files in NFS on environment of 20 pcs simultaneously.
I need +/- 400TB RAW and I will think use LACP.

How many IOPS/bandwidth I can do with this configuration ?

Thanks

 

[hugovsky]

I think you fall in the category of "give iXsystems a call". Anyway, if you need 400TB, with 90 disks you will be dangerously close to the 80% full rule. But this is way over me. Let more experienced members have a look.

 

[zambanini]

having fun with the sas3 lsi controller, have we?

like hugovsky said: call ixsystems. that is far above for one with zero zfs knowledge.

 

[Mateus Bapista]

Thanks for reply

This is just a project, I want understand every part that I have to worry,
I think ixsystems will sell me something and just that.

So are with 90 disks I will be dangerously close to the 80% full rule, I think I can use stripe and ajust the RAW for 270 TB

You guys have any more change or concept that should follow?

Thanks

 

[gpsguy]

Paging @jkh for his input on a 90 disk system.


Sent from my phone

 

[MtK]

Let me start with a big WOW!
Now, @Mateus you wrote 17x32Gb... 17? Are you sure?
No dual/triple/quad channel...?

 

[Mateus Bapista]

Hey @MtK,

You're right, 18x32GB

Thanks for your help

 

[MtK]

Which board and case are you planning to use?

 

[Mateus Bapista]

mainboard and chassi ?

 

[MtK]

Si

 

[Mateus Bapista]

Mainboard Supermicro X10DRU-i+
Chassi Supermicro 6028U-TR4T+

 

[MtK]

Is this system for sharing the files (copy, move, edit) or playing/streaming them?

 

[Mateus Bapista]

Share files.

 

[Robert Trevellyan]

I'm newbie in Frenas's world ...
90x ST6000NM0034 Hard Drive 6TB SAS 12Gb/s 7200RPM 3.5in (540TB)

These two statements seem incompatible to me. I agree with the suggestions to call iX.

If you're determined to go the DIY route, start with something much smaller and learn from your mistakes with that. Also, seriously consider planning for two or more large systems rather than one huge system.

 

[alpaca]

These two statements seem incompatible to me. I agree with the suggestions to call iX.

If you're determined to go the DIY route, start with something much smaller and learn from your mistakes with that. Also, seriously consider planning for two or more large systems rather than one huge system.

I was (key phrase) in the boat of "I want to converge all my workloads to one big array".... Splitting workloads into separate, still large, systems really was a boon for me. I have total confidence that ZFS can be run at huge scale, but practicality has to factor in. Plus how will you go about backing up those 180 disks?

 

[Mateus Bapista]

Thanks for reply

The bakup is the second step :p

 

[cyberjock]

I'm with Robert. This is a terrible idea if you don't already have extensive ZFS knowledge. At the scale you are trying, this isn't something for someone to learn on. Going big (which yours is probably one of the largest) is not going to be easy for a newbie to "just pickup and learn". You really need to get lots of experience with something much much smaller. Perhaps 50TB or so, before trying to go all-in.

Large scale has very specific problems, and it absolutely requires you to have some understanding of things. You can't just "buy the hardware and it work".

This really sounds like something that iXsystems should be involved in.

I think ixsystems will sell me something and just that.

I'm not sure exactly what you mean when you say that, but iXsystems sells white-glove support contracts. They will provide as much or as little assistance as you need to use the server properly.

 

[kspare]

Don't use 12gb controllers, especially if you are just going to run a large stripe with no redundancy in your drives, your data will be lost before you even get it all moved onto your drives.

There is a thread on here about doing a burn in on your drives, i'd highly recommend it. it helped us identify 4 drives that had errors.

I know from experience with the 12gb controllers, so if you'd like to buy a couple let me know i'll sell you mine :)

We've been able hit 80% saturation on dual 10gb links (16-17 GB/s) for transfer speeds on 24 2TB drives running mirrored vdevs, so a 12gb controller isn't going to do anything for you if you are just using gig links.

Also the 12gb controller has been giving false failures on drives and causing them to drop out...huge risk.

Also ( and someone correct me here) but there isn't much need for all that ram with such large files?

make sure you have a UPS hooked up to the box so that you can have it shutdown gracefully in the event of a power outage.

If you are running all that ram anyway, you should be looking at an intel 750 to help protect all that memory in case of a power outage. I use one for l2cache but i'm fortunate enough to have a generator for power.

Thats all I can think of for now.

 

[Mateus Bapista]

Thanks very much for all replies

About 12gb controllers, is a complicated situation because I just know 1 jbod with 90 bays and this jbod is sas3, so I need this type of controller, but I'm thinking about it.
After being helped by the forum, and have read a lot of documentation, I did some design modifications:

Server - SYS-6028U-TR4T+
2x Xeon E5-2620 v3 2.40GHz
24x 32GB DDR4 PC4-17000 ECC Registered (768GB)
4x HBA 8 Ports external SAS3 12Gb/s - AOC-SAS3-9300-8E
1x SATA DOM 16GB SATA 6Gb/s internal
4x RJ45 10GBase-T LAN ports ( Maybe with LACP)

JBOD - CSE-946ED-R2KJBOD
2x SAS Expander 8 Ports SAS3 12Gb/s
90x ST6000NM0034 Hard Drive 6TB SAS 12Gb/s 7200RPM 3.5in (540TB RAW) - 258TB w/ Striped Mirrored
4x Intel Hard Drive SSD 800GB SATA 6Gb/s 2.5in (2 for L2ARC and 2 for zil)

Networking
Netgear ProSafe M7100

About the UPS, yes is included in this solution!

Thaks
Please comment on

 

[kspare]

You won't see any benefit running the 12gb controller. I have a 12gb backplane on my server you can get cables to go frmo 12gb to 6gb it's not a big deal. i'm telling you from experience you will have catastrophic failure if you use that 12gb card. In fact i'll put money on it. Nor will you see any faster performance to be honest. By all means get the backplane it helps to future proof your setup.