1PB storage server using FreeNAS

[uberwebguru]

I am working on building a 4U 1 PB storage server using this Supermicro server https://www.supermicro.com/products/system/4U/6048/SSG-6048R-E1CR90L.cfm
Here are the questions i have

#1. What kind of performance can i get from this write/read performance?
Will connect to multiple 10GE ports on the server to 10GE switch(es)
Clients will be using thunderbold adapters on MAC with nvme storage

#2. How many pools is advisable to maintain reliability (and maybe performance) on this monster?

#3. What is recommended RAID level? Z3?

#4. If hard drives fails, how long will i be looking at to rebuild? (now back to #2 will multiple pools help reduce rebuild time?)

Please help and THANKS!!!! in advance

 

[mattbbpl]

Wow.... What kind of drives are you using? Wouldn't 90 bays still require over 11 TB drives to reach a PB?

 

[Spearfoot]

Yeah... wow!

#1. What kind of performance can i get from this write/read performance? Will connect to multiple 10GE ports on the server to 10GE switch(es) Clients will be using thunderbold adapters on MAC with nvme storage

No idea what you should expect in the way of performance, and I envy you the opportunity you have to find out!

#2. How many pools is advisable to maintain reliability (and maybe performance) on this monster?

Hard to make pool recommendations 'cause we don't know how you plan to use this beast. If you plan on providing block storage for hypervisors, you will probably want to have a least two pools: one consisting of striped mirrored vdevs for block storage, and another pool for general-purpose storage, probably comprised of RAIDZ2 or RAIDZ3 vdevs.

#3. What is recommended RAID level? Z3?

Again, it depends on your use-case. This article ("ZFS RAIDZ stripe width, or: How I Learned to Stop Worrying and Love RAIDZ") will give you some ideas to mull over regarding RAIDZ.

#4. If hard drives fails, how long will i be looking at to rebuild? (now back to #2 will multiple pools help reduce rebuild time?)

This will depend on the individual vdevs: mirrors don't take nearly as long as any flavor of RAIDZ, and RAIDZ resilvers will depend on the size and number of drives in the vdev.

You'll need to install every bit of RAM the system supports - 256GB.

What kind of drives DO you intend to use?

Come back after you've built it and give us a report! Inquiring minds want to know!

 

[mattbbpl]

Come back after you've built it and give us a report! Inquiring minds want to know!

Definitely. I apologize that I can't offer more input OP, but this is WELL beyond the realm of my experience.

In the meantime, don't mind me. I'll just be lurking to see what results.

 

[uberwebguru]

Wow.... What kind of drives are you using? Wouldn't 90 bays still require over 11 TB drives to reach a PB?

I will be using the 12TB HGST hard drives https://www.hgst.com/products/hard-drives/ultrastar-he12
so will wait till its released

 

[uberwebguru]

You'll need to install every bit of RAM the system supports - 256GB.

Why do i need to use up to 256GB memory? for speed and performance and more caching?

What kind of drives DO you intend to use?

https://www.hgst.com/products/hard-drives/ultrastar-he12

Come back after you've built it and give us a report! Inquiring minds want to know!

[/QUOTE]
Yup

 

[snaptec]

Question 2-3 depends on your workload. Where 1) depends on 2-3.
so please give us more Input

What else will you connect?
Even with 11 raidz2 vdevs containing 8x8tb hdds (total of 88 disks) you will have about 440 tb storage, maybe even less.

4)
Rebuild is per vdev. Doesn't matter if its a single vdev pool or a 20 vdev pool.

You will have a backup from that machine, right?


Gesendet von iPhone mit Tapatalk

 

[uberwebguru]

Question 2-3 depends on your workload. Where 1) depends on 2-3.
so please give us more Input

What else will you connect?
Even with 11 raidz2 vdevs containing 8x8tb hdds (total of 88 disks) you will have about 440 tb storage, maybe even less.

4)
Rebuild is per vdev. Doesn't matter if its a single vdev pool or a 20 vdev pool.

You will have a backup from that machine, right?


Gesendet von iPhone mit Tapatalk

If rebuild is for each vdev...then from requirement seems will be 1 giant RAIZ Z3 vdev then..yes there will be backup

 

[Spearfoot]

Why do i need to use up to 256GB memory? for speed and performance and more caching?

Yes, to all of those. The rough rule-of-thumb for specifying RAM is 1GB of RAM per 1TB of storage. That's just a guide, not a formula... but it implies that you'll need to fully populate the RAM on your system.

 

[snaptec]

One 90 wide vdev?
If you are really planning that, please let someone else, f.e. Ixsystems Plan that System for you.

Again, everything depends on your workload.
It can be possible that 32gb are enough or maybe 1Tb ram.
Please be more specific


Gesendet von iPhone mit Tapatalk

 

[Spearfoot]

If rebuild is for each vdev...then from requirement seems will be 1 giant RAIZ Z3 vdev then..yes there will be backup

A single 90-disk RAIDZ3 pool? Oh, no! That's just crazy talk! :p

The more disks in a vdev - and the larger they are - the longer a resilver will take. So you definitely want to split your 90 drives up into multiple vdevs.

Again, as @snaptec pointed out -- we need more information about how you plan to use the system before we can advise you on pool design.

 

[uberwebguru]

A single 90-disk RAIDZ3 pool? Oh, no! That's just crazy talk! :p

The more disks in a vdev - and the larger they are - the longer a resilver will take. So you definitely want to split your 90 drives up into multiple vdevs.

Again, as @snaptec pointed out -- we need more information about how you plan to use the system before we can advise you on pool design.

but he said all vdevs will all be rebuilt anyways, so how will splitting help me rebuild faster?

 

[gpsguy]

No, @snaptec said resilvers were based on vdev size. Smaller vdev's will resolver faster than large ones. For RAIDz3, 11 drives is about the widest we'd recommend.

A 90 drive vdev would probably take months or years to resilver.

 

[uberwebguru]

No, @snaptec said resilvers were based on vdev size. Smaller vdev's will resolver faster than large ones. For RAIDz3, 11 drives is about the widest we'd recommend.

A 90 drive vdev would probably take months or years to resilver.

I see

 

[Spearfoot]

but he said all vdevs will all be rebuilt anyways, so how will splitting help me rebuild faster?

Resilvering takes place at the vdev level.

You don't want to use RAIDZn vdevs that are too 'wide', one of the reasons being that resilvering would take so long. Another is that the IOPS available from a pool scale with the number of vdevs making up the pool. With a 90-drive RAIDZ3 pool, your IOPS would be that of... a single disk. Yikes! Performance would be awful!

Realistically, you will need multiple vdevs, probably something on the order of ten 9-disk RAIDZ3 vdevs. Or you might want one pool made up of 10 mirrored pair vdevs (for block storage), with the remaining 70 disks in a pool made up of seven 10-disk RAIDZ3 vdevs. Or...

Do you see why we keep asking how you plan to use this beast? :)

I suggest you study ZFS and FreeNAS before you take any big steps; it will pay off big in the long run. Here are a few articles to get you started:

http://doc.freenas.org/9.10/
https://forums.freenas.org/index.php?threads/zfs-primer.38927/
https://forums.freenas.org/index.ph...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/
https://forums.freenas.org/index.php?threads/comprehensive-diagram-of-the-zfs-structure.38865/
https://forums.freenas.org/index.php?threads/hardware-recommendations-read-this-first.23069/
https://forums.freenas.org/index.php?threads/building-burn-in-and-testing-your-freenas-system.17750/
https://forums.freenas.org/index.php?threads/how-to-hard-drive-burn-in-testing.21451/
https://forums.freenas.org/index.php?threads/some-insights-into-slog-zil-with-zfs-on-freenas.13633/
http://nex7.blogspot.com/2013/04/zfs-intent-log.html
http://blog.delphix.com/uday/2013/02/19/zfs-write-performance/
http://blog.delphix.com/matt/2014/06/06/zfs-stripe-width/

 

[uberwebguru]

I suggest you study ZFS and FreeNAS before you take any big steps; it will pay off big in the long run. Here are a few articles to get you started:

http://doc.freenas.org/9.10/
https://forums.freenas.org/index.php?threads/zfs-primer.38927/
https://forums.freenas.org/index.ph...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/
https://forums.freenas.org/index.php?threads/comprehensive-diagram-of-the-zfs-structure.38865/
https://forums.freenas.org/index.php?threads/hardware-recommendations-read-this-first.23069/
https://forums.freenas.org/index.php?threads/building-burn-in-and-testing-your-freenas-system.17750/
https://forums.freenas.org/index.php?threads/how-to-hard-drive-burn-in-testing.21451/
https://forums.freenas.org/index.php?threads/some-insights-into-slog-zil-with-zfs-on-freenas.13633/
http://nex7.blogspot.com/2013/04/zfs-intent-log.html
http://blog.delphix.com/uday/2013/02/19/zfs-write-performance/
http://blog.delphix.com/matt/2014/06/06/zfs-stripe-width/

Yes will check those out...thanks a lot

 

[uberwebguru]

One 90 wide vdev?
If you are really planning that, please let someone else, f.e. Ixsystems Plan that System for you.

Again, everything depends on your workload.
It can be possible that 32gb are enough or maybe 1Tb ram.
Please be more specific


Gesendet von iPhone mit Tapatalk

Can take up to 1TB memory actually ==> https://www.supermicro.com/products/system/4U/6048/SSG-6048R-E1CR90L.cfm

 

[snaptec]

I know, thats why I wrote from 32 to 1TB ram.

 

[Chris Moore]

Did anything ever happen with this?

 

[skyyxy]

whats the raid or hba card u are using for this monster?