BUILD Thoughts on build for performance storage

Status
Not open for further replies.

joeschmuck

Old Man
Moderator
Joined
May 28, 2011
Messages
10,970
Capacity is not important at the moment, I am more concerned with speed. I plan on having a 45 sata disk pool and when I can afford it a SSD pool(striped) for extended performance.

As for the SSD pool you desire, I would never stripe drives in a NAS but if the data is fully recoverable (on some other device for when a SSD dies and all data is lost) then it's just a risk your company would have to understand and accept. I would at a minimum make it a RAIDZ1. SSDs are fast by nature and if you wanted to use those to create your high speed pool and leave the 45 hard drives for a lesser speed, that is fine too.

For the hard drives...
My first pool design would be for speed and redundancy:
1 pool containing 3 vdevs of 3 mirrors (9 drives) = ~9TB of storage, where each additional vdev is ~3TB of added storage. 45 drives would = ~42TB of storage. Keep in mind that this would all be very high speed access assuming you need that much storage and to be honest, I'm not even sure that this is a reasonable configuration.

Pool 2 is just for storage with good speed but not like a mirrored setup could net you:
1 pool containing 3 vdevs of 5 drives in a RAIDZ2 (45 drives in total) = ~27TB of storage. This configuration might be acceptable for the overall project but I'm skeptical.

Pool 3 is just storage with better than normal speed but gives you maximum value for your investment for storage:
1 pool containing 2 vdevs of 16 drives in a RAIDZ3 (32 drives in total) = ~80TB of storage. As I'm not a pool designer I'm thinking there may be an upper limit to how many drives can be in a vdev but I do know what that is. If you can up that vdev size to 22 drives for each vdev then that would give you ~120TB of storage. Again, that would not be lightning fast but it won't be terribly slow either.


Good luck with your testing, I think you have the tools to create a very fast NAS.
 

Mirfster

Doesn't know what he's talking about
Joined
Oct 2, 2015
Messages
3,215
As I'm not a pool designer I'm thinking there may be an upper limit to how many drives can be in a vdev but I do know what that is.
Based on cyberjock's FreeNas Guide it was stated:
It is not recommended that VDevs contain more than 11 disks under any circumstances.
Keep in mind that is the 9.2.1 guide (don't think there is a newer one). As well, I have heard of others running a little bigger (like 12 - 15 drives). I have just always adhered to the 11 drive limit myself personally.
 

joeschmuck

Old Man
Moderator
Joined
May 28, 2011
Messages
10,970
I think that was based on "best practices" by Oracle from a few years ago but not what the real limits are.

@Tino Zidore , just thinking outside the box here... Maybe you can build two FreeNAS systems, one for the hard drives and one for the SSDs. Compared to the cost of the drives along, the cost of another computer is nothing in the grand scheme of things, also it build in redundancy. You don't want to have all your eggs in one basket.
 

mav@

iXsystems
iXsystems
Joined
Sep 29, 2011
Messages
1,428
1 pool containing 2 vdevs of 16 drives in a RAIDZ3 (32 drives in total) = ~80TB of storage.
It should be noted, that so wide RAIDZ vdevs with modern 4K disk block sizes require data to be written in very large chunks to reach expected space efficiency. (16-3)*4KB=52KB, so, hoping for some compression, it can be efficient only for volblocksize/recordsize and file sizes of 128KB and above. It is fine for some video archive, but obviously not for bunch of small files or especially some VM storage. Excuse me if I missed planned pool usage in this thread.
 

Mirfster

Doesn't know what he's talking about
Joined
Oct 2, 2015
Messages
3,215
I think that was based on "best practices" by Oracle from a few years ago but not what the real limits are.
I think you are correct. I just stick to that rule personally until told otherwise. ;)

Excuse me if I missed planned pool usage in this thread.
For sure I would value your input; kinda like getting it "straight from the horse's mouth". :)
 

joeschmuck

Old Man
Moderator
Joined
May 28, 2011
Messages
10,970
It should be noted, that so wide RAIDZ vdevs with modern 4K disk block sizes require data to be written in very large chunks to reach expected space efficiency. (16-3)*4KB=52KB, so, hoping for some compression, it can be efficient only for volblocksize/recordsize and file sizes of 128KB and above. It is fine for some video archive, but obviously not for bunch of small files or especially some VM storage. Excuse me if I missed planned pool usage in this thread.
I feel any good advice is welcomed. I've already stated (my disclaimer) then I'm not a pool designer and of course I'm not talking about swimming pools.

I think you are correct. I just stick to that rule personally until told otherwise. ;)
I can't argue that at all.
 

c32767a

Patron
Joined
Dec 13, 2012
Messages
371
Based on cyberjock's FreeNas Guide it was stated:

Keep in mind that is the 9.2.1 guide (don't think there is a newer one). As well, I have heard of others running a little bigger (like 12 - 15 drives). I have just always adhered to the 11 drive limit myself personally.

The guide isn't wrong. It's just very conservative. If you follow it, you're pretty much guaranteed to have a reliable working system.

From reading the traffic on here over the years, it makes sense. Lots of people don't bother to do research and testing, or expect some old clunker to run like it's a modern machine..

Having said that, at $dayjob, we build systems with multiple 15 drive Z3 Vdevs. Our data is basically sequential ~1Gb files, We write once, read a lot and eventually delete.

A while ago I posted the config we've been using #9 ,#31. I haven't built any new boxes in a while, but we're about to start looking at life-cycling some of our stuff, so I guess I'll be out shopping again. I may try to hold off till skylake and FreeNAS become better friends. :)
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
The guide isn't wrong. It's just very conservative.

Yeah, Cyberjock is very much into the whole black and white thing. There are certainly hazards as you stray from the things that we've recommended over the years. My recommendations are also very conservative, but as far as configurations go there's a lot of room for designing something for a particular application that might not be a "good idea" elsewhere.
 
Status
Not open for further replies.
Top