Misaligned Pools and Lost Space...

[snicke]

So @jgreco, in a mixed environment with a lot of big files like photos, videos and movies but also smaller files like documents and source code etc. and a place to put VMs and jails on, which recordsize would you choose? Because we all have to make a choice. ;)

Sent from my Galaxy S6 Edge

 

[Dice]

which recordsize would you choose?

My bet is that @jgreco will advice "go default" :P

 

[anodos]

My bet is that @jgreco will advice "go default" :p

Or criticize whatever choice you make. :D

 

[jgreco]

Or criticize whatever choice you make. :D

Wow, what brought that on?

I've been wondering who's been pilfering my nasty pills.

 

[anodos]

Wow, what brought that on?

I've been wondering who's been pilfering my nasty pills.

Been playing with dtrace and samba this morning trying to make something work. Forgot to make coffee till a little bit ago. In other words, I'm wearing my grumpy pants. :D

 

[jgreco]

Been playing with dtrace and samba this morning trying to make something work. Forgot to make coffee till a little bit ago. In other words, I'm wearing my grumpy pants. :D

I'd be happy to lend you my five pound sledge, makes most problems go away (possibly introducing new ones).... ;-)

 

[jgreco]

So @jgreco, in a mixed environment with a lot of big files like photos, videos and movies but also smaller files like documents and source code etc. and a place to put VMs and jails on, which recordsize would you choose?

Big files will naturally benefit from a large recordsize. Data such as VM's will suffer on a pool that is designed for large file use, because they're just so different. If it's only a VM or two and poor performance is acceptable, then you're probably just going to go do the RAIDZ2/RAIDZ3 thing with the default recordsize, and then make a zvol with a 16K or maybe 32K block size. You can't "fix" this, all you can do is mitigate.

Because we all have to make a choice. ;)

Well, if you have to choose, choose to put your VM's on a mirror of SSD's and then this becomes a nonissue. And at this point SSD is almost cheap enough that I don't even feel bad saying it.

 

[snicke]

Well, if you have to choose, choose to put your VM's on a mirror of SSD's and then this becomes a nonissue. And at this point SSD is almost cheap enough that I don't even feel bad saying it.

And then go for recordsize 1M on your main RAIDZ2/RAIDZ3 pool for mixed files of which large photos, movies and videos stands for the absolute largest part of the pool?

Sent from my Galaxy S6 Edge

 

[Bidule0hm]

Whether any of these downsides are applicable to any given scenario is, of course, a different matter entirely.

That's the important line; I made this thread for home systems, not enterprise systems with VMs, etc.. Plus we are talking about RAID-Z pools only, which aren't very recommend for VMs or DBs...

But I'll make things a bit clearer (it's actually the first version, I didn't re-read, corrected and maked things better) ;)

And then go for recordsize 1M on your main RAIDZ2/RAIDZ3 pool for mixed files of which large photos, movies and videos stands for the absolute largest part of the pool?

Yep.

 

[jgreco]

And then go for recordsize 1M on your main RAIDZ2/RAIDZ3 pool for mixed files of which large photos, movies and videos stands for the absolute largest part of the pool?

Absolutely. RAIDZ3 with a fairly large vdev (~11 drives) and large files is a big fat WIN.

 

[jgreco]

That's the important line; I made this thread for home systems, not enterprise systems with VMs, etc.. Plus we are talking about RAID-Z pools only, which aren't very recommend for VMs or DBs..

Right, but this is the software/configuration equivalent of the "server grade hardware" problem. People come in here hoping to make do, because they are hoping that their 2009 era 8GB box with five drives in RAIDZ2 can store their files AND host a few VM's. I fully appreciate their despair at discovering that their hopes are unrealistic. Hardware is now cheap enough (and has been for awhile) that running a few VM's is not impractical or impossible.

 

[snicke]

Big files will naturally benefit from a large recordsize. Data such as VM's will suffer on a pool that is designed for large file use, because they're just so different. If it's only a VM or two and poor performance is acceptable, then you're probably just going to go do the RAIDZ2/RAIDZ3 thing with the default recordsize, and then make a zvol with a 16K or maybe 32K block size. You can't "fix" this, all you can do is mitigate.

Well, if you have to choose, choose to put your VM's on a mirror of SSD's and then this becomes a nonissue. And at this point SSD is almost cheap enough that I don't even feel bad saying it.

Would you say that jails and VMs are comparable regarding file sizes? I.e. you would go with SSD with default record size (128K) for jails also and preferably two mirrored SSDs? Keeping ashift value = 12 (and never change ashift value regardless of the situation (lots of VMs vs lots of big files) as long as you have 4K sector drives?)?

 

[jgreco]

No, the reason that jails are better for this sort of thing is explained here

https://forums.freenas.org/index.ph...res-more-resources-for-the-same-result.28178/

While that specifically describes "iSCSI", it is generally relevant to ANY block access thing (including local VirtualBox VM disk files) and should help you understand why jails with actual files in the ZFS filesystem namespace tend to work better.

You can absolutely put your jails on SSD and that'll make things go faster, and putting your VM's on SSD will make them go lots faster.

 

[Bidule0hm]

Right, but this is the software/configuration equivalent of the "server grade hardware" problem. People come in here hoping to make do, because they are hoping that their 2009 era 8GB box with five drives in RAIDZ2 can store their files AND host a few VM's. I fully appreciate their despair at discovering that their hopes are unrealistic. Hardware is now cheap enough (and has been for awhile) that running a few VM's is not impractical or impossible.

Yeah but I can't do anything about that... but if they didn't read the most basic pieces of info they likely won't read my thread either, so in the end... :D

 

[jgreco]

Yeah but I can't do anything about that... but if they didn't read the most basic pieces of info they likely won't read my thread either, so in the end... :D

I'm approaching five years of having fought that battle here on the FreeNAS forums.

 

[nightshade00013]

I'm approaching five years of having fought that battle here on the FreeNAS forums.

Even those of us who are still learning but trying to help take some of the pressure off you guys are fighting the battle of people who insist on doing it their way. I remember the other day where the Linus was strong with one and I likely made him very upset.

 

[Mirfster]

I remember the other day where the Linus was strong with one and I likely made him very upset.

So maybe it was you who has been pilfering jgreco's nasty pills... :D

 

[nightshade00013]

So maybe it was you who has been pilfering jgreco's nasty pills... :D

That day, Yep I can claim it. I have a good reason too.

Show : if you have a strong stomach

Stitches are still in and I still feel crappy at times. I am trying my best to be nice and the pain is technically less than I have dealt with for a long time but I get stiff and sore fast.

 

[jgreco]

Even those of us who are still learning but trying to help take some of the pressure off you guys are fighting the battle of people who insist on doing it their way. I remember the other day where the Linus was strong with one and I likely made him very upset.

Yeah, they just don't like hearing bad news. Honestly, I've been building FreeBSD servers since before-FreeBSD (back when it was 386BSD) and one of the simultaneously best and worst aspects of it is that it works on PC grade hardware, which ranges from awesome to total crap. I have a certain amount of sympathy for the guy who just wants to recycle his old desktop with a Pentium 4 and 1GB of RAM. I have less sympathy for the guy who turns up having purchased all his parts a week ago and then trumpets "Here's my build" and it turns out to be ... bad stuff, obviously never having read anything we've carefully documented here.

Basically when it comes down to it, most folks on the forum are just looking for each other to have the best possible outcome, and to learn some good stuff while making a safe haven for their data. Unfortunately that does not end up being "it works great on whatever random crap you decide to make your NAS out of." Sigh.

 

[RichTJ99]

Hi,

I am thinking about increasing my Vdev from 6 drives to 8 drives - which would be ideal budgetwise. I know i need to backup the data before killing the Vdev & making a new one.

I still dont exactly get why 6 or 10 drives is better than 8 but what is the downside for bulk storage (no VM's).

Thanks,
Rich

Here is a link as to why the rule 2^n + p no longer applies from one of the zfs devs.

http://blog.delphix.com/matt/2014/06/06/zfs-stripe-width/

Nice write up, thanks for the post.