FreeNAS 9.1 - Performance

Status
Not open for further replies.

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Don't take this as me telling arguing with you. But a few things I noticed from your post:

1. You spent BIG money on a small pool. Unfortunately, most of us want high speed without the 5+ digit costs.
2. You striped your ZILs. If your ZIL fails and have an unclean shutdown of the system, ZFS is going to have a cow. You will be able to recover, but the extent of recovery is up for a heated debate. As such, this is not a recommended configuration.
3. You disabled ZFS prefetch. For most people, that causes a major performance penalty. You could have likely gotten by with less hardware if you had left that enabled.
4. You are at the upper limit for the amount of L2ARC you should have for RAM. Performance might improve if you decrease that ratio(that is, more RAM or less L2ARC, or both).
5. You also disabled the L2ARC prefetch. For most people wanting to maximize performance with an L2ARC, disabling the prefetch is usually a performance killer. Again, you could have probably gotten by with less hardware if you had left that enabled.

This kind of shows that if you have no budget restraints you can definitely get a high performing system and that newbies will likely never get ZFS tuning correct. The second one is no surprise at all, and not meant as a criticism.

Glad you did get it working for you though. ZFS is pretty awesome. :)
 
Joined
Jun 25, 2013
Messages
41
Hi cyberjock,

Thx for all the comments. Working with ZFS always makes my day :)

I kind of agree with all your points - but we have not done anything other than using our experience / meassurements and listening to people like yourselves.

Ad 1) This is a PoC setup - built to accomodate 3 - 4 VMs. Mostly in a LAB environment. We wanted to proove that we could, before suggesting to the CIO that we would be better off building our NAS ourselves in the future.
Ad 2) I totally agree - in a production setup we would mirror them. Although post pool ZFS v28, people seem to say that you could argue that ZFS will always be mountable - even with a SLOG failure. I have not followed those threads to verify though.
Ad 3&5) Disabling ZFS prefetch - and on the L2ARC - was hinted a few placed as to make the pool more responsive ( lower latency ) in a VMware environment, using the FreeNAS as a NFS-only datastore. I can neither confirm nor uncornfirm this, as we have no numbers to back this up. I will try and update my knowledge on this though.
Ad 4) We built to have a lot of L2ARC. Mainly because we wanted the entire dataset in either ARC or L2ARC. I know we are stretching it.

We have been running ZFS in a production environment since late 2009 - mainly on our Oracle ZFS 7000 series. All the ZFS tuning there is almost strictly hidden / forbidden - per the contract and support. Hence starting "from-scratch" with the default setting in FreeNAS was not as easy as we had forseen.

I know we are not masters of ZFS, but there has got to be people out there who deserves the predicate 'newbie' more than we do.

However, we are in it for the long haul, and eager to learn.

And a question if I may ... In a later production environment we would need approx. 100 TB to store all our DR / Backup needs - before they are shipped to tape or external disks.

We have our sights set on some SAS Nearline 4 TB disks... and since you should not use vdevs with a single failure-disk with anything larger than 2 TB devices, we consider using vdevs with 6 devices and RaidZ2 - and multiple of 6, until the pool-size is as needed.

Would that be a kosher thing to do?

Or would you go straight to a 3-way mirror?

Best regards.
Bjarke
 

pbucher

Contributor
Joined
Oct 15, 2012
Messages
180
Ad 2) I totally agree - in a production setup we would mirror them. Although post pool ZFS v28, people seem to say that you could argue that ZFS will always be mountable - even with a SLOG failure. I have not followed those threads to verify though.

It's great news to hear to you've gotten things smoothed out at last.

I don't know if I'd go as far as mirroring it, though with the money you are saving from getting out from Oracle's heavy foot it's probably affordable. More importantly in the end though you need to know how important your data is and what sort of backups you have of the data.

For myself the data is very very important, but I snapshot the ZFS pool every 60 min and replicate it to another box. So I leave my ZIL un-mirrored because I'm only risking a max of 60 min of data and I've yet to have an unexpected shutdown of FreeNAS(UPS /w monitoring turned on) so I'm feeling fairly safe.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Hi cyberjock,

Thx for all the comments. Working with ZFS always makes my day :)

I kind of agree with all your points - but we have not done anything other than using our experience / meassurements and listening to people like yourselves.

Ad 1) This is a PoC setup - built to accomodate 3 - 4 VMs. Mostly in a LAB environment. We wanted to proove that we could, before suggesting to the CIO that we would be better off building our NAS ourselves in the future.
Ad 2) I totally agree - in a production setup we would mirror them. Although post pool ZFS v28, people seem to say that you could argue that ZFS will always be mountable - even with a SLOG failure. I have not followed those threads to verify though.

For #2, what I've read is that you have to force the mount, and any files that were "in progress" may be corrupted. For virtualization, the VMDK might not be as consistent as the guest OS that uses it thinks or expects. One person started getting BSODs immediately and had to do a recover from backup. It kind of makes sense as the ZIL is nothing more than extension of the pool that is suddenly ripped out without warning.
Ad 3&5) Disabling ZFS prefetch - and on the L2ARC - was hinted a few placed as to make the pool more responsive ( lower latency ) in a VMware environment, using the FreeNAS as a NFS-only datastore. I can neither confirm nor uncornfirm this, as we have no numbers to back this up. I will try and update my knowledge on this though.

If you read up on them, the prefetch is deliberately conservative and the defaults were set where they were as they were found to provide 'free' performance. Since the majority of the latency is head seeking, reading some extra sectors before and/or after really isn't a penalty as the disk head was already there. If your L2ARC is big enough for your entire dataset and on SSDs, then disabling(or even enabling) prefetch doesn't really buy you any more or any less. It's a 1/2 dozen of one and 6 of the other.
Ad 4) We built to have a lot of L2ARC. Mainly because we wanted the entire dataset in either ARC or L2ARC. I know we are stretching it.

We have been running ZFS in a production environment since late 2009 - mainly on our Oracle ZFS 7000 series. All the ZFS tuning there is almost strictly hidden / forbidden - per the contract and support. Hence starting "from-scratch" with the default setting in FreeNAS was not as easy as we had forseen.

I know we are not masters of ZFS, but there has got to be people out there who deserves the predicate 'newbie' more than we do.

Honestly, I consider pretty much everyone to be ZFS newbies. It's pretty darn complex and if you haven't dealt with the source code you practically need a 4 year degree in ZFS to actually have an informed idea of what is going on. To add to the burden, with each zfs revision some behavior changes and others have settings that are increased, decreased, or ripped out completely. The good example is the vdev cache which was "disabled" as it was found to not provide much benefit with prefetch. Prefetch itself being something that didn't exist in v1. There's only 1 person on this forum that I think could have an informed idea of ZFS tuning(well, maybe 2 with your experience), and even he acknowledges his knowledge is fairly limited. Sometimes I wish ZFS had a course like MCSE that you could attend. I'd love to get that cert just to get it!

However, we are in it for the long haul, and eager to learn.

I commend your willingness to learn and devote yourselves to it. It will pay off. Many people show up and make that claim, but as soon as the rubber hits the road they change directions. They realize they are in over their head and feel that ZFS doesn't provide enough additional reliability to justify the cost in man-hours and/or hardware. ;)

And a question if I may ... In a later production environment we would need approx. 100 TB to store all our DR / Backup needs - before they are shipped to tape or external disks.

We have our sights set on some SAS Nearline 4 TB disks... and since you should not use vdevs with a single failure-disk with anything larger than 2 TB devices, we consider using vdevs with 6 devices and RaidZ2 - and multiple of 6, until the pool-size is as needed.

Would that be a kosher thing to do?

Or would you go straight to a 3-way mirror?

Best regards.
Bjarke

Honestly, as a backup server won't need extremely high speeds, you could probably get by with 8 disk RAIDZ2s. ZFS, if it is staying fairly busy with constant reads or writes(which is what backups and restores are) don't generally suffer the performance penalty with the rule of 2^n+p per vdev. Home use does of course as its not the same workload. But those users(like myself and many of the admins) are limited to their Gb LAN ports. I have an 18 drive RAIDZ3 and it works great... for me. I wouldn't necessarily recommend anything like what I've done but it does perform sufficiently to max out my Gb 5 times over.

I'm not a fan of mirrors except when performance is important. So I think RAIDZ2(or even 11 disk RAIDZ3s) would be good for backups.
 
Joined
Jun 25, 2013
Messages
41
Thx cyberjock and pbucher!

Was not claiming to be a ZFS oracle at all - we are humble, and I truly acknowledge your input - it is much appreciated!

I will try and play with the prefetch again - never tested it other than in the disabled mode.

On the other note with beeing a newbie, yes! Never before have I worked with something for such a long time, and still feeling I am only scratching the surface :) That is the part that makes it interesting I guess...

Thx again!

Bjarke
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Honestly, if it isn't broke, I wouldn't try to fix it. If your performance is acceptable I'd leave it alone.
 
Status
Not open for further replies.
Top