Upgrade to 11.2 Causing Huge Problems

verisof

Cadet
Joined
Apr 29, 2018
Messages
4
I decided to do an upgrade to FREENAS 11.2 RELEASE U1 from 11.1. And I've had nothing but troubles since.

Let me preface this by saying I am not a LINUX or FREEBSD guru by any stretch of the imagination. I know just enough to be dangerous and build my system.

I'm running an older, but serviceable HP Workstation with a Xeon multi-core processor loaded to the gills with RAM (32GB). I've got 10 4TB HD running in a RAID configuration off of two controllers. I use this workstation as a file server, FTP box, and Plexserver. I also have a decent video card to allow for video transcoding.

This box worked great until I did the upgrade. And, like an idiot, I did not back up the boot device (USB Stick) before the upgrade. I did save the configuration file.

Now? I reboot and it works okay at first. There is still a noticeable hesitation in response my network share. BUT, it deteriorates rapidly within about 6 hours. The Plex server can no longer handle the transcoding by this point and the file I/O speeds via the network are abysmal.

I've tried rolling back to the original installation and there is no difference. I even tried to do a NEW install on a different USB stick, importing the configuration file, and there is still no joy.

I'm concerned because the upgrade modified the ZFS pool to take advantage of new features.

Can anyone give me any possible insight into this that will prevent me from having to move all data off the drives, destroying the RAID, and then doing the whole thing from scratch?

On possible insight here. I had to muck with file permissions, which look to be handled differently. I now get a line message on the FREENAS box about "Dangerous write permissions." And it does appear that the performance deterioration happens as these errors pile up.

Also, the central file share is down to about 1.5 TB. (20 TB total). Does Freenas have problems as the free disk space gets smaller? I thought that was strictly a windows or NTFS problem.

I must admit that I'm very annoyed. Freenas is great when it works, but it is a pain to do actual maintenance on it. A Windows box might not be as good a choice, but it is easier to maintain.

Please don't flame me for being a relative neophyte. LOL. Thanks in advance.
 

pschatz100

Guru
Joined
Mar 30, 2014
Messages
1,184
Please post your system configuration. It is not possible to give useful advice without knowing what the system is.
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
Now that we're all here.

I'm running an older, but serviceable HP Workstation with a Xeon multi-core processor loaded to the gills with RAM (32GB). I've got 10 4TB HD running in a RAID configuration off of two controllers. I use this workstation as a file server, FTP box, and Plexserver. I also have a decent video card to allow for video transcoding.

I'll second the request for the full system spec configuration - HP model number will go a long way to begin with, then let's see CPU type and maybe verify that you're using ECC memory. The other line that scares me is "10 4TB HD running in a RAID configuration" - if you have any kind of hardware or BIOS-level RAID in play, you have made a serious mistake in your configuration and we need to address this.

I reboot and it works okay at first. There is still a noticeable hesitation in response my network share. BUT, it deteriorates rapidly within about 6 hours. The Plex server can no longer handle the transcoding by this point and the file I/O speeds via the network are abysmal.

My knee-jerk response looking at the symptoms here is that it's under memory pressure from plugins fighting with ARC, and there's swapping to disk. If this is the problem, it will be exacerbated by the low free space.

Also, the central file share is down to about 1.5 TB. (20 TB total). Does Freenas have problems as the free disk space gets smaller? I thought that was strictly a windows or NTFS problem.

Fragmentation on disk is a problem with every file system (unless you're 100% flash) - whether it's fragmentation of the files causing slow reads, or fragmentation of the free space causing slow writes. You've got 92.5% space used - it's very likely that many metaslabs are past the fragmentation threshold and the ZFS space allocator has switched from "first fit" to "best fit" - this is slooooow.

I must admit that I'm very annoyed. Freenas is great when it works, but it is a pain to do actual maintenance on it. A Windows box might not be as good a choice, but it is easier to maintain.

ZFS is a complex filesystem certainly, but once you understand some of its intricacies and particularities, it's extremely powerful. No other filesystem will protect your data as thoroughly or fully.
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
Also, the central file share is down to about 1.5 TB. (20 TB total). Does Freenas have problems as the free disk space gets smaller? I thought that was strictly a windows or NTFS problem.

Slideshow explaining VDev, zpool, ZIL and L2ARC
https://forums.freenas.org/index.ph...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/

Terminology and Abbreviations Primer
https://forums.freenas.org/index.php?threads/terminology-and-abbreviations-primer.28174/
Also, the central file share is down to about 1.5 TB. (20 TB total). Does Freenas have problems as the free disk space gets smaller? I thought that was strictly a windows or NTFS problem.
ZFS slows down significantly between 80% and 90% capacity and completely falls off a cliff after it is 90% full.
On possible insight here. I had to muck with file permissions, which look to be handled differently. I now get a line message on the FREENAS box about "Dangerous write permissions." And it does appear that the performance deterioration happens as these errors pile up.
I don't know what you did to your file permissions, but I think you need to review this document and share some details so we can help you.

Updated Forum Rules 12/5/18
https://forums.freenas.org/index.php?threads/updated-forum-rules-12-5-18.45124/
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
ZFS slows down significantly between 80% and 90% capacity and completely falls off a cliff after it is 90% full.

In fairness, ZFS may start to slow down well before 80%, but for the average use case, it does seem to be between 80 and 90%. You can actually get up to about 97 or 98% in very specific cases without seeing much of a falloff, but that's mostly archival "write-once" use.

This is a side effect of the copy-on-write technology ZFS uses to provide many of its advanced features. ZFS trades increased resource requirements in the form of memory and free disk space and gives you higher performance in return. This often comes as a shock to users who want to be able to fill their disks to 99% on a 1GB system. That just doesn't work, and will never work, under ZFS. ZFS works best when it has large amounts of free space in the pool, because it will make good use of those to keep write speeds high, even for random write workloads. ZFS requires lots of RAM in order to cache metadata and actual file data, and when you give it that, ZFS will be able to more easily find free space to write, and is more likely to have your data already read into memory.

I strongly suggest reading the response by @HoneyBadger above, I feel strongly that the answer is probably found therein.
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
Can anyone give me any possible insight into this that will prevent me from having to move all data off the drives, destroying the RAID, and then doing the whole thing from scratch?
There are many ways to add capacity to an existing pool. If you will respond here, we can discuss your options.
 

verisof

Cadet
Joined
Apr 29, 2018
Messages
4
Thank you guys so much, for the quick early responses! It is much appreciated!! And I also greatly appreciate your patience with my being a relative newbie with this OS. I've had many places I've gone to where people are very arrogant if you don't innately understand the intricacies. So, again, THANK YOU!

I found the initial specifications for my setup under General. Is there another place that conveniently lists the entire configuration in an easily digestible format? I note that everyone seems to have the same info in the same formatted way.

Here's what I've got so far:

Build FreeNAS-11.2-RELEASE-U1 Platform Intel(R) Xeon(R) CPU E31240 @ 3.30GHz Memory 32685MB

So, some quick answers. . .


Nope, I deliberately did not use hardware RAID but stuck with the ZFS raid included with the OS. The setup guide I used made it very clear that using a hardware RAID would negate any advantage in the use of FreeNAS or ZFS. So I am using a nice controller, but have it set up as RAIDZ2 I believe. (Where can I go to confirm that?)

I agree with the memory pressure issue. Since the first post, I deleted all the plugins that I didn't really use but just experimented with and the associated jails, leaving only Plex server. I also went through and systematically backed up & deleted all of the media items that are currently available to stream via Netflix & Amazon Prime. (I added approximately 3 TB of free space to the storage).

I did notice an increase in performance, although there are still problems with transcoding. Particularly when I'm trying to stream 4K. I should also note that this same box used to stream 4K video seamlessly before I did the upgrade.

I would definitely be interested if someone could send me to a good link to explain how fragmentation works on ZFS. Because that was my first thought when I saw that I was eating up my disk space.

Thanks again, guys!

--Andrew

P.S. - Other things I would like to do here. I understand that there is an entirely different way of handling JAILS in this new version. I've made a lot of manual changes to the entries in Plex to make the metadata match up properly. I'd prefer not spending another day fixing it all again. Is there a way for me to take the metadata info from the jail in the old version and replacing it with the default stuff in the NEW version of Plex server?
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Fragmentation happens on a copy-on-write filesystem and is a natural inevitability unless you have the very unusual case of being a write-once filesystem.

So if you write sequential data blocks A,B,C,D the filesystem will look for a contiguous spot to write that, and do so. But if you overwrite "C" with "E", what happens is that "E" gets written in some DIFFERENT spot, not on top of "C". This allows for the possibility of snapshots, and also allows the filesystem to remain in a consistent state while the update happens. So perhaps you end up with "ABCDE" on disk, if there was a snapshot, or "AB_DE" if there wasn't, or more likely the "E" ends up somewhere else entirely. When the userland reads the data, now it still gets "ABED" back, but in order to retrieve that, ZFS probably needs to seek in order to fetch the "E" block. If you read the snapshot, though it gets "ABCD" back, and it does so with a sequential read.

Of course this is a totally trite example. You have to imagine this happening with millions of blocks. ZFS will look around for "large amounts" of free space to allocate out of, because this is invariably fastest for writing. So you already noticed that your free space is nearly exhausted. That will be bad-ish.

Unfortunately I have to get going here. I do talk about fragmentation a lot on the forums, so feel free to look around and see if you can find the nice graph I have that shows the steady state performance once fragmentation has reached a high amount. It will show that one a high degree of fragmentation is achieved, ZFS write performance is not that great on pools with greater than 50% occupancy, but that you can throw more free space at it and gain large performance boosts. This is really just basic compsci -- trading one thing for another.
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
If you could share with us the output of the command zpool list
 

verisof

Cadet
Joined
Apr 29, 2018
Messages
4
If you could share with us the output of the command zpool list

Hi Chris,

Here ya go. . .

% zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
data 36.2T 21.0T 15.3T - - 1% 57% 1.00x ONLINE /mnt
freenas-boot 57.5G 780M 56.7G - - - 1% 1.00x ONLINE -
 

Yorick

Wizard
Joined
Nov 4, 2018
Messages
1,912
That's a Passmark of 8k. I have a 1230 v6 with a Passmark of 9k8, and it's almost maxed with a single 4k stream transcode to a phone. It's possible that 11.2 added just enough CPU pressure to make 8k not be enough any more for a 4k transcode.
 
Top