Adjusting tunables using web UI freezes FreeNAS for 3-4 minutes

[psouza4]

I performed an upgrade from:

FreeNAS-8.3.0-ALPHA-r11952-x64​

To:

FreeNAS-8.3.1-ALPHA-r12356-x64​

In an effort to try and resolve some performance issues I have with FreeNAS. However, this new version has a pretty severe bug: any time I adjust a tunable using the web UI, the entire FreeNAS box hangs/freezes for a few minutes (I timed this last one -- it was 4 minutes and 10 seconds long) with the 'Please wait...' status. Any SSH session I have is also hung as well as any SMB activity. After a few minutes, the system will resume and the adjustment to the tunable will be recorded (additions, removals, enable/disable, alterations are all recorded okay). This did not happen with the prior version.

The performance in FreeNAS 8 has been woefully poor compared to FreeNAS 7 and while I presume it's a matter of configuration, I just can't get to the bottom of it with a problem like this, so I'm turning it loose here in the hopes that someone has any ideas. Additionally, I tried modifying /boot/loader.conf.local in vi, but the changes do not persist upon reboot so I am effectively stuck with the web UI. I would try a different build, but during my journey of troubleshooting the performance issues, I upgraded the ZFS pool (35.1 TiB usable capacity that I can't just offload and start over with) so I'm more or less stuck with current builds.

Any thoughts?

 

[Stephens]

My first thought is you're rather brave to experiment with things like alpha releases and upgrading pools on a live NAS if you don't have backups. What happens if a really bad bug is introduced in one of the Alphas and you lose 35TB?

Also, I think you're somewhat stuck in trying to look at FreeNAS 8 through a FreeNAS 7 filter. I guess I'm lucky in that way in that I never used FreeNAS 7. Starting from ground zero and learning can be easier than unlearning first. I say that because there are many posts here talking about the fact that FreeNAS 8 basically loads into ram and runs, so modifying most files directly won't work as you're just modifying a RAM copy. You have to follow the instructions provided on the forum here (several times) to make "sticky" changes. There's no shortcut for reading the manual (and forum) to learn how FreeNAS 8 works. Or rather, there is a shortcut, but it's a risky way to live.

It's been stated here that v28 pools are slower than v15 pools even if no additional features are used. But really, you haven't specified what performance issues you're having and what you're doing to attempt to address them. Then only thing you've said so far is the Web GUI is slow in r12356. If others report the same using the same version, time for an official bug report (which isn't here). I would try r11952 again just to make sure the GUI slowness problem doesn't show up there too now. If it shows up there too now, something else is going on. Oh, and hopefully you're using multiple flash drives for multiple versions, not overlaying.

 

[psouza4]

My first thought is you're rather brave to experiment with things like alpha releases and upgrading pools on a live NAS if you don't have backups. What happens if a really bad bug is introduced in one of the Alphas and you lose 35TB?

Then I lose 35 TB of data (or rather, the part that's occupied). I'm no dummy and I'm not here to be scolded on best practices, I'm explaining why I don't have the luxury of rolling back to a different FreeNAS build (i.e v7, 8.2, whatever). Rather than go through several posts back and forth of "well try this build, or that build": now you know up front that I can't do that.

Also, I think you're somewhat stuck in trying to look at FreeNAS 8 through a FreeNAS 7 filter. I guess I'm lucky in that way in that I never used FreeNAS 7. Starting from ground zero and learning can be easier than unlearning first. I say that because there are many posts here talking about the fact that FreeNAS 8 basically loads into ram and runs, so modifying most files directly won't work as you're just modifying a RAM copy. You have to follow the instructions provided on the forum here (several times) to make "sticky" changes. There's no shortcut for reading the manual (and forum) to learn how FreeNAS 8 works. Or rather, there is a shortcut, but it's a risky way to live.

I realize you've probably run into many times more users that cut corners and need hand holding through even basic steps than users who are simply reporting a very specific problem, but there's really no call to be abrasive. I'm reporting a bug that causes the FreeNAS box (even the console) to completely freeze in this build -- if I can get some kind of confirmation, I'll post it to the bug tracker. If I'm doing something wrong, or this is a known issue that I can work around, then all the better.

It's been stated here that v28 pools are slower than v15 pools even if no additional features are used.

I haven't lost performance between v15 and v28. It's the same (I was using v15 with 8.3 r11952 for the last two months). As a side note, scrubs and reslivering performance are supposed to have been improved in v28 pools and was one of the reasons I upgraded, but I haven't come that far yet and have not tested their performance. This thread isn't about that.

But really, you haven't specified what performance issues you're having and what you're doing to attempt to address them.

That's because this thread isn't another "help me with my performance" thread. It's about one bug. That's all I'm posting about right now. I only mentioned the performance in passing so you can appreciate why I upgraded to the latest alpha build at all.

Then only thing you've said so far is the Web GUI is slow in r12356.

It's not actually -- it's fast and very responsive. I said that one specific feature (adjusting tunables) hangs the machine via ALL interfaces (web, SSH, console, FTP, NFS, SMB, you name it) for a few minutes. No data in or out. After that, the machine resumes. This behavior is unique to this build.

Oh, and hopefully you're using multiple flash drives for multiple versions, not overlaying.

I'm using the same flash drive and used the GUI update method. There's no disclaimer along the way indicating I should be using a backup flash drive or anything like that, just to backup your configuration file (which I did in advance). If there's more that should be done, it should likely be documented (and I don't mean buried somewhere in the forums or commented on an issue in the bug tracker).

I have no idea where all the hostility is coming from, but it's really undue.

 

[cyberjock]

I don't think the hostility was intentional. Stephen and I both see alot of people with no clue what they're doing show up and say "my server is broken.. plz fix" and "this must be a bug". There are exceptions of course.

I don't have any idea how to fix your performance issue or your bug, but is it possible that the 2 are somehow related? I did just install the x64 12356 build and I had no problem changing/adding tunables. /shrug

Maybe you should try a blank USB drive and a default config and see what happens if you change tunables. If it works at least you've ruled out a hardware issue and it is likely related to a configuration setting you have.

Edit: If I didn't think you knew what you were doing I wouldn't be trying to help you.. and Stephen wouldn't have replied. ;)

 

[psouza4]

Thanks for your follow-up. I really wanted to stress that I didn't come to the forums to look for help with my performance issue. I only wanted to even mention it to give a bigger picture of why I was running experimental builds on my production data. I may, at some point, post elsewhere soliciting suggestions and ideas for my performance troubles with v8, but I'm not at that point yet -- I still have a lot to troubleshoot on my end first.

Part of that troubleshooting is coming in the form of new equipment early next week and then a new way to attach these drives to the server. Once done, I'll do some more testing with r12356, and probably replace it with another build as necessary, and will update this thread again. I really just want this thread to be about the discussion of a possible bug with the UI (and/or whatever happens under the covers when swapping out tunables).

 

[psouza4]

Just an update: I went and used the console option #8 to "Reset to factory defaults", rebooted, and tried to add a tunable (even with no drives attached) and it froze again. I'm going to try a different build now.

 

[psouza4]

Further update: same problem in version 8.3.0 BETA 3.

I exported my zpool, wiped the USB thumb drive, imaged it from the 8.3.0 BETA 3 .xz image contents (FreeNAS-8.3.0-BETA3-x64.img.xz), booted fresh, and attempted to add a tunable ('vm.kmem_size'). I haven't even tried importing my pool or anything (so I have no user mount points currently).

 

[cyberjock]

Maybe try a different USB drive? I know I have a USB drive that was terrible at writes. The delay was so long when any amount of data had to be written that I threw the USB in the trash.

 

[psouza4]

A swing and a miss, I'm afraid -- I wish it were something so simple.

Writing to the USB thumb drive (from a host OS doing imaging or FreeNAS itself) is fine, so the catch has to be somewhere else. It's so bizarre that adjusting any other setting in the system (I started fresh after all) is completely fine: it's only these tunables that are causing me grief. Somewhere between r11952 and r12421M/r12356, something seems to have broken.

Is ZFS v5/v28 available in 8.3.0 alpha r11952? If so, I might try to revert that far back to when I know tunable adjustments weren't causing this behavior.

(P.S.: I've resolved my earlier performance issues -- and as expected, were not a problem with FreeNAS, but with hardware configuration.)

 

[William Grzybowski]

Tunable remount the root (/) to write /boot/loader.conf.local. The others settings do not.

I would take noobsauce80 advice.

 

[psouza4]

Tunable remount the root (/) to write /boot/loader.conf.local. The others settings do not.

I've dropped to shell, remounted /, and ran iozone and dd to test performance on that partition -- speed is fine.

I would take noobsauce80 advice.

To ditch the USB drive? Based on symptoms I've posted above, it'd seem silly to go out and buy another USB drive simply to prove that that's not the issue. I'll do it if it comes to that, but there's little reason to at this point. It'd be nice to know precisely what steps django is taking when it receives a request to update a tunable - I know there's some database backend stuff, etc., but if I had the specifics, I could run some commands from a shell prompt and find out where the sticking point is.

 

[William Grzybowski]

I already told you what is the difference...

It is not about the speed, but sync from cache virtual memory to the device.

# time sh -c "mount -uw / && echo > /boot/loader.conf.local && mount -ur /"

or:

# time sh /etc/rc.d/ix-loader start

 

[psouza4]

It is not about the speed, but sync from cache virtual memory to the device.

# time sh -c "mount -uw / && echo > /boot/loader.conf.local && mount -ur /"

or:

# time sh /etc/rc.d/ix-loader start

This part was critical: thank you!

So here's what I'm seeing:

# time sh -c "mount -uw / && echo > /boot/loader.conf.local && mount -ur /"
0.000u 0.019s 4:41.49 0.0% 40+2272k 1+3182io 0pf+0w

# time sh -c "mount -uw / && mount -ur /"
0.000u 0.019s 4:48.96 0.0% 60+3408k 0+3178io 0pf+0w

# time sh -c "mount -uw /"
0.000u 0.001s 0:00.57 0.0% 0+0k 0+5io 0pf+0w

# time sh -c "sync"
0.000u 0.001s 0:00.00 0.0% 0+0k 0+31io 0pf+0w

# time sh -c "mount -ur /"
0.000u 0.017s 4:44.34 0.0% 60+3408k 0+3171io 0pf+0w​

Why it would take so long to mount back to a read-only state is beyond me, especially considering that (a) nothing was written to the disk anyway, (b) a sync didn't cause the issue, and (c) this problem didn't occur in the first build and suddenly occurs now.

I've went and placed an order for new thumb drives:

http://www.newegg.com/Product/Product.aspx?Item=N82E16820211572
http://www.newegg.com/Product/Product.aspx?Item=N82E16820226078
http://www.newegg.com/Product/Product.aspx?Item=N82E16820220252​

All three have excellent reviews and decent IO speed, so I expect good performance from at least one of them. It's still very curious why this problem would suddenly manifest when it wasn't a problem before. EVen if there was a bad fstab or something, this is a fresh format and re-image.

 

[psouza4]

The issue happens with all four USB drives, although the time varies from device to device. The quickest was the A-Data, which advertises a 30 MB/s write speed (although I typically see about 22 MB/s in practice). It took half a minute to adjust a tunable (every time):

# time sh -c "mount -uw / && echo > /boot/loader.conf.local && mount -ur /"
0.000u 0.018s 0:25.21 0.0% 20+1136k 0+2773io 0pf+0w

# time sh -c "mount -uw / && echo > /boot/loader.conf.local && mount -ur /"
0.000u 0.017s 0:26.51 0.0% 180+10224k 0+2778io 0pf+0w​

I mean -- I guess I can live with waiting 25+ seconds every time I want to change a tunable (it's not that frequent I need to, once I dial in my settings), but it's still much, much longer than before. I still say there's an issue somewhere.

(P.S.: these tests were conducted on a fresh image write using v8.3.0 BETA 3 and with no drives attached)

 

[i4m]

thanks for your persistance

thanks for your persistance psouza4, i too have noted this issue with 8.3. are you still seeing this issue or have you overcome it now?

Thanks

 

[psouza4]

thanks for your persistance psouza4, i too have noted this issue with 8.3. are you still seeing this issue or have you overcome it now?

I was forced to use a different build of FreeNAS where the bug was not prevalent. I rolled back to FreeNAS-8.3.0-BETA3-x64 (r12421M) and have not experienced the issue since. I'm afraid to upgrade as the bug may resurface (the development team could not confirm the issue, so there's a strong chance the issue still exists). I use a fairly vanilla setup and, as you've probably read above, had the issue with multiple flash drives, so I'll probably remain on this build for some time. Best of luck!

 

[cyberjock]

I've seen the issue myself with FreeNAS 8.3 p1 x64. I'm not sure of the exact cause but for me it seems to be random. Sometimes if I make a change it will take a couple of minutes, sometimes it won't. It's only an inconvenience for me since I don't change stuff often.

 

[i4m]

i'm on FreeBSD 8.3-RELEASE-p4 (FREENAS.amd64) #0 r241385M,
i'm a noob to freenas (long time ipso, ios & junos net engineer trying to find what i need to know to do what i need to do), looking for a backup solution for home media and laptop backups. i'm trying to tune the system hence running into this issue. i'll look to roll back to FreeNAS-8.3.0-BETA3-x64 (r12421M) too. can i assume i'll retain my data by exporting my zpool?
Thanks again!!