NFS Share Hangs after 9.2 Release Upgrade

[clownphish]

Hey there,

Has anyone else had issues with accessing files over NFS after 9.2 RC upgrade? I installed it the other night and since then I noticed my media PC wouldn't play videos from my FreeNAS box. SSHing into the media PC and running an ls on the NFS share would also hang the SSH session (control-c wouldn't break out of it then). Establishing a new SSH session and checking nfsstat didn't show anything unusual.

I ran iperf to confirm that the network was still working properly, which it seemed to be (around 790Mbit/s if I recall)

I didn't think it was a FreeNAS problem for a while so I played with different NFS mount options on the client, remounting inbetween and restarting the FreeNAS NFS service; however, the media PC worked fine with 9.1.1 Release with no tweaking which led me to start looking at the FreeNAS config.

I tried removing autotune, deleting all tunables and sysctl values, increasing the number of server on NFS, multiple reboots. The problem seemed intermittent. I could do a few accesses after a reboot but very quickly (few minutes) the media PC would hang on trying to access a file over NFS. At this point doing an ls on the share also hangs.

SSHed into the FreeNAS serer and all looks fine (nothing spiking CPU, low memory usage, all files accessable).

I tried downgrading to 9.1.1 from 9.2 release but it looks like this isn't possible. Eventually I burned 9.1.1 to a new USB stick and restored my config backup (taken before the 9.2 Release upgrade). Rebooted everything and I have no problems again.

I'm loathe to break my setup again but I do want to move to 9.2 at some point. Is there things that people would recommend trying to narrow down the problem?

Info on my systems:
FreeNAS-9.1.1-RELEASE-x64 (a752d35) (this is the working version - the problem was/is on 9.2 Release)
Platform AMD E-450 APU with Radeon(tm) HD Graphics
8 GB RAM
4x 2TB drives in RAIDZ1

Media PC is a Zotac ZBOX with an Atom D510 @1.66Ghz and NVIDIA ION GPU running Xubuntu 12.10 with XBMC 12.3.
Videos are mounted via NFS in a subdirectory under /media/
Music files are also mounted via NFS on a WD MyBook World (which worked fine throughout)

Connected into the same Asus Gigabit switch. Autonegotiation settings look good on both - both running at 1000Mbit/s full duplex.

 

[clownphish]

No one have any ideas? Should I/can I file a bug?

 

[cyberjock]

Well, there's no real data to what is broken except "NFS doesn't work". That leaves so much code to search for, and you're the only one with this problem that I've seen. I think you're going to need to have more evidence than just "its broke" to actually solve the problem.

What is your NIC? Are you using onboard? If so, what is your motherboard?

 

[DJ9]

Has anyone else had issues with accessing files over NFS after 9.2 RC upgrade

The 9.2 release is now out, did you try it on that vs the RC?

 

[clownphish]

Well, there's no real data to what is broken except "NFS doesn't work". That leaves so much code to search for, and you're the only one with this problem that I've seen. I think you're going to need to have more evidence than just "its broke" to actually solve the problem.

What is your NIC? Are you using onboard? If so, what is your motherboard?

Fair point. I'll try and gather what I can!

My NIC is a Realtek 8111E and it is indeed onboard. My motherboard is an Asus E45M1-I DELUXE.

In response to DJ9, sorry I was actually running the release version (FreeNAS-9.2.0-RELEASE-x64 (ab098f4)), not the RC.

I've tried upgrading my media PC to the latest version of Ubuntu (13.10) but the problem still exists with 9.2. FreeNAS 9.1 still works great.

 

[cyberjock]

Realteks are a major problem for users. In your case I'd get an Intel NIC and try that to see if it solves the problem. Realteks often suck so badly there's no indication that they are the cause. But time and time again people buy an Intel and the problem disappears.

Other than that, I'd do a RAM test and make sure your ram is good(www.memtest.org).

I will say that 95% of people that do custom settings for protocols, sysctls, and tunables get them horribly wrong. So I'd get rid of any that you might have. FreeNAS has some great default settings and they work for nearly 100% of users. You shouldn't be using autotune either, so if that's enabled get rid of them.

 

[clownphish]

Thanks for the replies

Realteks are a major problem for users. In your case I'd get an Intel NIC and try that to see if it solves the problem. Realteks often suck so badly there's no indication that they are the cause. But time and time again people buy an Intel and the problem disappears.

While I take your point (and Realtek are by no means my favourite), I have to point out that it's worked happily since FreeNAS 8.3. I do have one expansion slot on my case & board so I'll see if I can grab an Intel NIC next week and test that. Assuming that works fine, is this pointing to an issue with the Realtek drivers in FreeBSD 9.2 causing the issue (if all is fine in 9.1)?

Other than that, I'd do a RAM test and make sure your ram is good(www.memtest.org).

I'll give this a go too, just to be safe. Again though, if everything works in 9.1 I'd not expect it to just stop in 9.2

I will say that 95% of people that do custom settings for protocols, sysctls, and tunables get them horribly wrong. So I'd get rid of any that you might have. FreeNAS has some great default settings and they work for nearly 100% of users. You shouldn't be using autotune either, so if that's enabled get rid of them.

All tuneables gone now, except the one to enable USB3. I do have jumbo frames enabled on my computers which I could disable to test.

Any more ideas, do shout.

 

[cyberjock]

While I take your point (and Realtek are by no means my favourite), I have to point out that it's worked happily since FreeNAS 8.3. I do have one expansion slot on my case & board so I'll see if I can grab an Intel NIC next week and test that. Assuming that works fine, is this pointing to an issue with the Realtek drivers in FreeBSD 9.2 causing the issue (if all is fine in 9.1)?

Very possible.

I'll give this a go too, just to be safe. Again though, if everything works in 9.1 I'd not expect it to just stop in 9.2

Except 9.2 has a new kernel and might use RAM in a different fashion that makes this problem come to light. ;)

All tuneables gone now, except the one to enable USB3. I do have jumbo frames enabled on my computers which I could disable to test.

Oh my.. Jumbo Frames can be a major nightmare if all of your hardware doesn't support it. And for 99.99% of home users you probably don't have hardware that supports it to the full extent you are needing.

Quite frankly, I'd practically bet money jumbo frames is your problem now that you've mentioned you use that. And the common diagnosis that jumbo frames isn't supported.... random hangs on file transfer. Sound familiar?

 

[clownphish]

Just an update on this one from me. I saw 9.2.1 was out so I duplicated my working USB stick with dd, then booted into it and performed an upgrade via the GUI. Rebooted and had the same problems as before. It also seems to take out the wireless network on my router (which is a Cisco RV120W).

I disabled jumbo frames (set the MTU back to 1500) on the Realtek NIC, rebooted and everything seems to be working happily now. Based on this behaviour it looks to me like something change in the Realtek drivers between 9.1 and 9.2 regarding jumbo frame operation. Is it worth opening a bug on this issue?

Thanks to everyone who helped troubleshoot this.

 

[cyberjock]

To be honest, not really. Don't stop on my account. Realteks are known to be complete crap. Their support on FreeBSD is virtually nonexistent. So while you can put a ticket in, the chances of actually getting a fix that solves your problem is almost zero. I'd expect the ticket to be closed to '3rd part to resolve' as that's not really "our" problem.

I'd highly recommend you stick with the recommended NICs.. Intel. We say it again and again, but people still go with Realtek. :(

 

[clownphish]

Understood. As I said before, Realtek is certainly not my NIC of choice and I understand vendors have hardware compatibility lists for a reason. However, it is the on-board chip that comes with the motherboard which was why I wanted to use it. Motherboard choice (Asus E45M1-I Deluxe) was driven by wanting a very quiet system (it was next to the bedroom) with as much memory as I could get, 5 SATA connections and a low-power consumption processor (preferably passively cooled). It's served me very well so far, other than this nasty driver issue.

Network seems to be working nicely without the jumbo frames now. Who knows, maybe the support will improve on FreeBSD in the future :)

 

[cyberjock]

Ok.. message received. But do you realize that almost everyone has the same excuse when the show up in the forums and pollute it with the same 'my realtek nic doesn't work' posts? Seriously... go check out other threads. They always end with:

1. But it came with my motherboard
2. But I can't afford an Intel
3. But I don't have a free slot.

And maybe next time you'll use a motherboard with the actual recommended hardware....

smh