NFS client writes to zfs pool put nfsd in an infinite loop

[reify]

I have a fresh FreeNAS install and testing things on an imported zfs pool it seems that a single large file copy over NFS crashes nfsd. All nfs client queries hang, and top reports nfsd at 100% cpu. Trying to restart nfsd hangs. My only recovery approach so far is to hard reboot the system. There is no log output indicating an issue. It is easily reproducible as almost any writes trigger it.

Edit: `kill -9` on the spinning nfsd process followed by an nfsd restart seems to recover from this much more easily than a reboot.

Problem:

1. Start system
2. NFS client mounts share
3. Client machine performs a large write (~10G via dd reliably crashes it)
4. Operations inside the share start hanging
5. Web GUI still present and working, but all information boxes are forever loading bars
6. top reports nfsd using 100% CPU

System Information:

ESXi Host

• Gigabyte X58A-UD3R (rev 2) with VT-d enabled bios
• i7 930
• 24G RAM, non-ecc
• Syba 2x1Gb PCI-e card (Realtek 8111E chipset)
• LSI 9211-8i HBA IT firmware
• OCZ Revodrive 120G, original model (sil 3124 with 2x 60G drives)
• Vertex 3 SSD as datastore

FreeNAS VM

• 18G Memory
• Single CPU core
• 9211-8i passed through via VT-d with 8x2T SATA disks
• Revodrive passed through via VT-d
• Installed to 8GB virtual disk on the datastore
• Single ethernet (not sure which adapter, odds are a realtek)
• FreeNAS Version 9.1.1 64bit

ZFS Pool

• Created by FreeNAS 9.1, same hardware but without ESXi in the middle
• 8x2T disks in raidz2
• One half of the revodrive as Zil (added after import)
• One half of the revodrive as L2Arc (added after import)
• ~85% capacity

 

[cyberjock]

Well, for starters you are breaking a few cardinal rules for ZFS.

You should assign the VM at least 2 cores. I do 3.

You should not be putting your ZIL and L2ARC on the same device. It's been discussed in this forum plenty of times. It sounds great in theory, but in practice it has its own major problems. I'm no sure how big your ZIL is since you made no mention of the size of your drive, but anything bigger than 2-3GB is MAJOR overkill. Even at 2 or 3GB that's a very fat ZIL for your system and it could easily be argued that 3GB is too big.

Intel NICs are the preferred NIC for VMs(when you are having problems its a no-fail situation for Intel). Otherwise you are welcome to find and hack in the vmxnet3 driver which provide 10Gb support. But considering you are already having problems I'd absolutely stick with the Intel. Your pciconf file says its an Intel. :P

First thing I'd try is getting rid of ESXi. Run FreeNAS on bare metal and see if it still crashes. Virtualizing FreeNAS is like playing with fire. It might work or it might not. But you should rule it out as the cause for your problem. And for troubleshooting purposes I wouldn't be running ESXi when trying new things. Only after you figure out the problem should you go try the solution in ESXi.

FreeNAS absolutely needs ECC RAM to function properly. I just wrote a book on this http://forums.freenas.org/threads/ecc-vs-non-ecc-ram-and-zfs.15449/ like 8 hours ago.

Your board is the one of the boards I experimented with back in December with VT-d. Bad news bro. Gigabyte's implementation of VT-d is not very reliable. So that comment above about playing with fire, is burning much bigger and hotter than you think.

I'm not all bad news though. If you check out the CPU compatibility list, some Xeons are supported(and they are cheap on ebay). You should be able to drop one in and buy some ECC RAM and be able to reuse your motherboard. I used the E5504 and it smoked FreeNAS. But if you are still sworn to ESXi, then you'll need a CPU with more power for your VMs.

 

[reify]

Thanks for the response.

The single core VM is in response to a series of posts I read recently but cannot find about FreeNAS in ESXi having "interrupt flood" crashes, which were fixed by doing a single core. I don't do anything close to straining it, so I doubt this is a performance issue. Non-crashing I hit 10% on my maximum load. That being said, if I'm not going to have crashes because of it I'll definitely give it all 4/8 cores.

The Zil/L2ARC description I gave is a bit ambiguous, the Revodrive is a pair of sandforce controllers attached to a si3124 fakeraid card. Skipping the raid gives two 60G drives to the OS, so I have one set up as Zil one as L2ARC. The 60G Zil is overkill, but it does the trick and I have it.

The system was running fine, same hardware, bare metal, so my fear was that the crashing was the VT-d implementation. The whole process of setting it up seemed super sketchy. What testing did you do?

The ECC ram thing is a pain, the issue there is I have this hardware and I don't have ECC ram capable hardware. In an ideal world I'd change hardware, but I can't. My strategy is just optimism + no critical data stored on the pool. This is also the motivation for ESXi - I have one server machine and would like a more than just FreeNAS.

 

[cyberjock]

The system was running fine, same hardware, bare metal, so my fear was that the crashing was the VT-d implementation. The whole process of setting it up seemed super sketchy. What testing did you do?

.

My testing was just trying to get it to work reliably. No matter what I tried it was riddled with error messages and problems. The errors and problems were nondescript and google was no help either. The fix(and even the problem) was non-apparent after a week of solid experimenting. Ultimately I gave up on it as VT-d is normally reserved for server grade boards only and Gigabyte's boards(while I use them in every desktop I own and I love them) are not server grade. If they didn't do thorough testing of VT-d(which I wouldn't expect them to do since its a desktop board) then there could be bugs in VT-d that you and I cannot identify or even fix. Of course, as soon as I got rid of ESXi and went back to bare metal everything worked flawlessly. I ended up buying server grade parts and built a whole new machine. At some point you just have to accept that your time is worth more than trying to fix something you might not be able to fix. So I saved up and bought new hardware.

The ECC ram thing is a pain, the issue there is I have this hardware and I don't have ECC ram capable hardware. In an ideal world I'd change hardware, but I can't. My strategy is just optimism + no critical data stored on the pool. This is also the motivation for ESXi - I have one server machine and would like a more than just FreeNAS.

And your reasoning is why many people try to do the same thing. Most get away with it without problems, but there are a few that lose their pools and its data forever. The risk and reward is yours. My problem is that you won't know the bus is coming until its driven over you. I prefer to know when something is failing so I can take action. Hard drives, if properly monitored, will usually warn you before dying(I say usually loosely). RAM failing is without warning, and it could be days or weeks before you figure it out. By then, its too late. Only 1 person has not lost their pool from bad RAM. They just happened to be testing their system and had a weird problem.

 

[jmichealson]

Hoping to revitalize this post... I've recently attempted to utilize an all in one ESXi , FreeNAS solution. Previously I had been using CentOS as the NFS server and a 8888ELP card. 4x 2Tb disks R5. Worked well for years. I have a few different scenerios for IOmeter. Decided to upgrade the single proc super micro to a dual 6x core with more ram. Did that, ended up getting a IBM card and flashing it to LSI to use as a JBOD as the 8888ELP simply wouldn't pass through to FreeNAS properly. Everything works properly, even used the ESXi vmxnic3 driver I found on the forum here. Problem is NFS on the 10Gb vmic is 20Mbs compared to 128Mbs on the raid 5 and much higher on a single SSD datastore.

Willing to post any data about the system, anyone have any thoughts? Ping @ 9k frames works fine... wireshark really doesn't show fragmentation like I expected....?

Tell me to start a new thread if that's necessary.

Thanks!