FreeNAS Mini keeps locking up.

[Ste]

I set up my new Mini (description in signature) today, with two iSCSI targets and added it to my Xenserver 6.2 setup (a pool of two Dell R810s), which is already configured with two iSCSI-based storage repositories on another server (a Dell PE 2950, running 9.2.1.2). In between everything is a Dell N2048 switch. My existing setup has been stable since I set it up in March (except when I updated the 2950 to 9.2.1.7, whereupon it started rebooting every few minutes - reverting it to 9.2.1.2 resolved that, but that's another story).

I attempted to migrate my virtual disks (I have 16 of 50GB each) from the 2950 to the Mini. Ten minutes into the first one, the Mini became unresponsive, stalling the transfer, and required a reset to recover. Once it came up, XenServer continued the disk transfer and it completed ~20 minutes later. I tried a second one and it went flawlessly, completing in 29.5 minutes. Twenty minutes into the third one, the Mini has gone unresponsive again. I've not yet reset it, but will shortly. I saw nothing in the logs to indicate why it locked up the first time, and am not expecting to this time, either (but would love to be proven wrong).

Not sure where I go from here. Suggestions?

 

[Ste]

Additional info: No, nothing in the logs to indicate why it goes unresponsive. Thinking about it, this is what I saw when I updated the 2950 to 9.2.17, too - it either rebooted or simply became unresponsive and had to be rebooted, every half hour or so, and nothing in the logs to indicate why. Is there something wrong with 9.2.1.7, perhaps?

 

[Ste]

Reading the forum rules, I see I should be providing some additional info. Everything about the Mini is supposedly supported, as it was purchased in this configuration from iXsystems.

It isn't clear to me what info I should provide in this case. I have generated the Debug info, but it's a rather huge amount of stuff to paste here. I'll happily post anything that's requested, however.

 

[cyberjock]

Ste,

Not sure, but usually systems going unresponsive is a sign of hardware problems. I'd recommend you put a ticket in with iX and see what is going on. My guess is you are going to see a new motherboard sent to you very soon. :P

 

[Ste]

I opened a ticket with them last night. In it, I selected the option to be contacted by phone. I described what I did here (basically tweaked my original post here, and added a little more info). This morning I received an email that was all about VMware ESXi and troubleshooting steps using ESXi commands and URLs to items in VMware's knowledge base. To top it off, they spelled my name wrong. They appear to have given my ticket a cursory glance, saw that virtualization was mentioned in it, and fired off a canned response.

I'm not impressed or amused. We'll see where it goes from here.

 

[cyberjock]

I'm gonna PM you...

 

[Ste]

I should update this to say that, after the initial communications hiccup, the support has been excellent and what I'd expect, if not better even. After a few more tests on my end (I can reliably get it lock up when transferring the 50 GB VDIs to it), I sent them the test results and FreeNAS's debug output, and they are going to send me a new motherboard. I did discover I'd used a CAT5e cable instead of the CAT6 cable they'd included with the Mini, but using that didn't solve the issue (I'll still keep using it though - and I noticed some of my other cables aren't CAT6 - can anyone recommend good ones from Newegg or someplace?).

I'll update this again, once we get this thing working right, which I have no doubt we will. :)

 

[cyberjock]

I get my cat6 cables from monoprice. Lots of colors to choose from, quality seems to be on-par with other companies, price is pretty low (don't do a bunch of small orders as shipping can get expensive), and they've got a variety of kinds to choose from.

 

[Ste]

Thanks! I'll check them out.

 

[rabiat]

I should update this to say that, after the initial communications hiccup, the support has been excellent and what I'd expect, if not better even. After a few more tests on my end (I can reliably get it lock up when transferring the 50 GB VDIs to it), I sent them the test results and FreeNAS's debug output, and they are going to send me a new motherboard. I did discover I'd used a CAT5e cable instead of the CAT6 cable they'd included with the Mini, but using that didn't solve the issue (I'll still keep using it though - and I noticed some of my other cables aren't CAT6 - can anyone recommend good ones from Newegg or someplace?).

I'll update this again, once we get this thing working right, which I have no doubt we will. :)

It seems you might be hitting the same issue I had with my C2750D4i motherboard explained in this thread.

 

[Ste]

The new motherboard solved the silent locking up issues, but then I started seeing kernel and iSCSI driver crashes, under load (transferring 50GB VDIs to/from it - 1, 2 0r 4 at a time). We looked at my configuration and I was told it wasn't configured optimally, so the following changes were made:

Changed from RAIDZ2 to RAID10.
Added the ZIL.
Added the L2ARC.
Set 'sync=always' on the ZVOLS underlying each iSCSI storage repository.
Set up MPIO, including editing XenServer's multipath.conf file, as described in these forums, so it actually did round-robin the paths.

Doing the above cause the software crashing to stop, but performance pretty much tanked. Whereas I could pull 4 50GB VDIs off of the Mini in 4 hours, previously (with the occasional lockup), it was now taking up to 12 hours to do it.
Then one of the disks in the Mini developed a bad sector. They said they'd send me a replacement, but currently have none in stock and are waiting on the manufacturer to ship them one. After waiting the better part of a week, I had Amazon overnight me a drive, so that problem is resolved. Eventually they'll send me a drive that I'll put on the shelf.

Since every "improvement" to my iSCSI setup improved the stability of the system, but made performance worse, I decided to switch to NFS storage repositories and give them a test.
I left 'sync=standard' on the datasets underlying the NFS shares. It wasn't clear to me if I should set it to "always" or not, as with iSCSI. The ZIL gets a lot of use when transferring VDIs to the Mini, so it seems like "standard" is okay (correct me if I'm wrong, please).
Since doing that, performance is much better, but I've had one spontaneous reboot and I've seen ahcich timeouts on various disks when transferring VDIs (no more than 2 at a time) to the Mini.

I'm coming to the conclusion that the Mini might not be appropriate for my use case: using it as the NAS for two XenServer 6.2 servers running 18 VMs (which might grow to 25). The VMs mostly idle, from the Mini's POV, and it handles them, in their running state, just fine. But when I want to move around VDIs, I seem to have one issue after another. I'm open to making further configuration changes for either the iSCSI or NFS setups, as maybe I'm still not configuring them as optimally as I could, but I'm going to have to decide soon, I think, whether to keep this or return it.


(Edited to reflect that they currently have no disks in stock - they told me they usually do.)

 

[titan_rw]

I don't have a freenas mini, nor do I run xenserver, but I do something similar with my esxi server, and generic freenas box.

I copy all my vmdk's over nfs every night from esxi to freenas. I recently added a ZIL (slog) to do things the right way. Previously was running sync=disabled. With sync=disabled I was getting ~113 MB/sec. Adding the slog and setting sync back to standard, I'm down to ~75 MB/sec if I remember right. Not bad for obeying sync.

I've been doing this for 4-5 months without any problems. I also push windows backups to the same box over cifs every night as well. Another freenas machine also replicates stuff to it too.

Ohh wait. I thought you were only using freenas as a backup destination. It seems you're using it for live vm's? I probably read the thread wrong. I only have one vmware host at home, so I run local storage on it.

Sorry, I really have nothing to add to your problem.

 

[Ste]

So, to follow-up ...

After my last post, iXsystems asked me to send the Mini back to them. They determined that the SATA chipset on the motherboard was defective. They built me a new Mini (but keeping my drives). Aside from exactly two ahcich timeouts - neither associated with spinning platter drives (one was with the L2ARC SSD and one was with the SATA DOM boot "drive") - the system has behaved flawlessly. The timeouts were during large, sustained transfers, when the SATA bus would have been quite busy. I've been told by iXsystems that the occasional timeout, successfully retried, is not something to be concerned with.

Given that, I think I finally have a working Mini.

For those who've lost count, this is after replacing the motherboard twice and replacing a hard drive.

Would I buy another one? If this one continues to perform well, then yes, I would (and might, so that I can replicate to it). But you can bet I'll beat on it during testing, and I'd recommend you do too, if you get one. The folks at iXsystems know their stuff and were very helpful, but I do have reservations about the quality of the motherboards their supplier is providing them.

-ste

 

[Ericloewe]

So, to follow-up ...

After my last post, iXsystems asked me to send the Mini back to them. They determined that the SATA chipset on the motherboard was defective. They built me a new Mini (but keeping my drives). Aside from exactly two ahcich timeouts - neither associated with spinning platter drives (one was with the L2ARC SSD and one was with the SATA DOM boot "drive") - the system has behaved flawlessly. The timeouts were during large, sustained transfers, when the SATA bus would have been quite busy. I've been told by iXsystems that the occasional timeout, successfully retried, is not something to be concerned with.

Given that, I think I finally have a working Mini.

For those who've lost count, this is after replacing the motherboard twice and replacing a hard drive.

Would I buy another one? If this one continues to perform well, then yes, I would (and might, so that I can replicate to it). But you can bet I'll beat on it during testing, and I'd recommend you do too, if you get one. The folks at iXsystems know their stuff and were very helpful, but I do have reservations about the quality of the motherboards their supplier is providing them.

-ste

The ASRocks seem to have been hit-or-miss, tending towards hit.
The Atom's connectivity limitations don't help when you're trying to build an NAS motherboard, so I'd give them a bit more of a benefit of the doubt, besides that from them being relatively new to the server motherboard market.