Freezing after heavy transfers

[thatguuuy]

I've done a lot of searching but haven't found any solutions (although a lot of different possible things to try and rule out).

Problem:

Random hard freezes of FreeNAS (v9.2.1.5) that overwhelms entire home network at random times, usually while streaming media or transferring large files across network. There's no obvious rhyme or reason as to why it's happening.

Specs:

Supermicro X10SLM-F (BIOS v 2.0)
Samsung DDR3-1600 8GB/1Gx72 ECC CL11 Server MemoryD38GRE160S
Intel Core I3 4130 CPU
3x3TB HGST Deskstar NAS 7200RPM drives -- ZFS RAIDZ
4 x 120MM case fans
430W CX430M Bronze 80+ PSU
8gb USB 2.0 drives (Sandisk, Microcenter, HP) - Currently using Sandisk
Connected to home network via gigabit switch


Attempted:

New flash drives
Clean install and fresh user/group and all settings
Wiped and created new pool from scratch
All sensors read normal
Updated BIOS
Updated to 9.2.1.6RC
Disabled plugins

Details:

I first discovered the issue when I thought my ISP went down, but after connecting the modem directly to the PC, I discovered it worked. I didn't have access to any device, hardwired or not, including connecting to my HTPC. After figuring it was the router, I went to pick one up before the store closed. After getting it set up, while having only my desktop hardwired, everything worked fine until I plugged in the rest of the devices (FreeNAS box included). Everything instantly stopped working.

After unplugging the NIC connection, but leaving IPMI connected, everything returned to normal. Logging in via IPMI's remote console, the FreeNAS OS was completely frozen.

The router's traffic stats page showed there was a whopping 2GB/sec being transferred outbound. I ran Wireshark to see if I could pick up any packets being broadcast from the server, but I wasn't getting anything relevant at all, so no luck there.

I've ran Memtest for a few hours with no errors, did several short and long SMART tests with no errors, and all sensors read normal temperatures and voltage. I've tried clean installs with several flash drives and even wiped the pool and started fresh once.

I'm currently running SABNZBD, CouchPotato, and Sick Beard, and Plex... all of which work fine (as far as I know). I have CIFS enabled, but haven't used it in a few days and rarely access it.

I checked /var/log/messages and there are zero errors, or messages at all for that matter, from the last successful use and after freezing each time. I've attached a screenshot of the log which shows the freezing sometime between 1:45AM and 11:30AM. This has occurred a few times in the last few days.

As a temporary measure, I've disabled Couchpotato and Sickbeard as they are not 100% needed, but have left SABNZBD and Plex up for now. I also updated to v9.2.1.6RC that was just released.

Is there anything I should be looking out for? For all I know, 9.2.1.6 could fix whatever bug I was experience, since it haven't frozen since this morning, but I'd like to get a head start. Thanks for reading and thanks for any help.

 

[cyberjock]

Well, I'd start with stopping all plugins and jails and see how it performs. There's been some random problems that seem to be Plex-related. But as I don't run Plex on my FreeNAS box I can't really say for sure what the problem is as I haven't seen it first-hand.

Your hardware is quite good and I doubt it's a hardware problem aside from potentially bad hardware.

 

[joeschmuck]

Well I have a few questions and suggestions mixed in:

1) Please describe your network physical connections and run lengths. I would expect you to have a network switch as well and other computers it sounds like.

2) You stated that if you disconnect your FreeNAS box that your network returns to life. Have you replaced the network cable between the FreeNAS and it's destination?

3) If possible, can you connect the FreeNAS machine to a computer directly and run it for a period of time, transferring tones of data back and forth? Check to see if it freezes.

4) If you simply disconnect the FreeNAS machine from the network, while all is fine, will the FreeNAS machine go cat?

5) If your MB NIC card is the issue, you could pop in a NIC add-on card and disable the MB NIC, see if that helps.

6) Memtest should be run longer than a few hours, run it a day or three. I use to say for 3 complete passes but I'm now aware it could take a few days for a memory error to pop up.

That's about all I can think of right now.

 

[DrKK]

Someone reported a similar issue, and we tracked it down to Plex doing something odd. Just a data point.

 

[thatguuuy]

Thanks for the replies.

It's been 24 hours without a freeze so far and I have been using PLEX and SABnzbd, but have left Couchpotato and Sickbeard disabled. I think I'm going to take a more lax method to discovering this while taking a few points and suggestions into mind.

I'm going to try some more heavy transfers and see if I can get it to happen again.... but so far, with 20-30gb of transfers via AFP (time machine) backup and multiple movies played via PLEX, nothing has happened yet. What are the chances that 9.2.1.6RC has helped?

If it happens again, PLEX will be disabled first considering several people have mentioned that very could well be the culprit.

To answer previous questions, I'm running an unmanaged 8 port gigabit switch shared among an Xbox, Denon HTPC, and a PS4 and that's run to the office to the router. IP addresses of 50-75 are for FreeNAS, 75-99 are for reserved devices (printer, Raspberry Pi, etc), and 100+ are for DHCP addresses. All runs between devices and switch are <6ft and switch to router is about 20 ft. As far as the rest of the questions, I'm going to see what happens from here, with the exception of memtest, which I'll run all day while I'm at work tomorrow. I'm going to swap the network cable regardless of whether or not it's needed since they're so cheap.

I'll update if/when anything else arises. If anything, hopefully this thread could be useful to others that might come across this issue. Thanks for the help.

 

[cyberjock]

The chances that 9.2.1.6RC has helped.. pretty unlikely IMO unless you are using hardware you shouldn't (like a Realtek NIC).

 

[Hyperion]

Id check all cables, move them about, get new ones.
Don’t Buy from China.

 

[solarisguy]

Overheating on the CPU would should down the system (on most motherboards). CPU fault would more often crash the OS, than freeze it.

RAM fault would be caught or at least detected by ECC. Chances of having multiple undetectable ECC RAM failures are very, very small.

After checking for loose connections, you may want to try a better power supply (regardless of what you have now) and consider a motherboard replacement.

 

[cyberjock]

RAM fault would be caught or at least detected by ECC. Chances of having multiple undetectable ECC RAM failures are very, very small.

Yeah, if you had a RAM error every second since the universe was created you still wouldn't have a 50% chance of having an undetectable error yet. :P