Large uploads to 9.3 gradually slow, stall, then fail

[JohnDigital]

Please forgive me if I am completely helpless here. I have no issues admitting that I know just a bit more than a complete noob. I have searched and read for several days but not found my exact problem. This might be only the second time I have posted to a forum, ever. I installed 9.3 to a Dell PowerEdge 1900 Server which previously ran 8.3 v2 without issues. I would just go back to it, but I cant now that I updated my ZFS Volume.

Hardware:
Dell PowerEdge 1900 Server (Beast)
Dell Perc/5i Raid
8x2TB WD Drives RAIDZ1 (All drives are configured as RAID0 on the PERC, then RAIDz1 in FreeNAS)
Dual Xeon 2.3Ghz
16GB RAM (Server class ECC I believe)
Broadcom Dual GB Network
SanDisk 8GB Flash

The exact issue as best I can explain it is when I go to upload files from Windows 7 the write starts fast, like normal, then stalls out. I have tried both CIFS and FTP with the same result. Reading files is apparently not affected. The transfer starts at 100MB/s then after it transfers for about a minute or two it stalls out to 20MB/s or so, but the transfer has stopped, and eventually I get an error message from Windows saying that the FreeNAS box is no longer found and would I like to try again, skip, or cancel. I can upload smaller files without issue. Its just the big video files that hang it up. After this happens I can also not access the web GUI nor browse the shares. After a few minutes the machine starts responding again. The same thing basically happens with FTP, only the transfer rate actually gets to zero. Normally the system is headless in my basement, but when I connect a monitor to the FreeNAS machine it has some suspicious info on the screen. I don't know what it means, and Google was not much help. I took photos of everything, but am unable to post. (This server is saying the photos and debug-log.TGZ file are too large to post??)

I tried booting up a Linux Mint Live USB and transferring from that with no luck. I also tried all the little tweaks and suggestions I did find, like a different NIC, different cables, large frame, and several others, with no help. Everything used to work perfectly on the 8.3 version. I wish I hadn't upgraded now.

I am not the greatest at the command line, but I can usually manage when I need to. If there is anyone out there that can help me, I can post whatever logs, photos or whatever you require to troubleshoot. Tell me what to run in the SSH terminal and ill get it to you. I also have a smoking 150Mbit connection and TeamViewer if you prefer.

I have a different machine I used to use for backing up when I needed to, so I installed 8.3 on it, and now its cracked out too. I keep getting an error that says "getty repeating too quickly on port ttyv0, pause 30 secs" or something to that effect, so I am stuck here. If anyone can help with this, id be in your debt.

I cant lose the data on the RAIDz volume or Id just blow it all away and start over.

I have no problem buying you a beer or nine for helping me. Thank you kindly.

 

[cyberjock]

That kind of sounds like a problem where a failing disk brings the whole pool down. Once you've shoved enough data into the ZFS write cache it won't fit any more until it is emptied. If you have a failing disk it could take a very long time to empty. Small files usually transfer fine unless you transfer multi-GB of small files because you actually have to fill the write cache.

The problem: You've done exactly what we've warned people not to do. You put ZFS on hardware RAID. So now ZFS can't properly report disk errors and you can't even run SMART tests to confirm if I'm right.

I can't tell you how many people I have talked to on the phone in the last month that have lost their pools and data permanently because they did exactly what you did on the PERC. The bottom line is PERC is a disaster waiting to happen and they shouldn't be used. If you can't do true JBOD you shouldn't be using FreeNAS. I'd say the number is at 7 or 8.

So now you've got a dilemma to deal with, and I don't know how to fix it. You really need to get proper hardware and move your data over. But in your present condition I don't know if I would recommend anything except to make a solid backup before going any further.

 

[JohnDigital]

I hear you loud and clear. When I put this thing together several years ago, I threw it together on a whim. No research, no foresight, and it worked like a dream. Now from reading the past few days, it does indeed appear that the way I ended up isn't ideal, boneheaded in fact. Ill get to backing up what I can and order the SAS card you recommend. Thank you for your time.