Bizzare network dropouts... no logs

[Rainwulf]

New to freenas, but i have installed it on a dual cpu intel board, 32 gig ECC ram, the works. Its all high end server gear. I have 8x8tb drives.

I keep getting these weird drop outs when copying data to the machine :

http://i.imgur.com/ztWVNOc.jpg

When that drop out occurs, i cant reach the freenas box from anything at all, but as you can see, 20-30 seconds later it comes back, i can retry the copy and everything keeps going on as if nothing had happened.
Sometimes it will work for 6 hours, other times, 6 minutes.

Any ideas? There is nothing coming up on the console apart from samba messages about forcing browser elections. It has to be a major networking issue as the web console drops out too.

EDIT: new message this time!
http://i.imgur.com/gl5iwee.jpg

Broken pipe. Definitely points to a major network fault. I am not familiar enough with freebsd though to make any calls though.

 

[anodos]

New to freenas, but i have installed it on a dual cpu intel board, 32 gig ECC ram, the works. Its all high end server gear. I have 8x8tb drives.

I keep getting these weird drop outs when copying data to the machine :

http://i.imgur.com/ztWVNOc.jpg

When that drop out occurs, i cant reach the freenas box from anything at all, but as you can see, 20-30 seconds later it comes back, i can retry the copy and everything keeps going on as if nothing had happened.
Sometimes it will work for 6 hours, other times, 6 minutes.

Any ideas? There is nothing coming up on the console apart from samba messages about forcing browser elections. It has to be a major networking issue as the web console drops out too.

EDIT: new message this time!
http://i.imgur.com/gl5iwee.jpg

Broken pipe. Definitely points to a major network fault. I am not familiar enough with freebsd though to make any calls though.

I'd start with layer 1 and work my way up. :D

 

[Rainwulf]

Well i replaced the network cable, and now between the new freenas box and my old windows server there is just 1 switch. The dropouts are still occuring, but im not losing web gui anymore. Its just CIFS/file transfer thats slowing right down.
my next test will be to plug the two machines in side by side with a single cable between them.
I really want to get this to work, as these archive drives dont work in normal raid configurations, ZFS is pretty well the only thing that allows you to run the drives like this. I realllly want ZFS to work!
I have all this disk space and only ZFS will allow me to use it with any kind of normalcy.

 

[DrKK]

There are various "gotchas" with the 8TB drives I thought.

Undoubtedly, there's some kind of timeout, or setting, you're going to have to liberalize. I don't know enough about it to suggest anything though. @cyberjock might know...?

 

[Rainwulf]

Damn, i didnt think about that. That wouldnt explain "broken pipe" though. Or why the webgui dies.

I wonder if its because of my dell HBA.

the weird network issues are really confusing me. It drops down to 63 meg a second nearly every time, then either speeds back up to 110, or slowly drops like that screen shot i posted.

 

[DrKK]

Oh yeah, Dell HBA. :(

Definitely could be the problem.

I am sure you'll find some forum posts.

 

[cyberjock]

There is very little information on the 8TB drives. Can you provide a link to the model?

If it is the model I think it is, would you be willing to do a Teamviewer session with me? There's some unanswered questions about 8TB that the community would like answered (as well as yourself since you are having a problem).

 

[Rainwulf]

Looks like im one on the bleeding edge here haha.
Quite happy to do a teamviewer session. my email address is <removed by Cyberjock>
I can setup teamviewer on the machine thats copying the data TO the freenas box, and you can have a look around.
BTW: System specs so far:
S5500BC
2 x Xeon 5620s 2.4 gig quad cores
32 gig ECC Ram.
Currently a Dell Perc H200 flashed to Dell 6gpbs IT mode firmware
8 x 8tb Seagate Archive Hard drives, model ST8000AS0002

 

[Darren Myers]

Ouch those archive drives :( , i think theres a few threads on them being terrible\

EDIT: see here https://forums.freenas.org/index.php?threads/seagate-8tb-archive-drive-in-freenas.27740/

 

[Robert Smith]

I have seen similar dropouts from a USB3 network adapter.

 

[Rainwulf]

Yea, as normal hard drives they are. ZFS is pretty well the only file system that runs ok on them, and even seagate has made noises in that regard.

Robert, i have seen dodgy network stuff too. However in this case its intel to intel over a decent netgear gigabit switch. The two machines are both intel xeons, its all fairly high end gear as i didnt want to screw around with dodgy stuff. All ECC, the source server is 2008R2 with perc 6s, the freenas box is a high end dual socket intel.

 

[cyberjock]

Just did a remote session with Rainwulf.

Benchmarks showed write performance as pretty good, averaging about 75MB/sec per disk. It was a little bursty, you'd get 180MB/sec for a second or two, then a second or two of 200KB/sec, then more 180MB/sec. That averages out to about 75MB/sec per disk. That's obviously fast enough that you could saturate 1Gb (or even LACP'd 1Gb) easily.

The read performance was a totally different story though. It averaged something like 8MB/sec. Typical speeds were 8MB/sec, with sudden bursts to 60MB/sec for a second or two before going back to 5MB/sec.

He is running P7 firmware in IR mode on his Dell controller, so that may be a major factor. I asked him to contact me if he gets P16 flashed on it so I can redo the tests. I tend to think that the odd results are because of the controller firmware, but time will tell.

Overall I'd call the 8TB Seagates 'questionable' as the speeds I saw for writes are obviously acceptable but the read behavior was very peculiar. After further testing I'll be able to tell if these really are acceptable for home use or not. I definitely would never recommend them for production use except as archive storage.

 

[Rainwulf]

Cyber, P16 flashing successful!!
Machine boots up a bit quicker as well, so whatever was goin on with that older firmware, it was screwing with something.

http://imgur.com/z3HCLvf
What a difference a bit of code on a HBA can make.
And no more alerts in FreeNAS.

 

[cyberjock]

Sa-weet! If you have time tonight send me a PM. I'd like to rerun my tests and see what the numbers changed to.

 

[titan_rw]

I'd also like to see the difference in benchmarks with the right IT firmware vs Dell's IR.

 

[Rainwulf]

Still network alerts.. broken pipe.
"Could not write response: 3051:gid to sid) to client, broken pipe.
I dont think thats the freenas box though.

The windows server im using to copy from has an intel ALB team on it, so i think that may be the issue.
Thats my next debug option, disabling the team.

 

[Rainwulf]

ok, after Cyberjock had a remote session on my server, and clue-batted me in on a couple of things:
Removed the intel teaming on the server SENDING the data.. it was causing arp flapping on the freenas box and just generally being a dink.
Changed to RAIDZ2.
Explained that adding a drive to the zpool doesnt actually do what i think it does and saved me a potential future disaster.
Discovered a potential read slowdown issue that at the moment wont be resolved until i get the M1015, as im running on basically a dell with a 9211IT firmware flash.
I have also fixed my sharing so that im not going against best practices.

Data transfer is now solid at 99.75 percent utilization of the outgoing gigabit nic on the windows server.
Proper CIFS sharing has also made my network run a bit better as well, in that im sharing properly to my windows machine.
I will call this network issue solved once i can run the file transfer with no "broken pipe" errors.