Another CIFS Performance Issue Thread - odd transfer timeout and slow write speed

Status
Not open for further replies.

Shepard

Cadet
Joined
Nov 14, 2015
Messages
6
Hello all, I've been reading up on the posts and help offered here on the forum and I've already learned a lot about FreeNAS compared to a month and a half ago when I decided to make the move over (still definitely a noob to FreeNAS and FreeBSD though). I've read up on recommended CIFS config in the FreeNAS documentation and on previous posts as to issues with CIFS share write performance issues, but I've run out of ideas (not that I had many to begin with) and I've tried as many solutions as I could find from previous posts.

Basically, I'm getting timeout issues when trying to transfer large junks of data - around 1TB - FreeNAS will write about 5-10GB of data and then timeout for about 30 seconds before transferring another 5-10GB. I keep seeing the following message in the console message area of the WebGUI: STATUS=daemon 'smbd' finished starting up and ready to serve connectionsfany (ipv4:192.168.1.104:62481) closed connection to service Media

The timeout happens even in the middle of files and sometimes causes the file to be skipped. Additionally, even when the transfer is actually moving data, I'm only seeing write speeds of about 35-40 MB/s.

My FreeNAS setup is as follows:
OS Version: FreeNAS-9.3-STABLE-201511020249
Chasis: SuperMicro SSG-6048R-E1CR36N
Motherboard: SuperMicro Super X10DRi-T4+
CPUs: 2 Intel Xeon E5-2680v3
RAM: 128GB Kingston ECC DDR4 2133 (8 16GB DIMMS)
NICS: Onboard Quad Intel 10GBase-T X540 (Static Assigned IPs)
HBA: SuperMicro LSI 2308 HBA (IT firmware version 19, driver version 20 in FreeNAS)
OS HDDs: Mirrored Samsung 850 500GB SSDs
ZFS HDDs: 12x Hitachi 8TB Ultrastar SATA Drives in RAIDZ2

The server sending data to the FreeNAS box is a node from a Dell C6100 with 2 Intel Xeon L5520 CPUs, 72GB ECC DDR3 RAM and Dual 1GB Intel NICs running Windows 8.1. I've consistently written to the Dell at a little less than 200MB/s in the past and read speeds are even faster, so I don't think the Dell is the bottleneck. The two machines are connected through an HP 2920-48G switch without LACP setup on the particular ports. However, the switch has no issue transferring at speeds much higher than 40MB/s.

After reading numerous other posts about write speed issues with CIFS shares, I've disabled Hyperthreading in the BIOS settings and have added the following Auxiliary Parameters to my CIFS service config:
ea support = no
store dos attributes = no
map archive = no
map hidden = no
map readonly = no
map system = no

I also have local master disabled and I've enabled UNIX extensions, with the server maximum protocol set for SMB3_00. Disabling Hyperthreading has seemed to help a bit with the write speed, but not much (maybe a 5-10MB/s increase - I was getting transfer speeds below 30MB/s before), but I'm still getting the same timeout issue. I do have encryption enabled on my pool along with AES-NI enabled on the CPUs in the BIOS.

Another odd, but maybe unrelated, issue is that when I disabled Hyperthreading a couple days ago I also enabled x2APIC in the CPU BIOS settings which subsequently kept causing a "kernel trap 12 with interrupts disabled" error at FreeNAS startup. As soon as I disabled x2APIC, FreeNAS booted up normally - I tested my CPU and RAM and both came back normal.

Any guidance would be greatly appreciated and thanks for all the information that is generally available through this community - it's already proven to be an invaluable resource.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Those are the Hitachi heliums, not a SMR drive, correct? (just a clarifying question)

You're also aware that running mismatched HBA firmware/driver versions *can* be a bad thing?

I'd be a little suspicious that there's some sort of problem with the X540's. Those cards haven't been well-supported in FreeNAS (feel free to search) and so I'd suggest you might want to run some iperf tests to identify if maybe there's a problem with the networking.
 

Shepard

Cadet
Joined
Nov 14, 2015
Messages
6
Those are the Hitachi heliums, not a SMR drive, correct? (just a clarifying question)

You're also aware that running mismatched HBA firmware/driver versions *can* be a bad thing?

I'd be a little suspicious that there's some sort of problem with the X540's. Those cards haven't been well-supported in FreeNAS (feel free to search) and so I'd suggest you might want to run some iperf tests to identify if maybe there's a problem with the networking.

Yes, the drives are Helium filled, not SMR drives. I've read other posts about the firmware/driver mismatch issue on the HBA - the chasis actually came with a SuperMicro LSI 3108 RAID card and I was reading here in the forums that , first off, RAID cards are not to be used and, on top of that, the LSI 3108 and 3008 are not very well supported, driver wise, yet in FreeNAS. It seems like the LSI 2308 was the recommended card to go with here despite the driver mismatch issue (there were a couple posts I found talking about the instability in the version 20 firmware for the 2308 and that going with the mismatch was a slightly safer risk than upgrading to the unstable firmware).

Thanks Jgreco, I'll run some iperf tests and post the results as soon as I get back in front of the machine. Any chance the X2APIC causing a trap 12 issue is related too? I'm guessing not, but just hoping to kill two birds with one stone...

[EDIT]: I should also mention I've got autotune enabled as well.
 
Last edited:

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
I'm guessing that perhaps the X2APIC stuff isn't supported yet. Too lazy to check. :smile:
 

SweetAndLow

Sweet'NASty
Joined
Nov 6, 2013
Messages
6,421
Upgrade the firmware on your card to v20. Having mismatched versions isn't recommended. Not sure where you read that the v20 firmware was unstable because that's not true.

I also suggest running some iperf tests the rule out the network.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Upgrade the firmware on your card to v20. Having mismatched versions isn't recommended. Not sure where you read that the v20 firmware was unstable because that's not true.

I also suggest running some iperf tests the rule out the network.

I guess there've been some problems with firmware subrevs. "Blech."
 

Shepard

Cadet
Joined
Nov 14, 2015
Messages
6
Okay, it definitely appears to be a NIC issue. I tried to run the iperf test between the FreeNAS box and the Dell server (I set the FreeNAS box to iperf -s -w 128k and set the Dell to iperf -c donbot -p 5100 -f m -w 128k) and couldn't even complete the iperf test. I let it run for about 90 minutes before I stopped it and it came back with the following:

[4] 0.0-5489.6 sec 14.0 Bytes 0.02 bits/sec

I did some more searching around the forums and I did find comments that version 20.0.0.4 of the HBA firmware is pretty stable so I went ahead and resolved the mismatch issue and I found threads discussing the 10Gb NICs in FreeNAS and it appears as though the Intel X540 is supported, but needs to have some autotune/modifications which I had done previously. Specifically, I've got the following tunables set:

FreeNAS Tunables 11-15-15.jpg


I'm going to keep trying different iperf tests to see if any of them come back better than the iperfs I've run so far.

Jgreco, good enough for me on the X2APIC (that's what I was hoping it was, I just jumped to worst case when I saw the error though and thought I had defective CPUs or something).
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
I let it run for about 90 minutes before I stopped it and it came back with the following:

[4] 0.0-5489.6 sec 14.0 Bytes 0.02 bits/sec

I did some more searching around the forums and I did find comments that version 20.0.0.4 of the HBA firmware is pretty stable so I went ahead and resolved the mismatch issue and I found threads discussing the 10Gb NICs in FreeNAS and it appears as though the Intel X540 is supported,

Be careful, there's supported and then there's also """supported""" (airquotes).

In theory there's a whole bunch of hardware supported by FreeNAS, which it inherits from FreeBSD. That includes crap like Realtek ethernets, Marvell SATA controllers, all the various RAID controllers, systems lacking ECC RAM, etc.

For something like a web server, it might not matter that much.

For NAS, however, the trick is that what you really want is to pick the hardware that's known to work in a problem-free manner with the NASware. What that means is that we have evidence that people are using it in production on a large scale and that it has maybe a BILLION hours of problem-free service under its belt when you look at the whole userbase. For example, we know that the Intel em-based cards (Intel 1000 CT and friends) fall into that category, and that the 9211-8i HBA does, and venerable boards like the X9SCL/SCM and the X10SC7 do. That means you can drop that stuff into your FreeNAS and you are exceedingly likely to get the same problem-free experience as many thousands of other users. Because what you don't want are mystery problems cropping up one day when your switch reboots and the ethernet loses connectivity for a moment and a poorly written device driver starts queueing traffic and then runs the system out of mbufs or something similarly idiotic.

So for the X540, I guess one of the things you can try is to turn off TSO. Might want to read this

https://bugs.freenas.org/issues/4560

and then see if you still feel you want to use the X540. I've found the X520 to be stable but needed some tweaks to make it so. The go-to for 10G in FreeNAS appears to be Chelsio.
 

Shepard

Cadet
Joined
Nov 14, 2015
Messages
6
Be careful, there's supported and then there's also """supported""" (airquotes).

In theory there's a whole bunch of hardware supported by FreeNAS, which it inherits from FreeBSD. That includes crap like Realtek ethernets, Marvell SATA controllers, all the various RAID controllers, systems lacking ECC RAM, etc.

For something like a web server, it might not matter that much.

For NAS, however, the trick is that what you really want is to pick the hardware that's known to work in a problem-free manner with the NASware. What that means is that we have evidence that people are using it in production on a large scale and that it has maybe a BILLION hours of problem-free service under its belt when you look at the whole userbase. For example, we know that the Intel em-based cards (Intel 1000 CT and friends) fall into that category, and that the 9211-8i HBA does, and venerable boards like the X9SCL/SCM and the X10SC7 do. That means you can drop that stuff into your FreeNAS and you are exceedingly likely to get the same problem-free experience as many thousands of other users. Because what you don't want are mystery problems cropping up one day when your switch reboots and the ethernet loses connectivity for a moment and a poorly written device driver starts queueing traffic and then runs the system out of mbufs or something similarly idiotic.

So for the X540, I guess one of the things you can try is to turn off TSO. Might want to read this

https://bugs.freenas.org/issues/4560

and then see if you still feel you want to use the X540. I've found the X520 to be stable but needed some tweaks to make it so. The go-to for 10G in FreeNAS appears to be Chelsio.

Got it, I turned off TSO on each interface and it did seem to help as I'm now able to transfer at about 107MB/s (makes sense since the Dell is only connected on a single GbE connection right now), but that was only transferring a single 2GB file. I'll do some more testing and if the X540 starts acting up again, I'll look into the Chelsio cards or the X520. Thanks for all the help!
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
The Chelsio stuff's probably cheaper than the X520, and is reported to work by default (even the X520 may need tweaking). I finally got around to ordering a T420-CR recently mainly because it was cheaper and we needed some more X520's for hypervisors, so I can experiment with that to see how it goes.
 

Shepard

Cadet
Joined
Nov 14, 2015
Messages
6
The Chelsio stuff's probably cheaper than the X520, and is reported to work by default (even the X520 may need tweaking). I finally got around to ordering a T420-CR recently mainly because it was cheaper and we needed some more X520's for hypervisors, so I can experiment with that to see how it goes.

Great,would love to hear how the T420-CR works out for you. Thanks Jgreco!
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Yeah, I'm sure it'll be just fine. I'm not unhappy with the X520 or anything, but it'll be interesting to see if the Chelsio is any sort of noticeable difference (guessing: not really).
 

Shepard

Cadet
Joined
Nov 14, 2015
Messages
6
Ok, so I'm a bit baffled now - I've switched to a Chelsio T520-SO-CR Card in the FreeNAS box and I've got a 10G fiber connection on the Dell server as well (Both connections run through the fiber ports on a Netgear XS728T 10GBaseT Switch), but I'm getting the exact same issues. Although now I will see the transfer rate get up to about 159MB/s for a minute or so before it falls back down to less than 500k/s and then just times out. I can't get iperf to run a test either, just like before.
 
Status
Not open for further replies.
Top