CIFS Write performance very poor

Status
Not open for further replies.

Midwan

Dabbler
Joined
Dec 16, 2013
Messages
15
Hi all,

I realize this must be a rather common topic, considering how many similar issues I've seen in the past.
However, I haven't managed to fix it yet, nor discover what causes it. The other threads I've read through didn't help in identifying the issue in my case.

If I am mistaken and this has been answered somewhere already, I humbly apologize and would be grateful if you could just give me the relevant link.

I have a home-built NAS running FreeNAS 9.10 stable (201606270534). The hardware it's running on is as follows:

MB: ASUS E45M1-I Deluxe
CPU: AMD E-450 (onboard)
NET: 1000MBit
RAM: 8GB DDR3

ZFS: 5 2TB disks on a RAIDZ1 configuration. No encryption, LZ4 compression enabled.

The installation has just a few services enabled:
- CIFS
- SSH

And a few plugins:
- Transmission
- CouchPotato

The NAS connects to a TP-Link 10/100/1000 switch, to which the other machines in the LAN connect to also. The tests I've made were also done on my ASUS RT-N66U ports (also 1000Mbit) just to make sure. The results were identical.

The problem: Although CIFS read speeds are satisfactory considering the relatively low-powered hardware (I get around 80MB/sec read speed over the network to another Win10 machine), CIFS write speed is horribly slow - only around 4-5MB/sec. CPU usage during writing is no more than 9%.

I tested the pool write speed locally on the NAS itself:

dd if=/dev/zero of=testfile bs=1048576
32731+0 records in
32731+0 records out
34320941056 bytes transferred in 63.073537 secs (544141690 bytes/sec)

CPU usage during the test was at 100%.

I tried writing over SSH, from the same Win10 computer to the NAS, the speed was around 12MB/sec.

Since it doesn't seem to be limited to hardware, so I'm assuming it's related to the protocol/configuration. However, after trying several combinations and seeing no difference, I cannot understand what exactly could cause such a big difference in read/write speeds.

Here's the SMB4.conf:

cat /usr/local/etc/smb4.conf
[global]
username map = /usr/local/etc/smbusers
server max protocol = SMB3
encrypt passwords = yes
dns proxy = no
strict locking = no
oplocks = yes
deadtime = 15
max log size = 51200
max open files = 233543
logging = file
load printers = no
printing = bsd
printcap name = /dev/null
disable spoolss = yes
getwd cache = yes
guest account = nobody
map to guest = Bad User
obey pam restrictions = no
directory name cache size = 0
kernel change notify = no
panic action = /usr/local/libexec/samba/samba-backtrace
nsupdate command = /usr/local/bin/samba-nsupdate -g
server string = FreeNAS Server
ea support = yes
store dos attributes = yes
lm announce = yes
unix extensions = no
acl allow execute always = false
dos filemode = yes
multicast dns register = yes
domain logons = no
local master = no
idmap config *: backend = tdb
idmap config *: range = 90000001-100000000
server role = standalone
netbios name = KOULA
workgroup = MIDLAIR
security = user
pid directory = /var/run/samba
create mask = 0777
directory mask = 0777
client ntlmv2 auth = yes
dos charset = CP437
unix charset = UTF-8
log level = 0


[Backups]
path = /mnt/KoulaOne/Backups
printable = no
veto files = /.snapshot/.windows/.mac/.zfs/
writeable = yes
browseable = yes
shadow:snapdir = .zfs/snapshot
shadow:sort = desc
shadow:localtime = yes
shadow:format = auto-%Y%m%d.%H%M-2w
shadow:snapdirseverywhere = yes
vfs objects = shadow_copy2 zfs_space zfsacl aio_pthread streams_xattr
hide dot files = yes
guest ok = no
nfs4:mode = special
nfs4:acedup = merge
nfs4:chown = true
zfsacl:acesort = dontcare


[Movies]
path = /mnt/KoulaOne/Movies
printable = no
veto files = /.snapshot/.windows/.mac/.zfs/
writeable = yes
browseable = yes
vfs objects = zfs_space zfsacl aio_pthread streams_xattr
hide dot files = yes
guest ok = no
nfs4:mode = special
nfs4:acedup = merge
nfs4:chown = true
zfsacl:acesort = dontcare


[Projects]
path = /mnt/KoulaOne/projects
comment = Amiga Lightwave Projects
printable = no
veto files = /.snapshot/.windows/.mac/.zfs/
writeable = yes
browseable = yes
vfs objects = zfs_space zfsacl aio_pthread streams_xattr
hide dot files = yes
guest ok = no
nfs4:mode = special
nfs4:acedup = merge
nfs4:chown = true
zfsacl:acesort = dontcare


[TVShows]
path = /mnt/KoulaOne/TVShows
printable = no
veto files = /.snapshot/.windows/.mac/.zfs/
writeable = yes
browseable = yes
vfs objects = zfs_space zfsacl aio_pthread streams_xattr
hide dot files = yes
guest ok = no
nfs4:mode = special
nfs4:acedup = merge
nfs4:chown = true
zfsacl:acesort = dontcare


[incoming]
path = /mnt/KoulaOne/incoming
printable = no
veto files = /.snapshot/.windows/.mac/.zfs/
writeable = yes
browseable = yes
vfs objects = zfs_space zfsacl aio_pthread streams_xattr
hide dot files = yes
guest ok = no
nfs4:mode = special
nfs4:acedup = merge
nfs4:chown = true
zfsacl:acesort = dontcare

I'd be happy to provide any more info if you'd like.
 

Nick2253

Wizard
Joined
Apr 21, 2014
Messages
1,633
Part of your problem could be the Realtek NIC on your motherboard. I would get a dedicated Intel NIC and try with that.

Though it's less likely, it could also be a problem with your source. If your source isn't capable of moving data at that speed, that would be an obvious bottleneck. If you have multiple computers, try moving data back and forth between them to confirm that your source isn't the problem.
 

Midwan

Dabbler
Joined
Dec 16, 2013
Messages
15
Thanks for the reply. Both are fair observations.

The source is a 4.2 GHz i7 with Intel 1000MBit NICs, so I doubt that's the issue. I've also tested the same from another computer however, having similar results (fast reads, very slow writes).

I'll see if I can track down a spare NIC to test on the NAS side, just in case.
 

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
9,543
Thanks for the reply. Both are fair observations.

The source is a 4.2 GHz i7 with Intel 1000MBit NICs, so I doubt that's the issue. I've also tested the same from another computer however, having similar results (fast reads, very slow writes).

I'll see if I can track down a spare NIC to test on the NAS side, just in case.
Are you writing lots of small files are writing a single large file?
 

Midwan

Dabbler
Joined
Dec 16, 2013
Messages
15
Sorry I didn't mention that.
The tests were done with relatively large files - A few linux distro ISO images, ranging from 1.4-1.7GB each.
The same was done for the reading tests as well, where I got the 80MB/s speeds.

Also worth noting again, over SSH the same write test managed more than twice the speed (12MB/s) than with CIFS... Not blindingly fast, but still. I would expect CIFS to be at least as fast, if not faster (since it doesn't have the SSH encryption overhead).
 

SweetAndLow

Sweet'NASty
Joined
Nov 6, 2013
Messages
6,421
12MB/s means your network connection isn't running at 1Gib it's running at 100mib. Get a better nic and test with iperf to eliminate the network bottleneck.
 

Midwan

Dabbler
Joined
Dec 16, 2013
Messages
15
I would agree with you, if 12MB/s was all I got from it.
As I mentioned above though, I can get 80MB/s on reads, which is not possible on a 100Mbit network... ;-)

12MB/s is the speed I got during writes over SSH from a Windows client to the NAS. Over CIFS, I get around 4MB/s for the same operation.

On the Windows client, I have an Intel 82574L Gigabit NIC, which is connected at 1G to the Gigabit switch.
 

SweetAndLow

Sweet'NASty
Joined
Nov 6, 2013
Messages
6,421
How do you know it's working at 1gib speeds in both directions?
 

Midwan

Dabbler
Joined
Dec 16, 2013
Messages
15
Iperf results from the Windows 10 machine as a server, with a Mac mini acting as a client node:

D:\midwa\Downloads\iperf-3.1.3-win64>iperf3.exe -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.1.88, port 49286
[ 5] local 192.168.1.46 port 5201 connected to 192.168.1.88 port 49287
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-1.00 sec 112 MBytes 937 Mbits/sec
[ 5] 1.00-2.00 sec 111 MBytes 929 Mbits/sec
[ 5] 2.00-3.00 sec 112 MBytes 939 Mbits/sec
[ 5] 3.00-4.00 sec 112 MBytes 940 Mbits/sec
[ 5] 4.00-5.00 sec 112 MBytes 940 Mbits/sec
[ 5] 5.00-6.00 sec 112 MBytes 940 Mbits/sec
[ 5] 6.00-7.00 sec 112 MBytes 940 Mbits/sec
[ 5] 7.00-8.00 sec 112 MBytes 940 Mbits/sec
[ 5] 8.00-9.00 sec 112 MBytes 940 Mbits/sec
[ 5] 9.00-10.00 sec 112 MBytes 940 Mbits/sec
[ 5] 10.00-10.00 sec 153 KBytes 839 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-10.00 sec 0.00 Bytes 0.00 bits/sec sender
[ 5] 0.00-10.00 sec 1.09 GBytes 939 Mbits/sec receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------

The opposite test (Mac mini acting as server, Win10 as client):

Mac-mini:Downloads midwan$ ./iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.1.46, port 50161
[ 5] local 192.168.1.88 port 5201 connected to 192.168.1.46 port 50162
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-1.00 sec 112 MBytes 937 Mbits/sec
[ 5] 1.00-2.00 sec 113 MBytes 948 Mbits/sec
[ 5] 2.00-3.00 sec 113 MBytes 948 Mbits/sec
[ 5] 3.00-4.00 sec 113 MBytes 947 Mbits/sec
[ 5] 4.00-5.00 sec 113 MBytes 948 Mbits/sec
[ 5] 5.00-6.00 sec 113 MBytes 948 Mbits/sec
[ 5] 6.00-7.00 sec 113 MBytes 946 Mbits/sec
[ 5] 7.00-8.00 sec 113 MBytes 948 Mbits/sec
[ 5] 8.00-9.00 sec 113 MBytes 947 Mbits/sec
[ 5] 9.00-10.00 sec 113 MBytes 947 Mbits/sec
[ 5] 10.00-10.01 sec 1.25 MBytes 933 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-10.01 sec 0.00 Bytes 0.00 bits/sec sender
[ 5] 0.00-10.01 sec 1.10 GBytes 946 Mbits/sec receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------


Testing NAS <-> Win10:

midwan@koula:~ % iperf -c 192.168.1.46
------------------------------------------------------------
Client connecting to 192.168.1.46, TCP port 5001
TCP window size: 129 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.1.18 port 12124 connected with 192.168.1.46 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 906 MBytes 759 Mbits/sec

And the opposite:

midwan@koula:~ % iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 128 KByte (default)
------------------------------------------------------------
[ 4] local 192.168.1.18 port 5001 connected with 192.168.1.46 port 50732
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.1 sec 295 MBytes 245 Mbits/sec
[ 5] local 192.168.1.18 port 5001 connected with 192.168.1.46 port 50792
[ 5] 0.0-10.0 sec 289 MBytes 242 Mbits/sec
[ 4] local 192.168.1.18 port 5001 connected with 192.168.1.46 port 50840
[ 4] 0.0-10.2 sec 390 MBytes 322 Mbits/sec


I noticed that when acting as a server, the NAS is significantly slower - though to be fair, I'd need to run another test a bit later, the NAS was being used serving files in the background.

However, even with these tests I see that there is a problem: I don't even reach the throughput mentioned above (245 Mbits/sec) over CIFS writes... :-/
 

SweetAndLow

Sweet'NASty
Joined
Nov 6, 2013
Messages
6,421
New nic will solve your problems
 

Midwan

Dabbler
Joined
Dec 16, 2013
Messages
15
As part of the tests I was doing, I decided to also do a fresh installation on a new USB stick, import the ZFS pool and re-do the CIFS config.

And guess what? Transfers are now 70MB/s on writes, 80MB/s on reads. So there was something in the older configuration that was causing this, though I haven't gone through all the items one by one to find out what exactly it was...

It was certainly NOT the NIC however in my case.
 

Donny Davis

Contributor
Joined
Jul 31, 2015
Messages
139
You should get a new nic anyways... An intel nic can be had for a very reasonable price, and will yield much better performance.
 

Midwan

Dabbler
Joined
Dec 16, 2013
Messages
15
Agreed.
I just wanted to post my solution in case someone else runs into something similar in the future.

In my case, I'm planning an upgrade for the mainboard anyway, since this one doesn't support ECC RAM and it's rather weak for running plugins on top of FreeNAS. When the time comes, a new NIC will be part of the package.
 
Status
Not open for further replies.
Top