Strange behaviour when copying from desktop PC to CIFS Share hosted by Freenas

[fcueto]

Hi,

My "regular" transfer speed to freenas is around 60MB/sec when copying large files.

I have a strange behaviour when copying or moving a lot of small files from my PC to a CIFS Share hosted by Freenas.

Yesterday, I moved my whole MAME directory (~16k files, size from 100kB to 100MB) to the samba share.

The transfer started at 60MB as usual. But every 10sec, the transfer stopped completely during 3 to 5 sec (this "freeze" could happen between 2 files, or even in the middle of a file). After these 5 sec the transfer resumed, and so on:
- 10 sec of tranfer at full speed
- 3 to 5 sec "stalled"'
- 10 sec of full speed
- 3 to 5 sec "stalled"'
- and so on...

What can be the reason ? I've read some posts about the ZFS write cache being written (I have 16GB of RAM), some other posts about CIFS AIO (the option is not present in the CIFS options of the latest freenas release), ....

I don't think this is a CPU limitation as the CPU was almost idle during the transfer.

Help me please.

 

[Whattteva]

do a ping test and keep it going for a while and see if you get any timeouts.

 

[fcueto]

Hi,

no network timeouts.

 

[anodos]

Post your hardware specs (including NIC details and how your pool is configured) as well as your network configuration (are you using wifi).
Upload your smb4.conf file
Do you see any errors in /var/log/samba4/log.smbd or /var/log/messages?

 

[fcueto]

My hardware is as follow:
- HP Microserver gen 8
- upgraded CPU Intel Xeon E3-1265L V2
- 16 GB of ECC UDIMM
- 4 x 4TB seagate NAS HDD ST4000VN000
- Single RAIDZ1 pool

I'm not using Wifi, only the onboard 1GB NIC (NIC is NC332i Broadcom BCM5720)

I'm really using freenas "out of the box". I have 5 datasets shared through CIFS.

My smb4.conf is as follow:
[global]
server max protocol = SMB2
encrypt passwords = yes
dns proxy = no
strict locking = no
oplocks = yes
deadtime = 15
max log size = 51200
max open files = 11070
load printers = no
printing = bsd
printcap name = /dev/null
disable spoolss = yes
getwd cache = yes
guest account = nobody
map to guest = Bad User
obey pam restrictions = Yes
directory name cache size = 0
kernel change notify = no
panic action = /usr/local/libexec/samba/samba-backtrace
server string = FreeNAS Server
ea support = yes
store dos attributes = yes
hostname lookups = yes
time server = yes
acl allow execute always = true
local master = yes
idmap config *:backend = tdb
idmap config *:range = 90000000-100000000
server role = standalone
netbios name = FREENAS
workgroup = WORKGROUP
security = user
pid directory = /var/run/samba
smb passwd file = /var/etc/private/smbpasswd
private dir = /var/etc/private
create mask = 0666
directory mask = 0777
client ntlmv2 auth = yes
dos charset = CP437
unix charset = UTF-8
log level = 1


[documents]
path = /mnt/volume1/documents
printable = no
veto files = /.snap/.windows/.zfs/
writeable = yes
browseable = yes
recycle:repository = .recycle/%U
recycle:keeptree = yes
recycle:versions = yes
recycle:touch = yes
recycle:directory_mode = 0777
recycle:subdir_mode = 0700
vfs objects = zfsacl streams_xattr aio_pthread
hide dot files = yes
guest ok = no
nfs4:mode = special
nfs4:acedup = merge
nfs4:chown = yes
zfsacl:acesort = dontcare


[downloads]
path = /mnt/volume1/downloads
printable = no
veto files = /.snap/.windows/.zfs/
writeable = yes
browseable = yes
recycle:repository = .recycle/%U
recycle:keeptree = yes
recycle:versions = yes
recycle:touch = yes
recycle:directory_mode = 0777
recycle:subdir_mode = 0700
vfs objects = zfsacl streams_xattr aio_pthread
hide dot files = yes
guest ok = no
nfs4:mode = special
nfs4:acedup = merge
nfs4:chown = yes
zfsacl:acesort = dontcare


[downloads2]
path = /mnt/volume1/downloads2
printable = no
veto files = /.snap/.windows/.zfs/
writeable = yes
browseable = no
recycle:repository = .recycle/%U
recycle:keeptree = yes
recycle:versions = yes
recycle:touch = yes
recycle:directory_mode = 0777
recycle:subdir_mode = 0700
vfs objects = zfsacl streams_xattr aio_pthread
hide dot files = yes
guest ok = no
nfs4:mode = special
nfs4:acedup = merge
nfs4:chown = yes
zfsacl:acesort = dontcare


[musique]
path = /mnt/volume1/musique
printable = no
veto files = /.snap/.windows/.zfs/
writeable = yes
browseable = yes
recycle:repository = .recycle/%U
recycle:keeptree = yes
recycle:versions = yes
recycle:touch = yes
recycle:directory_mode = 0777
recycle:subdir_mode = 0700
vfs objects = zfsacl streams_xattr aio_pthread
hide dot files = yes
guest ok = no
nfs4:mode = special
nfs4:acedup = merge
nfs4:chown = yes
zfsacl:acesort = dontcare


[photos]
path = /mnt/volume1/photos
printable = no
veto files = /.snap/.windows/.zfs/
writeable = yes
browseable = yes
recycle:repository = .recycle/%U
recycle:keeptree = yes
recycle:versions = yes
recycle:touch = yes
recycle:directory_mode = 0777
recycle:subdir_mode = 0700
vfs objects = zfsacl streams_xattr aio_pthread
hide dot files = yes
guest ok = no
nfs4:mode = special
nfs4:acedup = merge
nfs4:chown = yes
zfsacl:acesort = dontcare

I have no errors in /var/log/samba4/log.smbd or /var/log/messages

 

[fcueto]

I'm seing something close to http://forums.freenas.org/index.php?threads/zfs-samba-cifs-abysmal-performance-in-freenas-8-2.6738/

except I don't have the problem with large files

 

[SwampRabbit]

I am also having strange behavior with CIFS.
http://forums.freenas.org/index.php...rformance-issue-with-build.22564/#post-136136

My CPU is a little weaker than yours, but twice as much RAM, but we are using the same HDDs.

I think I can see it staling with large files too, but haven't proven it.
I still can't get much over 50MB/sec.

What version of FreeNAS are you running?

Did you run dd and iperf tests yet?

If you look at my thread, I've been troubleshooting for awhile, maybe it could help.

Edit: Thanks for sharing that link, I didn't see if before, but it looks good and may help me.

 

[fcueto]

I'm using the latest release of freenas, v9.2.1.7

I performed some dd tests here:
http://forums.freenas.org/index.php?threads/raidz-reads-are-slower-than-writes.22711/

I have very good results with ZFS:
Write : ~249MB/sec
Read : ~279MB/sec

I did iperf tests too, I'm close to the gigabit cap:
[root@freenas] ~# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[ 4] local 192.168.0.51 port 5001 connected with 192.168.0.2 port 53669
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 1.10 GBytes 941 Mbits/sec
[ 5] local 192.168.0.51 port 5001 connected with 192.168.0.2 port 53670
[ 5] 0.0-10.0 sec 1.10 GBytes 941 Mbits/sec

so I'm pretty sure it's not a network issue.

My issue seems to come from CIFS.

 

[SwampRabbit]

I getting similar dd and iperf tests too.
I have Intel NICs on the motherboard, but the Realtek and the Intel NIC I tried on the client give about the same result.

I am using RAIDZ2, instead.

I suspect it is CIFS too, maybe just the version, or a Samba configuration change.
I haven't seen many people complain about this, not sure if anyone that upgrades or that does a fresh install, but uses their old FreeNAS config file would be affected.

 

[anodos]

Have you tried disabling antivirus / security software on client pc?

 

[fcueto]

Antivirus disabled.

on a "big" file (20GB):
- copy from freenas to PC = average 80MB/sec (which is OK for me)
- copy the same file from PC to freenas = average 40MB/sec (the file was partially in the cache on the PC side, I have 16GB ram)

 

[Whattteva]

You have any dedup, encryption, compression, etc.. enabled?

 

[fcueto]

No dedup but lz4 encryption is enabled.

But I checked the CPU during transfer and it's not using more than 20%

 

[anodos]

Is the 'bursty' write-speed consistent and reproducible? Did the bursty writes disappear when A/V was disabled? Is it cross-protocol (do you experience it when writing via SFTP)? Does the behavior persist if you use mount_smbfs to locally mount your share on your freenas box and copy some directories around?

Code:

mkdir /mnt/foo
mount_smbfs 192.168.0.42 //username@HostName/share /mnt/foo

Many factors can effect your read/write performance. You may be hitting an IOPS barrier on some copy jobs. You may have inferior networking hardware. You may have a failing hard drive. The key to resolving issues is lots of testing. That being said, the default settings for CIFS are good you should start with testing for other problems first (some people really end up mucking up their smb4.conf file when troubleshooting).

By the way, 4TB drives plus RAIDZ1 is a bad idea.

 

[Whattteva]

Just to clarify, your "PC" client is an actual baremetal and not some sort of VM?

 

[fcueto]

My "PC" client is a non virtualized core-i7 2600k with 16GB of DDR3 running Windows 7 x64

I don't think I have disks issues or network issue because as I posted before:

dd test gives me:
Write : ~249MB/sec
Read : ~279MB/sec

iperf test gives me 941 Mbits/sec

Why is 4TB drives plus RAIDZ1 a bad idea ? Is it because "RAID5 is dead in 2009" ?

 

[anodos]

My "PC" client is a non virtualized core-i7 2600k with 16GB of DDR3 running Windows 7 x64

I don't think I have disks issues or network issue because as I posted before:

dd test gives me:
Write : ~249MB/sec
Read : ~279MB/sec

iperf test gives me 941 Mbits/sec

Why is 4TB drives plus RAIDZ1 a bad idea ? Is it because "RAID5 is dead in 2009" ?

I would still select a suitably large set of real data from your client PC. Copy it with SFTP then copy with CIFS. Note any significant performance differences between the two. If there is a significant difference, begin troubleshooting why they are different. For instance depending on the operations you are performing, file size, etc, you may be hitting the IOPS limit of your pool (which wouldn't be shown in your dd and iperf tests).

Regarding RAIDZ1/RAID5 you can check out the discussions here:

http://community.spiceworks.com/top...-rebuilds-addressed-by-wd-red-series-nas-hd-s
http://community.spiceworks.com/topic/356486-why-is-raid5-so-bad

I ran several multi-terabyte RAID50 arrays back in the days when a 500GB SATA drive was considered large. It was fine, but the situation is totally different with large (2+TB) disks. The idea of sweating out a RAIDZ1 rebuild with 4TB disks is somewhat nauseating. RAIDZ1 would probably be fine if you had smaller disk sizes or were using SSDs. With 4 drives my default is RAID10 and not to mess with parity RAID.

 

[fcueto]

I'll try with SFTP.....

 

[gpsguy]

As a real world example, one of our members today mentioned that the resilvering process took 79.1 hours for a 3Tb disk in a 6 disk RAIDz2 pool. With RAIDz2 the user was protected against another drive failure, but if it were RAIDz1 I might be worried. Especially, since many FreeNAS users don't have any backups.

The idea of sweating out a RAIDZ1 rebuild with 4TB disks is somewhat nauseating.

 

[fcueto]

I can't see why the rebuild of a 3Tb disk could take so long.
Last week I had to rebuild a 1TB disk of a 4 disks RAID5 in a synology DS-408 (old hardware, small CPU) and it took less than 6 hours.

I guess than it would have taken less than 3x6=18 hours to rebuild a 3TB disk. Where does this 79.1 hours comes from ?? I know than RAID6 is less efficient than RAID5 but hey ! not that much ?!?