Very slow speed with CIFS share

[AMiGAmann]

I upgraded from 16GB to 32GB of memory and had to run some days of memtesting.

Finally I started up FreeNAS again. The result is totally disappointing: There is no difference, the maximum transfer speed is approx. 60MB/seconds.

:-(

 

[anodos]

I upgraded from 16GB to 32GB of memory and had to run some days of memtesting.

Finally I started up FreeNAS again. The result is totally disappointing: There is no difference, the maximum transfer speed is approx. 60MB/seconds.

:-(

Post a debug file. 'System' -> 'advanced' -> 'save debug'

 

[jde]

Have you tested with a client directly connected to the server (i.e. no switch)? You mentioned earlier that you tried testing by directly connecting two clients. Its unclear to me whether you did any of the server testing by directly connecting a client machine?

 

[hugovsky]

Your iperf tests are pretty disappointing. You need to get that sorted out though.

Have you managed to improve iperf tests? Have you trayed to use direct link with another computer? Remember that test system must be able to write at wire speed.

 

[hugovsky]

what jde said. I'm really slow today.

 

[AMiGAmann]

Have you tested with a client directly connected to the server (i.e. no switch)?

I did that test again. I connected two different (Windows) clients directly to the server (one after another) and copied a file from/to the local SSD of the client. The maximum transfer speed was around 60MB/s.

Have you managed to improve iperf tests?

I guess the bad iperf results were resulting from using a different iperf version on a different OS (Windows). When using the identical iperf version included in Ubuntu (booted from USB) the result was approx 117MB/s.

Post a debug file. 'System' -> 'advanced' -> 'save debug'

I think the complete debug file contains a lot of kind of sensitive information to be directly posted in a forum.

 

[anodos]

I did that test again. I connected two different (Windows) clients directly to the server (one after another) and copied a file from/to the local SSD of the client. The maximum transfer speed was around 60MB/s.

I guess the bad iperf results were resulting from using a different iperf version on a different OS (Windows). When using the identical iperf version included in Ubuntu (booted from USB) the result was approx 117MB/s.

I think the complete debug file contains a lot of kind of sensitive information to be directly posted in a forum.

As an additional data point, let's also see what sorts of raw speed you can get out of your samba server. In the cli of your freenas server

Code:

cd /mnt/pool
smbclient //<ip-address>/daten
get <large file>

You can PM me the debug file. I'll look at it this weekend if I get a chance.

 

[m0nkey_]

I have a link aggregation (loadbalance) configured in FreeNAS and have built a trunk in my HP 1810-24G switch for those two ports.

I'm surprised that nobody has considered looking at this yet. Can I suggest breaking the link aggregation and going with one NIC? Do you still get the same speed issue?

 

[anodos]

I'm surprised that nobody has considered looking at this yet. Can I suggest breaking the link aggregation and going with one NIC? Do you still get the same speed issue?

See post #3

 

[m0nkey_]

May bad! :) retracted!

 

[AMiGAmann]

As an additional data point, let's also see what sorts of raw speed you can get out of your samba server.

I tried to copy several files but after some time is always stops with "parallel_read returned NT_STATUS_IO_TIMEOUT".

 

[anodos]

I am reviewing the debug file right now. SMART information is somewhat concerning. I have attached below. Hard drives are WD Green that have not been wdidle-ed (load cycle counts above 200000. The server does not appear that smart monitoring has been configured. Pool consists of a single 5-disk RAIDZ1 vdev. Without looking much further, I'd say there's a good chance your issues may be hardware related. /dev/ada2 is probably failing, and I'd swap out the cable on /dev/ada4.

/dev/ada4 has the following:

Code:

199 UDMA_CRC_Error_Count    -O--CK   200   070   000    -    1576

/dev/ada2 has the following:

Code:

Error 5 [4] occurred at disk power-on lifetime: 2174 hours (90 days + 14 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 51 00 00 00 00 18 b7 da b0 40 00  Error: UNC at LBA = 0x18b7dab0 = 414702256

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  60 00 80 00 78 00 00 18 b6 05 08 40 08     06:45:37.124  READ FPDMA QUEUED
  61 00 40 00 70 00 00 22 19 ab a0 40 08     06:45:37.124  WRITE FPDMA QUEUED
  60 00 08 00 68 00 00 18 49 5a c8 40 08     06:45:37.124  READ FPDMA QUEUED
  60 00 80 00 60 00 00 18 b7 db 48 40 08     06:45:37.124  READ FPDMA QUEUED
  60 01 00 00 58 00 00 18 b7 da 48 40 08     06:45:37.123  READ FPDMA QUEUED

Error 4 [3] occurred at disk power-on lifetime: 2174 hours (90 days + 14 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 51 00 00 00 00 18 b7 da b0 40 00  Error: WP at LBA = 0x18b7dab0 = 414702256

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  61 00 40 00 50 00 00 22 19 ab a0 40 08     06:45:34.253  WRITE FPDMA QUEUED
  60 01 00 00 48 00 00 18 b6 04 08 40 08     06:45:34.250  READ FPDMA QUEUED
  61 00 40 00 40 00 00 22 19 a9 20 40 08     06:45:34.250  WRITE FPDMA QUEUED
  60 00 c0 00 38 00 00 18 b6 01 88 40 08     06:45:34.250  READ FPDMA QUEUED
  61 00 40 00 30 00 00 22 19 a4 60 40 08     06:45:34.248  WRITE FPDMA QUEUED

Error 3 [2] occurred at disk power-on lifetime: 2174 hours (90 days + 14 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 51 00 00 00 00 18 b7 da b0 40 00  Error: WP at LBA = 0x18b7dab0 = 414702256

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  61 00 40 00 00 00 00 22 18 d7 e0 40 08     06:45:31.348  WRITE FPDMA QUEUED
  61 00 40 00 00 00 00 22 18 d3 e0 40 08     06:45:31.345  WRITE FPDMA QUEUED
  61 00 40 00 00 00 00 22 18 bc a0 40 08     06:45:31.345  WRITE FPDMA QUEUED
  61 00 40 00 00 00 00 22 18 b9 60 40 08     06:45:31.331  WRITE FPDMA QUEUED
  61 00 40 00 00 00 00 22 18 b5 60 40 08     06:45:31.331  WRITE FPDMA QUEUED

Error 2 [1] occurred at disk power-on lifetime: 2174 hours (90 days + 14 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 51 00 00 00 00 18 b7 da b0 40 00  Error: WP at LBA = 0x18b7dab0 = 414702256

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  61 00 40 00 e0 00 00 22 18 76 20 40 08     06:45:28.453  WRITE FPDMA QUEUED
  61 00 40 00 e0 00 00 22 18 6c a0 40 08     06:45:28.452  WRITE FPDMA QUEUED
  61 00 40 00 e0 00 00 22 18 5e e0 40 08     06:45:28.452  WRITE FPDMA QUEUED
  61 00 40 00 e0 00 00 22 18 56 60 40 08     06:45:28.452  WRITE FPDMA QUEUED
  61 00 40 00 e0 00 00 22 17 fd 20 40 08     06:45:28.452  WRITE FPDMA QUEUED

Error 1 [0] occurred at disk power-on lifetime: 2174 hours (90 days + 14 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 51 00 00 00 00 18 b7 da b0 40 00  Error: WP at LBA = 0x18b7dab0 = 414702256

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  61 00 40 00 a0 00 00 22 17 72 20 40 08     06:45:25.568  WRITE FPDMA QUEUED
  61 00 40 00 a0 00 00 22 17 70 60 40 08     06:45:25.565  WRITE FPDMA QUEUED
  61 00 40 00 a0 00 00 22 17 6d 60 40 08     06:45:25.550  WRITE FPDMA QUEUED
  60 01 00 00 98 00 00 18 b7 da 48 40 08     06:45:25.548  READ FPDMA QUEUED
  60 00 40 00 90 00 00 18 b7 d5 48 40 08     06:45:25.548  READ FPDMA QUEUED

 

[AMiGAmann]

Hi anodos,

are you sure that you are not mixing something up? The information you wrote down is certainly wrong in some points:

Hard drives are WD Green

Hard drives are definately WD RED

load cycle counts above 200000

The highest LCC is currently 308

The server does not appear that smart monitoring has been configured

Long self-test is performed twice a month, short self-test is performed 4 times a month

Pool consists of a single 5-disk RAIDZ1 vdev.

Pool consists of 8-disk RAIDZ2

/dev/ada4, ... /dev/ada2...

Those devices do not exist, hds are da0 to da7

Very strange...

 

[anodos]

Hi anodos,

are you sure that you are not mixing something up? The information you wrote down is certainly wrong in some points:


Hard drives are definately WD RED


The highest LCC is currently 308


Long self-test is performed twice a month, short self-test is performed 4 times a month


Pool consists of 8-disk RAIDZ2


Those devices do not exist, hds are da0 to da7

Very strange...

That'd be because I was looking at someone else's debug file. Sucks to be that guy. :D

 

[AMiGAmann]

That'd be because I was looking at someone else's debug file.

Ah, okay, I was really wondering...

 

[anodos]

Ah, okay, I was really wondering... :)

Anyway /dev/da7 is showing a bunch of UDMA CRC Errors. Might want to check your cables. Samba is generating some strange errors I haven't seen before.

Code:

[2016/01/14 19:26:34,  2] ../source3/lib/tallocmsg.c:124(register_msg_pool_usage)
  Registered MSG_REQ_POOL_USAGE
[2016/01/14 19:26:34,  2] ../source3/lib/dmallocmsg.c:78(register_dmalloc_msgs)
  Registered MSG_REQ_DMALLOC_MARK and LOG_CHANGED

It might mean that your FreeNAS install is subtly jacked up. If possible, it might be worthwhile to try a fresh install on a new USB stick, import your pool, configure a test share and see what sorts of speeds you get. No link aggregation with SMB2_10 as max protocol.

 

[AMiGAmann]

Anyway /dev/da7 is showing a bunch of UDMA CRC Errors. Might want to check your cables.

I noticed the UDMA CRC Errors right after first installation of FreeNAS. I already replaced that cable and the number of errors did not rise after that, so the cable I am now using seems to be fine.

It might mean that your FreeNAS install is subtly jacked up.

How could that happen? "System/Update/Check Now" tells me there are no updates and "System/Update/Verify Install" tells me everything is valid (except resolv.conf which is actually a bug).

No link aggregation with SMB2_10 as max protocol.

I reconfigured Link Aggregation (Fail Over) but destroyed it for testing again. The transfer speeds did not really change, regardless of enabled/disabled link aggregation. Link Aggregation (Fail Over) and SMB3 are my preferred settings.

I created another test dataset with default settings and another Windows share with default settings. The transfer speed did not change when copying a big file to it.

If possible, it might be worthwhile to try a fresh install on a new USB stick

I don't have a different USB stick big enough. Is it possible to use one of the SSDs? Currently FreeNAS is installed on 2 SSDs.

 

[AMiGAmann]

Here is a summy of my dataset and samba configuration, maybe there is actually something configured wrong...

datasets
root dataset "pool":
user root, group wheel, permission type Windows
compression lz4, share type Windows, atime off, quota/reserved 0, dedup off

all other datasets:
user cyborg, group stream, permission type Windows
compression lz4, share type Windows, atime off, quota/reserved 0, dedup off

CIFS shares:
Apply Default Permissions: true
Export Read Only: false
Browsable to Network Clients: false
Export Recycle Bin: false
Show Hidden Files: true
Allow Guest Access: false
Only Allow Guest Access: false
Hosts Allow/Deny: empty
VFS Objects: aio_pthread, streams_xattr, zfs_space
Auxiliary Parameters: empty

CIFS Settings:
DOS charset: CP437
UNIX charset: UTF-8
Log level: Normal
Use syslog: true
Local Master: true
Domain logons: false
Time Server for Domain: false
Guest account: nobody
File mask: 0660
Directory mask: 0770
Allow empty Password: false
Auxiliary parameters: ea support = no, store dos attributes = no, map archive = no, map hidden = no, map readonly = no, map system = no
Unix Extensions: false
Zeroconf share discovery: false
Hostnames lookups: true
Server minimum protocol: NT1
Server maximum protocol: SMB2
Allow execute always: false
Obey pam restrictions: true
Bind IP Addresses: -
Idmap Range Low: 90.000.001
Idmap Range High: 100.000.000



I have configured ACLs from Windows Clients which give full access to the user of the share and read access to the group of the share.

 

[SweetAndLow]

You should keep your root partitions with the default permission settings. So root:wheel and Unix. You might have to rebuild your pool to get it right because in not sure of some of the inner workings line zfsacl mode and what it should be set for your root pool. This probably isn't causing your performance issues but is something you should fix.

 

[AMiGAmann]

It would be more easier for newbies as me if the wizard would give a hint that it is important to leave the defaults when first creating a dataset.

Do you know what consequences the "permission type Windows" on the root partition might have? I currently do not have any problems with ACLs, they are working fine.

Edit: Isn't it possible to just change the permission type from Windows to Unix? At least there is the option to change it in GUI.