From 300MB/s to 50KB/s

[melloa]

Another case of was working fine yesterday and today is not...

Server:
ESXi-6.7.0-8169922-standard (VMware, Inc.)
X9DRi-LN4+/X9DR3-LN4+
12 CPUs x Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
191.97 GB

VM:
4 vCPUs
100 GB
2xLSI2008
T320 10GbE Dual Port Adapter

FreeNAS:
Build FreeNAS-11.1-U7
Platform Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
Memory 102361MB
System Time Wed, 8 May 2019 23:17:51 -0400
Uptime 11:17PM up 25 mins, 1 user
Load Average 0.49, 0.54, 0.58

Some tests:

Code:

root@mellonas:~ # dd if=/dev/zero of=tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 39.906504 secs (2690643676 bytes/sec)
root@mellonas:~ # dd of=/dev/zero if=tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 18.964400 secs (5661881475 bytes/sec)
root@mellonas:~ # iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  512 KByte (default)
------------------------------------------------------------
[  4] local 10.10.10.200 port 5001 connected with 10.10.10.60 port 54472
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-11.0 sec  5.62 MBytes  4.28 Mbits/sec
root@mellonas:~ # iperf -c 10.10.10.60
------------------------------------------------------------
Client connecting to 10.10.10.60, TCP port 5001
TCP window size:  513 KByte (default)
------------------------------------------------------------
[  3] local 10.10.10.200 port 57950 connected with 10.10.10.60 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  7.39 GBytes  6.35 Gbits/sec

Code:

root@mellonas:~ # rsync -av --progress mello@10.10.10.60:/home/mello/Desktop/CentOS-7-x86_64-Minimal-1804.iso /mnt/raid/cifs/mello/temp
The authenticity of host '10.10.10.60 (10.10.10.60)' can't be established.
ECDSA key fingerprint is SHA256:bgjtne9Sx0cpl5LN5VwLeVBUmFeEk7hw3KYoRQ3WoAI.
No matching host key fingerprint found in DNS.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.10.10.60' (ECDSA) to the list of known hosts.
mello@10.10.10.60's password:
receiving incremental file list
rsync: mkstemp "/mnt/raid/cifs/mello/temp/.CentOS-7-x86_64-Minimal-1804.iso.Bu69nQ" failed: Operation not permitted (1)
      1,966,080   0%  103.74kB/s    2:32:18  ^CKilled by signal 2.

Pool:

Code:

oot@mellonas:~ # zpool status
  pool: freenas-boot
state: ONLINE
  scan: none requested
config:

    NAME        STATE     READ WRITE CKSUM
    freenas-boot  ONLINE       0     0     0
      da0p2     ONLINE       0     0     0

errors: No known data errors

  pool: nextcloud
state: ONLINE
  scan: scrub repaired 0 in 0 days 00:28:04 with 0 errors on Sun Apr 21 00:28:04 2019
config:

    NAME                                            STATE     READ WRITE CKSUM
    nextcloud                                       ONLINE       0     0     0
      mirror-0                                      ONLINE       0     0     0
        gptid/3f161577-23f4-11e9-9fff-00074307cb17  ONLINE       0     0     0
        gptid/954b5c1e-24a0-11e9-9fff-00074307cb17  ONLINE       0     0     0

errors: No known data errors

  pool: raid
state: ONLINE
  scan: scrub repaired 0 in 0 days 11:27:25 with 0 errors on Sun Apr 21 11:27:27 2019
config:

    NAME                                            STATE     READ WRITE CKSUM
    raid                                            ONLINE       0     0     0
      raidz3-0                                      ONLINE       0     0     0
        gptid/2ba49b8a-1182-11e9-b9c3-00074307cb17  ONLINE       0     0     0
        gptid/2c7b3306-1182-11e9-b9c3-00074307cb17  ONLINE       0     0     0
        gptid/2d4f6812-1182-11e9-b9c3-00074307cb17  ONLINE       0     0     0
        gptid/2e382afe-1182-11e9-b9c3-00074307cb17  ONLINE       0     0     0
        gptid/2f0c7ec9-1182-11e9-b9c3-00074307cb17  ONLINE       0     0     0
        gptid/2fe94dc7-1182-11e9-b9c3-00074307cb17  ONLINE       0     0     0
        gptid/30bb606e-1182-11e9-b9c3-00074307cb17  ONLINE       0     0     0
      raidz3-1                                      ONLINE       0     0     0
        gptid/31aaf28b-1182-11e9-b9c3-00074307cb17  ONLINE       0     0     0
        gptid/32875852-1182-11e9-b9c3-00074307cb17  ONLINE       0     0     0
        gptid/336ed487-1182-11e9-b9c3-00074307cb17  ONLINE       0     0     0
        gptid/344b98c8-1182-11e9-b9c3-00074307cb17  ONLINE       0     0     0
        gptid/35225ea8-1182-11e9-b9c3-00074307cb17  ONLINE       0     0     0
        gptid/35fc0c8e-1182-11e9-b9c3-00074307cb17  ONLINE       0     0     0
        gptid/36e8d960-1182-11e9-b9c3-00074307cb17  ONLINE       0     0     0

errors: No known data errors

Code:

root@mellonas:~ # zpool list
NAME           SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
freenas-boot  31.8G   755M  31.0G         -      -     2%  1.00x  ONLINE  -
nextcloud     2.72T   191G  2.53T         -     0%     6%  1.00x  ONLINE  /mnt
raid          50.5T  35.4T  15.1T         -     3%    70%  1.00x  ONLINE  /mnt

Reading from the SMB shares seems to be working fine, as Plex is playing without any issues, but when trying to copy files from my workstation to the the server smb share, I noticed the speed degradation.

Even saved the config and did a fresh install just in case and the transfer speed still crawling.

Reading:

Code:

mello@olinda:/tmp$ rsync -av --progress root@10.10.10.200:/mnt/raid/cifs/repo/FreeBSD-12.0-RELEASE-amd64-dvd1.iso ./
root@10.10.10.200's password:
receiving incremental file list
FreeBSD-12.0-RELEASE-amd64-dvd1.iso
  3,884,677,120 100%   75.02MB/s    0:00:49 (xfr#1, to-chk=0/1)

sent 43 bytes  received 3,885,625,641 bytes  70,011,273.59 bytes/sec
total size is 3,884,677,120  speedup is 1.00

Swap:

Code:

oot@mellonas:/mnt/raid/cifs/repo # swapinfo -h
Device          1K-blocks     Used    Avail Capacity
/dev/mirror/swap0.eli   2097152       0B     2.0G     0%
/dev/mirror/swap1.eli   2097152       0B     2.0G     0%
/dev/mirror/swap2.eli   2097152       0B     2.0G     0%
/dev/mirror/swap3.eli   2097152       0B     2.0G     0%
/dev/mirror/swap4.eli   2097152       0B     2.0G     0%
Total            10485760       0B      10G     0%

ARC:

Troubleshooting ideas welcome :)

 

[Heracles]

Hi,

Did you checked your network ? Looking for speed / duplex mismatch, CRC error, lost packets, ...

Because you say internal activity is fine but not remote, it really looks like a network problem...

Good luck doing your bug hunting,

 

[melloa]

Unless is related with that specific NIC, and only the RX side, I can't explain. From that main server (the one with slow write speeds) to the backup server to test write speed via my 10G switch:

Code:

3,765,824,899 100%  103.58MB/s    0:00:34 (xfr#3, ir-chk=2012/25012)

Normal for rsync.

 

[melloa]

And, by the way, it might be something with FreeNAS-11.1-U7. Upgraded the VM to FreeNAS-11.2-U4 and speeds are back to normal. Will keep it @ 11.2 without upgrading the pool for awhile for testing.

 

[melloa]

Update: transfers speed degraded again; recreated pool; copying data back for testing - speed is pretty slow. More tests when data is back ...

 

[melloa]

Did you checked your network ? Looking for speed / duplex mismatch, CRC error, lost packets, ...

I think you are on something here. It is probably a tunables that is causing the problem, as a fresh install brings back the network speed and, after some time, it goes down to the drain.

Was so slow this morning that I couldn't copy a file from a workstation to the NAS.

Removed the physical interface and added the virtual switch -> transfer speeds are over 300Mb/s.