Extreme slow downs while copying files > 40 GB

[jkalousek]

I have this problem for a while but after upgrade to 11.1 this went to extremes. I have setup Proxmox with several VMs, most of them small 5-20 GB, but one of them little over 160 GB.
For a few months I experienced weird slow downs of the Proxmox server while making backups to FreeNAS but in worst case scenario it were hiccups. Today morning after upgrading my storage server to FN 11.1 (yesterday) I found out that backups that run from 11 PM weren't done at 8 AM next day and that Proxmox was almost completely hanged with lot of hang messages poping up.
So I start researching, originally my thoughts were that it's Promox fault. But right now I'm not so sure.

Servers are interconnected by 10G network but same problem occurs on 1G network so I don't thing that it's problem with HW or at least not with NICs. I tested network with iperf and everything seem OK without droping or slowing down (I got stable 6.48 Gbits/sec for 15+ minutes) after multiple test I just start hammering Freenas with different dd commands and this is what happened:

42949672960 bytes (43 GB, 40 GiB) copied, 73.9054 s, 581 MB/s
and next
48318382080 bytes (48 GB, 45 GiB) copied, 454.241 s, 106 MB/s

I then used rsync and tried to copy backup of largest VM to actually see where it will starts slowing down and transfer started to slow down around 30 % (~ 45 GB) and were completely dead around 35 %. What I mean by that is below transfered 30 %, speed was around 300 MB/s and then beyond 35 % speed decreased to 2-10 MB/s and stay there.

I tested both NFS and CIFS shares and both were behaving nearly same. I tried to set NFS (on Proxmox site) to async, sync, soft, hard, different wsize but nothing seems to have any effects.

I found lot of similar problems but none of suggested solutions worked.

According to my observations from Graphs on FN there is no noticeable CPU usage, RAM will have some dips and risez at the start of the transfer (but that should be normal) and disc are busy around 70 %.

My HW config of FreeNAS server:
CPU: Intel Xeon E3-1220 v6
MB: Supermicro MBD-X11SSL-CF
RAM: 2x Hynix MEM-DR416L-HL01-EU24 16GB ECC
HDD: 6x Seagate Ironwolf 4TB in RAID 10 (3 groups x 2 disc) for data
SSD: 2x SUPERMICRO satadom in raid 10 as boot
NIC: SUPERMICRO AOC-STGN-I1S for 10GB interconnect

Is there any tests that could shine some light on what is bottleneck? Right now I can't let the biggest VM to do backups at all (across network) and I have to do some temporary backups on Proxmox server itself (local backups on Proxmox are not causing any slowdowns).

EDIT: VM backup is tar.gz, one big file...

 

[jkalousek]

OK, Update. I'm now sure that it's caused by FreeNAS, I just copied same file from Proxmox to Synology DS1513+ and transfer rate was constant ~55 MB/s on (moderately busy) 1G network. Almost no speed spikes and falls during transfer. I will make some additional tests and report new findings.

 

[jkalousek]

Results are getting weird:
Proxmox -> FreeNAS (NFS, CIFS): Slowing down (both on direct interconnect 1G & 10G speed through switch)
Proxmox -> Synology (CIFS): Normal Speed (1G through switch)
Windows -> FreeNAS (CIFS): Normal Speed (1G through switch)

So there must be something that don't work between Linux machine and FreeNAS, and its not tied to NIC or protocol. Is there anyone who could to try to help me point out what could cause this?

 

[jkalousek]

Today I had other backup failure on another much smaller vm because of backup timeout / slow speed. Anyone? I pretty much run out of ideas...

 

[bodriye]

Did you turn off large send offload and set send and receive buffers to 2048 on windows

 

[jkalousek]

@bodriye I do not have problem on Windows as stated in 3rd post. Only problem occurs during transfer from Linux (Proxmox) -> FreeNAS, but not while transferring from Proxmox to other servers so there is problem only between Linux and FN.

 

[bodriye]

@bodriye I do not have problem on Windows as stated in 3rd post. Only problem occurs during transfer from Linux (Proxmox) -> FreeNAS, but not while transferring from Proxmox to other servers so there is problem only between Linux and FN.

Another thing could be your 10Gbe nic, I know intel 10Gbe nics are broken in freebsd (I know pfsense firewall that runs on freebsd also has issues with intel 10Gbe nics) I would look into something like Chelsio T520-BT (10Gbase-T) or Chelsio T520-SO-CR (SFP+)

 

[jkalousek]

Another thing could be your 10Gbe nic, I know intel 10Gbe nics are broken in freebsd (I know pfsense firewall that runs on freebsd also has issues with intel 10Gbe nics) I would look into something like Chelsio T520-BT (10Gbase-T) or Chelsio T520-SO-CR (SFP+)

Yeas I'm aware about that (my SFP is using Intel 82599EN), but I would suspect that problems with NIC would affect all traffic, but I can get fast speeds from everywhere but Proxmox. And I tested multiple other 1G NICS on both sites and base speed is lower of course but same slow down occurs.

 

[jkalousek]

I still have this problem but now I believe that it's caused by ZFS + Proxmox not FreeNAS. Thanks to everybody that tried to help!

 

[marshalleq]

Well just to throw a spanner in the mix, I have this problem on Proxmox 5.2 on local drives - no network. Backups of virtual machines that took about 40 minutes to copy were still running this morning 7:00am - (started at 1am). This is since I did a rebuild on Proxmox and for some reason one drive which is formatted ext4 (for a backup drive) is now slowing down some time during a large copy. This didn't happen before the rebuild. Thought it might help your thought process - or just confuse it a little more!