I built a new FreeNAS (on ESXi) server, 11.2, and I'm trying to copy all the data from my old Ubuntu server to it. There's a point to point 10GbE cable between them, and when I run iperf in either direction it measures 4gbps. I'm happy with that, it's a limitation of virtualizing the NIC in ESXi instead of passing it through.
In any case, I've tried about a dozen different ways to copy the data from the old server to the new, but almost all of them max out at 75MBps (600mbps) or less. I've confirmed that the transfer is using the 10GbE path (both FreeNAS & Ubuntu show traffic on the 10GbE interfaces, not on the 1GbE), but I can't get the speed up no matter what I've tried.
Here are several of the things I've tried, with some notes/thoughts about what I think it tells me:
1.) Mount the FreeNAS pool under Ubuntu using NFS, run rsync on Ubuntu as a "local" transfer = 75-80 MBps
2.) Don't mount anything, run rsync on Ubuntu as a "remote" transfer to 'user@FreeNAS:/mnt/Tank' = 35-40 MBps
Clearly mounting the share and doing a local transfer is faster. Presumably because it skips the SSH tunnel, encryption, etc.
3.) Mount the FreeNAS pool under Ubuntu using NFS, run dd on Ubuntu = 75-80 MBps
4.) Mount the FreeNAS pool under Ubuntu using NFS, run cp on Ubuntu = I forget the speed, but it wasn't any higher, might have been a little lower
The above two show it's not a limitation of rsync, I think, since 'dd' runs the same speed. I didn't expect 'cp' to be fast, but I did it for the heck of it.
5.) Mount the Ubuntu share under FreeNAS using mount_smbfs, run rsync on FreeNAS as a "local" transfer = 75-80 MBps
The CPU usage on Ubuntu was never higher than 30-35% during any of the above methods, but I thought I'd try moving some execution off to FreeNAS instead since it's a beefier server. Basically #5 is #1 in reverse, hoping to offload everything but the actual data movement from Ubuntu. The above result seems to show that it's not a CPU limitation. The usage stayed around 30% on Ubuntu, and speed didn't improve at all.
6.) SSH into Ubuntu and do Step #1 three times on three different folders, hoping to get ~3x aggregate speed (ie: parallel transfers)
CPU utilization on Ubuntu went up to 60-70% or so, as expected, but transfer speed ended up being split between the three, totaling 80MBps, the same as before. So the limit is definitely not CPU/mem on the old server, it must be something about the 10GbE network path?
7.) Using a Windows workstation on the 1GbE network, mount both old and new servers' Samba shares, then drag and drop folders from old to new = 90 MBps
This last one is the real kicker. It uses the 1GbE network instead of the 10GbE network, but manages to run ~15% faster, clearly limited by the 1GbE interface on the Windows machine. I can't for the life of me understand why a point to point transfer over the 10GbE network goes even slower than this drag and drop over 1GbE through Windows and multiple switches. I would suspect something misconfigured about the 10GbE interface, maybe even misconfigured in ESXi, but using iperf between FreeNAS and Ubuntu (in either direction) I get 4gbps no problem..? I'd be thrilled to get even half that moving real data across, which would be more than 3x what I'm actually getting now. Does anyone have any suggestions?
I'm really hoping this is one of those "you idiot" moments where I've overlooked something very simple. It's sad and humorous at the same time that my best bet for copying data at this point is a drag and drop in Windows to get around the "slower" 10GbE link, lol :) It's no huge deal since this is a one-time transfer and even if it takes days it won't bother me. But in the future I want to run rsync backups from FreeNAS to Ubuntu over the 10GbE link, and I'd like to get as much bandwidth as possible out of the link. Thanks!
In any case, I've tried about a dozen different ways to copy the data from the old server to the new, but almost all of them max out at 75MBps (600mbps) or less. I've confirmed that the transfer is using the 10GbE path (both FreeNAS & Ubuntu show traffic on the 10GbE interfaces, not on the 1GbE), but I can't get the speed up no matter what I've tried.
Here are several of the things I've tried, with some notes/thoughts about what I think it tells me:
1.) Mount the FreeNAS pool under Ubuntu using NFS, run rsync on Ubuntu as a "local" transfer = 75-80 MBps
2.) Don't mount anything, run rsync on Ubuntu as a "remote" transfer to 'user@FreeNAS:/mnt/Tank' = 35-40 MBps
Clearly mounting the share and doing a local transfer is faster. Presumably because it skips the SSH tunnel, encryption, etc.
3.) Mount the FreeNAS pool under Ubuntu using NFS, run dd on Ubuntu = 75-80 MBps
4.) Mount the FreeNAS pool under Ubuntu using NFS, run cp on Ubuntu = I forget the speed, but it wasn't any higher, might have been a little lower
The above two show it's not a limitation of rsync, I think, since 'dd' runs the same speed. I didn't expect 'cp' to be fast, but I did it for the heck of it.
5.) Mount the Ubuntu share under FreeNAS using mount_smbfs, run rsync on FreeNAS as a "local" transfer = 75-80 MBps
The CPU usage on Ubuntu was never higher than 30-35% during any of the above methods, but I thought I'd try moving some execution off to FreeNAS instead since it's a beefier server. Basically #5 is #1 in reverse, hoping to offload everything but the actual data movement from Ubuntu. The above result seems to show that it's not a CPU limitation. The usage stayed around 30% on Ubuntu, and speed didn't improve at all.
6.) SSH into Ubuntu and do Step #1 three times on three different folders, hoping to get ~3x aggregate speed (ie: parallel transfers)
CPU utilization on Ubuntu went up to 60-70% or so, as expected, but transfer speed ended up being split between the three, totaling 80MBps, the same as before. So the limit is definitely not CPU/mem on the old server, it must be something about the 10GbE network path?
7.) Using a Windows workstation on the 1GbE network, mount both old and new servers' Samba shares, then drag and drop folders from old to new = 90 MBps
This last one is the real kicker. It uses the 1GbE network instead of the 10GbE network, but manages to run ~15% faster, clearly limited by the 1GbE interface on the Windows machine. I can't for the life of me understand why a point to point transfer over the 10GbE network goes even slower than this drag and drop over 1GbE through Windows and multiple switches. I would suspect something misconfigured about the 10GbE interface, maybe even misconfigured in ESXi, but using iperf between FreeNAS and Ubuntu (in either direction) I get 4gbps no problem..? I'd be thrilled to get even half that moving real data across, which would be more than 3x what I'm actually getting now. Does anyone have any suggestions?
I'm really hoping this is one of those "you idiot" moments where I've overlooked something very simple. It's sad and humorous at the same time that my best bet for copying data at this point is a drag and drop in Windows to get around the "slower" 10GbE link, lol :) It's no huge deal since this is a one-time transfer and even if it takes days it won't bother me. But in the future I want to run rsync backups from FreeNAS to Ubuntu over the 10GbE link, and I'd like to get as much bandwidth as possible out of the link. Thanks!