Rsync makes TrueNAS use all available memory for "services" | Rsync is very slow

[The_Saver]

Hello everyone,

I am in the progress of migrating over to TrueNAS-13.0-U1.
I have used MergerFS on Ubuntu Server before. This means that I have used JBOD so all disks store different files without redundancy.

My configuration is stock, except changing timezone and adding a user for use with SMB shares & enabling SSH to run Rsync. I am running Rsync so that I can move data from up to 15 disks at the same time, but I am encountering a bottleneck with just a single disk.

I have created a 6 disk RAIDZ2 with 8TB disks with ZSTD-1 compression.
I have added my PCI-E 4.0 NVMe SSD 2 TB as cache.

CPU is Ryzen 5800X.


Here are the commands I run:

Code:

gpart show # to get the disk number and partition
kldload fusefs # so I can use ntfs-3g
ls /dev | grep da0 # to find the correct partition
mkdir /mnt/disk1 # create folder to mount disk
ntfs-3g /dev/da0p1 /mnt/disk1/ # mount disk
rsync -P -a --remove-source-files /mnt/disk1/ /mnt/storage/storage/ # start transfer

At first, the disk is the bottleneck, but slowly my memory fills up and the speed drops to almost a quarter because it transfers and then waits ~2 seconds before continuing. CPU usage is always below 10%, often at just 1-2%.

I have used the same Rsync command when I had Ubuntu Server installed without any bottlenecks. Running multiple Rsync instances is still no problem at all because the disks are always the bottleneck. So I know that the hardware works. I ran Ubuntu Server this morning just fine.

I have tried 3 different disks, same result. I have also tried without the 2 TB SSD cache.

The cache fills up as quickly as I transfer files. Transferring 30 GB equals 30 GiB memory usage on Services.

When I stop the transfers, the Services memory usage stays the same. Rebooting fixes the issue, until I start transferring again.

 

[Samuel Tai]

Did you confuse the compression and dedupe settings for your dataset? This behavior smells like dedupe is turned on your system, but there’s not enough RAM to build the dedupe tables.

 

[Samuel Tai]

Also, is there any reason why you’re not using the built-in Import Disk procedure, which can handle NTFS disks?

 

[The_Saver]

Did you confuse the compression and dedupe settings for your dataset? This behavior smells like dedupe is turned on your system, but there’s not enough RAM to build the dedupe tables.

Thank you for your reply!

I don't think I did. Here are my settings:

I have tried to disable Sync and Atime. But I can't observe any difference.

 

[The_Saver]

Also, is there any reason why you’re not using the built-in Import Disk procedure, which can handle NTFS disks?

Good question!

I wanted to use the feature, but unfortunately it only works on a single disk at a time. I have 15 disks that I need to transfer to TrueNAS/ZFS.

Additionally, I think it is also easier for me to keep track of my data if I move it over to TrueNAS/ZFS because when I move the data the source disk is empty when I have finished transferring.



My file size range is 1GB-75GB. So I do not have a lot of small files.

 

[Samuel Tai]

Just to make sure we didn't overlook anything, did you enable the Rsync service under Services->Rsync?

 

[The_Saver]

Just to make sure we didn't overlook anything, did you enable the Rsync service under Services->Rsync?

Rsync service does not run.

Only S.M.A.R.T., SMB and SSH run. They are all 3 set to automatically start at boot up.

 

[Samuel Tai]

OK, what's probably happening is FUSE ntfs-3g running amok with RAM, because you've specified --remove-source-files. The Import disk procedure mounts the source disk read-only to avoid this case.

You're probably stuck with the Import Disk procedure, and moving disks from the To-Do pile to the Done pile after you complete each one. Then when all the imports are done, you can remount each disk read-write and then wipe them from the UI.

 

[The_Saver]

OK, what's probably happening is FUSE ntfs-3g running amok with RAM, because you've specified --remove-source-files. The Import disk procedure mounts the source disk read-only to avoid this case.

You're probably stuck with the Import Disk procedure, and moving disks from the To-Do pile to the Done pile after you complete each one. Then when all the imports are done, you can remount each disk read-write and then wipe them from the UI.

I got an idea which failed. I mounted my disks as described above and then I tried to mount them as SMB share via the GUI. But unfortunately I got this error message: the path must reside within a volume mount point.


This led me to try your suggestion to import the disks 100% via the GUI and wipe them afterwards.
So, I have now tried to import my disks that I have transferred from earlier. I have currently 2 different ones connected and they both give the same error message:

What are your suggestions for how I can import my disks (hopefully quickly)?

 

[Samuel Tai]

How are you physically connecting the disks?

 

[The_Saver]

How are you physically connecting the disks?

Backplane and HBA

 

[The_Saver]

If I run gpart show, this is what I get:

 

[Samuel Tai]

What types of files are you trying to import? You already said they're 1-75 GB in size. The error message indicates rsync tried to read in a line, and ran out of memory before finding a line ending.

 

[The_Saver]

What types of files are you trying to import? You already said they're 1-75 GB in size. The error message indicates rsync tried to read in a line, and ran out of memory before finding a line ending.

MKV video files.

 

[Samuel Tai]

OK, so rsync, which the Import Disk routine runs behind the scenes using arguments --info=progress2 --modify-window=1 -rltvh --no-perms doesn't appear to be the correct tool.

You may need to do some scripting. Since you've worked out how to mount the source disks, you could try something like:

cat /mnt/disk1/file1.mkv > /mnt/storage/storage/file1.mkv

for each file. Try a few manually first, to see if this will work. If it does work, then you can automate it with a shell script.

 

[winnielinnie]

cat /mnt/disk1/file1.mkv > /mnt/storage/storage/file1.mkv

for each file. Try a few manually first, to see if this will work. If it does work, then you can automate it with a shell script.

Wouldn't using cat instead of rsync not preserve the modification times?

 

[Samuel Tai]

Wouldn't using cat instead of rsync not preserve the modification times?

Yes, but this is easily remedied afterwards via touch -r /mnt/disk1/file1.mkv /mnt/storage/storage/file1.mkv.

 

[The_Saver]

OK, so rsync, which the Import Disk routine runs behind the scenes using arguments --info=progress2 --modify-window=1 -rltvh --no-perms doesn't appear to be the correct tool.

You may need to do some scripting. Since you've worked out how to mount the source disks, you could try something like:

cat /mnt/disk1/file1.mkv > /mnt/storage/storage/file1.mkv

for each file. Try a few manually first, to see if this will work. If it does work, then you can automate it with a shell script.

So I got a hunch.

Since I moved a lot of files using: "rsync -P -a --remove-source-files /mnt/temp/disk1/ /mnt/storage/storage/" it meant that folders were not deleted after the files were moved. So I had a lot of empty folders.

So I ran the command inside the root of the disk's mountpoint to delete all empty folders: find . -type d -empty -delete

Afterwards I was able to import the disk via the GUI. Unfortunately, it still uses all my RAM on Services. As soon as the RAM is used all up on Services, my transfer speed drops to a quarter or so. Please see the screenshot below:

And yes, after deleting all empty directories I restarted the server. Then I did the disk import in the GUI without running any commands and mounting any disks.

 

[The_Saver]

Wouldn't using cat instead of rsync not preserve the modification times?

I just tried to use the cat command - but unfortunately I experience the exact same behaviour where all my RAM gets used by Services.

Isn't this just very odd in general that all my RAM gets eaten up by Services?
I haven't even experienced my cache SSD being used. It is a very fast PCI-E 4.0 NVMe 2TB SSD.

 

[The_Saver]

If I just use SMB to copy and paste random files to and from my storage pool it works just fine. I use my RAM for ZFS cache (but I don't really think my 2 TB SSD cache is being used at all):

This made me think. How can I mount my devices to be used via SMB? Because then I could use SMB to move files from my disks over to my pool.