Large files > 4GB won't copy to linux jail

[squidfoo]

I am new to FreeNAS, though am not totally new to FreeBSD. I have this recent installation of FreeNAS 9.2 STABLE on HP Proliant Microservers with 8GB each, both upgraded to 9.2.1.

The issue I have is with copying files to a linux jail, i.e. rsync or scp from a client host connecting to a service that runs inside the jail. On every attempt I ended up with either one of the following error messages:

(inside the linux jail)

Code:

rsync: write failed on "/srv/samba/share/public/./VMs/VBox for iOS/VirtualBox VMs/IE7/Snapshots/{15ecfb1e-699a-4d24-9b66-e25f10b247e1}.vhd": File too large (27)
rsync error: error in file IO (code 11) at receiver.c(322) [receiver=3.0.9]

The file in question is around 11GB.

Another instance inside a linux jail, the transfer of another large file > 8GB fails/stops at around 4294836224 bytes.

From the outside, I get similar errors, going with or without ssh:

Code:

$ scp ext/INFRA-db-backup/old/ibdata1 squidfoo@192.168.200.29:~/ext/INFRA-db-backup/old/.
ibdata1                                                                                     39% 4085MB  18.8MB/s  19.4MB/s   05:31 ETAlost connection

Code:

rsync -e "ssh -l squidfoo"  ext/INFRA-db-backup/old/ibdata1 squidfoo@192.168.200.25:~/test/.
rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32)
rsync: write failed on "/home/squidfoo/ext/INFRA-db-backup/old/./ibdata1": File too large (27)
rsync error: error in file IO (code 11) at receiver.c(322) [receiver=3.0.9]
rsync: connection unexpectedly closed (524372 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9]

Tbh, I failed to watch memory or other stats while this happened. I did check rsync version: 3.0.9 in all cases, and recompiled rsync with xattr suppport to match the FreeNAS setup. No change in behaviour.

Transfers of files smaller than ~4GB usually transfer just fine. As a a test, I transferred the same files to my userhome on the NAS (on the same ZFS volume), which was much faster, about as fast as this 1Gbit/s network gets, and finished without any error or complaint.

This smells like a bug. What do you think? For the above test I had used a Ubuntu 13.04 jail, as configured in the GUI. The Ubuntu-12.04 jail I was not able to get going properly. I'd so like to let our developers do whatever they want confined to a linux jail.

Next test, will setup a Gentoo-20130820 linux jail and repeat above transfers to verify.

 

[cyberjock]

Check out https://bugs.freenas.org/issues/3970.

I don't think anyone knows what the problem is, or how to fix it. :(

 

[squidfoo]

That was quick. Apparently I hadn't searched the bug tracker properly.

 

[cyberjock]

I was directly involved with that problem, that's why I knew exactly what to search for. Unfortunately, the problem is confusing since one person did exactly what I did, but we got different results.

 

[squidfoo]

Ohmigod, things get worse. I did the same test with a Gentoo jail, and got this.

This is inside the jail. With the first part I was very pleased, but not with the second.

Code:

# scp squidfoo@192.168.200.204:~/ext/INFRA-db-backup/old/ibdata1 /tmp/.
Password:
ibdata1                                                                                    100%  10GB  19.5MB/s  20.3MB/s  08:49
 
# ls -lAh /tmp/
total 2.1G
-rw-r----- 1 root root 2.1G Oct  8 07:06 ibdata1

Looks like ~8GB of data went ... somewhere, but are not on this target system.

Next test, from a client an rsync transfer went without any trouble at all:

Code:

$ rsync -e "ssh -l squidfoo -p222" ext/INFRA-db-backup/old/ibdata1 squidfoo@192.168.200.31:~/test/.
$

This time, from inside the jail transfer looked good while it was running and after it finished:

Code:

 $ ls -lAh test/
ls -lA test/
total 10579073
-rw-r----- 1 squidfoo squidfoo 10822352896 Feb 19 06:48 ibdata1

Also, both sha256sums match. Success! But only in 'push' direction apparently. This is strange.

Many a way to go from here.

• I do want to isolate developers from the FreeNAS host system. No need to have the NAS tampered with
• Since I run most of my hosts on Gentoo anyhow, I might just continue testing this, developers will cope. SMB/CIFS is a main concern
• Probably will provide a FreeBSD jail if that fails
• Could also try all the other jails on offer, but theres only so much time for testing. Otoh, is there a linux-jail-herd with the FreeNAS team somewhere?

 

[cyberjock]

That is totally creepy. I wonder what would happen if you did it on the same linux version in a non-jail setting

 

[squidfoo]

I do not have a single 32bit system around atm. Will test, and report back. I expect no surprise whatsoever.

 

[cyberjock]

Could do a VM with VMWare Player or virtualbox.. hmmm...

 

[squidfoo]

Things get interesting, and at a later point I might come back. Practically it looks like linux jails need some more love.

My pragmatic workaround:

• offer isolated services and storage in native FreeBSD jails, developers will cope
• play around with linux jails more when I have time

I did get in contact with ryao to ask him for hints and/or advice; he's done a lot of work in the Gentoo FreeBSD, Gentoo and ZFS worlds.

Gentoo Linux is doable. ... I recall having to downgrade tar or symlink it to busy box. In addition, touch and one other utility from coreutils must be symlinked. I cannot recall which, but ln or chown seems likely. When the linuxulator complains about an unimplemented syscall, you could try grepping the binaries to see which has it in its ELF header. That might not necessarily work, but it is an idea. Lastly, glibc could not be built in the linuxulator because trace was not implemented.

Surely this is all doable, especially since teh internet already has some answers. But for me this has to be postponed. Meanwhile, the Gentoo-related bit of this discussion could continue over at the Gentoo forums.

 

[cyberjock]

Yeah, after the linux jails rolled out it became obvious it needs more love. The linux emulation layer needs more work, but apparently there's no active development going on from that front. So it is what it is.