mount_nullfs - mounting a directory in another directory from different pool, shared via NFS?

[FakeMoth]

Hi got a kind of a problem and I don't know how can I investigate further:

-have two pools on my FreeNAS: vol0, comprised of 3 Intel SSDs with a Intel M2 Optane SLOG and vol1 6x2TB Seagate;

-vol0 is shared via NFS on 10Gb/s to a couple of physical hosts, and I have a few KVM VMs running just fine for years;

-I am trying to backup the content of VMs (not the whole VM, working, as the hosts just scp on vol1): on vol0 works just fine, but they are producing 100GB/day in backups, so I don't want to ruin my SSDs plus is bad practice; used rsync and stuff to vol1, whatever;

-so the VM ssh-es into the host and just writes the backup on the mounted NFS share; they have publicc IPs, and I don't want to mount my internal NAS network in a VM running public services... so please do not recommend that;

-figured I have to mount some directory from vol1 to vol0; and I did after moving all the backups on vol1: mount_nullfs /mnt/vol1/Backups/DC/BIGSTORAGE /mnt/vol0/4-Backs/SSDSTORAGE;

-as I understand it this is the equivalent onf Linux mount --bind

-everything seemed fine, I can see in FreeNAS CLI the vol1 directories and files but in vol0, I can touch and mkdir stuff from the FreeNAS CLI in /mnt/vol0/4-Backs/SSDSTORAGE

But backups from VMs aren't writing on host, or host not writing on FreeNAS in the mounted dir and everything is happening as root:

.. upload failed! root@1.2.3.4's password:
scp: /mnt/4-Backs/SSDSTORAGE/NewDir: Input/output error
Killed by signal 1.

Ummm my reasoning was something like: if my NFS performance tanks when mounting vol1 directly in host, I should still use vol0 but let FreeNAS write with like 200MB/s directly, instead of 5MB/s via NFS :)

Guess I am wrong?

 

[dlavigne]

You shouldn't be getting a signal 1, so something is going on. Were you able to figure out what?

 

[Chris Moore]

.. upload failed! root@1.2.3.4's password:
scp: /mnt/4-Backs/SSDSTORAGE/NewDir: Input/output error
Killed by signal 1.

This looks like a password error.

and everything is happening as root:

My guess, when you did it on the FreeNAS side, you set it up as the root user, and it worked, but when you are trying to access it from the remote system it is asking for password validation, and that is failing.

 

[FakeMoth]

Hi, thanks for replying.

But I gave up on everything, and mounted via NFS the second volume (vol1) on the host servers, even it it writes to a crawl. And the VMs just write their backups SSHing into the hosts. I will move this pool to another FreeNAS machine and will make use of a SLOG also.

The reason is... weird. It didn't had anything to do with the password or permission though. It's because even when trying to unmount the directory it "left" somekind of shadow file/inode, that you couldn't do anything about... Never seen anything like this. Left it like that, too busy with stuff and made a note to deal with it later.

Soon after I received an email from the router, that it couldn't get the UPS status; it is a NUT slave configuration to the FreeNAS server which is the NUT master. I ignored it.

Than I received an alarming number of email alerts from netdata, running in the VMs. That's when I logged into FreeNAS and in the interface just showed some weird kernel and php errors. Never saw those either.

In panic I just restarted FreeNAS; physically, because I couldn't do anything in the interface and couldn't login with SSH.

So I will never mount shit between pools of a FreeNAS, shared via NFS :)

A Happy New Year to the FreeNAS community!​