Core: docker and NFS

[Yorick]

After some experimentation, I have Ubuntu 20.04 working with docker and an NFS mount on TrueNAS Core 12.0-U8.1

On TrueNAS, configure the NFS service to support "NFSv4" and use "NFSv3 ownership model", and start it. If it was already started, stop and start it.
Create a dataset. 16kb recordsize can be prudent if you are going to run DB workloads in docker.
Configure an NFS share for that dataset and set "Maproot User" to "root", "Maproot Group" to "nogroup". "All dirs" if you like, and "Enabled". I also added my network under "Authorized Networks", but that doesn't sound necessary.

On Ubuntu

Install prerequisites: sudo apt update && sudo apt -y install nfs-common fuse-overlayfs
Stop docker services: sudo systemctl stop docker && sudo systemctl stop docker.socket
Create a directory to mount into: sudo mkdir -p /mnt/NFS
Add a line to fstab: TRUENAS-IP:/mnt/poolname/dataset /mnt/NFS nfs nfsvers=4,rsize=1048576,wsize=1048576,timeo=14,intr 0 0
Where you'll set TRUENAS-IP to the IP address of TrueNAS Core and /mnt/poolname/dataset to the path you used for the NFS share
Test it works: sudo mount -av
Provided it got mounted, copy the docker files over: sudo cp -rp /var/lib/docker /mnt/NFS
Rename the old docker file location: sudo mv /var/lib/docker /var/lib/docker.old
Create a docker configuration file: sudo vi /etc/docker/daemon.json
and place into it:

Code:

{
   "data-root": "/mnt/NFS/docker",
   "storage-driver": "fuse-overlayfs"
}

Lastly, start docker again: sudo systemctl start docker

I tried to test performance with sudo fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=/mnt/NFS/test --bs=4k --iodepth=64 --size=150G --readwrite=randrw --rwmixread=75 && sudo rm /mnt/NFS/test and that didn't go anywhere in a hurry. I then tried it with a 4G file and got r/w IOPS of 3900/1300, which is pretty terrible for SSD-backed storage. A speed daemon this ain't.

Once you are happy that everything is working the way you want it to you can remove the old docker files, sudo rm -rf /var/lib/docker.old

 

[BrianLakstins]

I have a similar configuration, with the following differences.

1. Each volume I use is mounted to a different NFS share.

This is the generic format I used in docker-compose:

volume-name:
driver_opts:
type: "nfs"
o: "nfsvers=4,addr=ip.of.server(dns name did not work),rw"
device: ":/mnt/pool-name/dataset-name/dataset-sub-name/"

This is an example:

grafana-data:
driver_opts:
type: "nfs"
o: "nfsvers=4,addr=10.101.0.9,rw"
device: ":/mnt/data/docker-volumes/dd_grafana-data-nfs/"

2. In the Advanced section of setting up my NFS share, I set "Maproot User" to "root" and "Maproot Group" to "wheel"

3. 128 KiB dataset size.

The speed for using a MySQL database was not acceptable, but there are other factors that I think led to it. My hard drives seemed to be dying a slow death (errors on Truenas about degraded speed), and my Ubuntu server is a VM on a Windows 10 system using Hyper-V which seems to give my spotty network speed. It's probably better to have the MySQL database on a local volume, and back it up to a volume on the NAS anyway.