Input/output error / NFS server not responding

[Kai Timmer]

Hello,
I'm quite new to Freenas and BSD. With my installation I ran into the following problem.

I'm using freenas nfs shares as a backup space for some servers. All my script does is mount the freenas nfs share and than use rsync to copy a few folders on a regular basis. Freenas/ZFS than does snapshots on the shares.

So far so good, but what I experience is, that I get a lot of these messages:

Code:

rsync: readdir("/mybackup/files/files.txt"): Input/output error (5)

and in dmesg:

Code:

nfs: server freenas.intern.lan not responding, timed out

cat /proc/mounts shows the following mount options

Code:

 nfs rw,relatime,vers=3,rsize=16384,wsize=16384,namlen=255,soft,proto=udp,timeo=60,retrans=20,sec=sys,mountaddr=172.16.4.25,mountvers=3,mountport=726,mountproto=udp,local_lock=none,addr=172.16.4.25

Any ideas on what might be the problem? I'm happy to provide more information on the setup if needed.

 

[dlavigne]

Does the problem persist if you use -o tcp when mounting the share?

 

[Kai Timmer]

Yes, same Problem.

 

[Kai Timmer]

Anyone with another idea? Maybe someone pointing me in the right direction? I'm kinda lost here :)

 

[cyberjock]

What's your server hardware and FreeNAS version? ZFS or UFS?

Can you post the script you are using?

 

[Kai Timmer]

I'm runnning FreeNAS on an ESXi 5.5 with 4 CPU Cores and 10GB RAM dedicated to the Freenas VM and I'm using ZFS.

A stripped down version of the script looks like this:

Code:

for dir in `find ${BACKUPDIR} -maxdepth 1 -mindepth 1 -type d`
do
  host=$( basename $dir )
  mount -o remount,rw $dir
 
  rsync -a -S --delete-after --fuzzy root@${host}:/simple-backup/ ${BACKUPDIR}/${host}/
 
  mount -o remount,ro $dir
done

Where every folder under $BACKUPDIR is a FQDN of a server where the folder /simple-backup/ should get synced.

 

[ser_rhaegar]

What processor do you have in your ESXi host? And is the system running the script on the same host? What specs on that?

 

[Kai Timmer]

I'm running a Dell Inc. PowerEdge R710 with 16 x Intel(R) Xeon(R) CPU E5640 @ 2.67GHz.

 

[ser_rhaegar]

Specs on the other vm running the script? Also how many VMs are running and how many have more than 1 vCPU?

 

[Kai Timmer]

The vm running the script has 1GB of RAM and 2 Cores. The vm does nothing else than running this script on a regular basis.

But, as I just noticed, is running on a different ESXi. The two ESXi are connected through a 1Gbit backend. But if you think it might help, I can just move the VM and see if anything changes.

 

[ser_rhaegar]

No don't move the vm. How many are running and the same host as FreeNAS and how many have more than 1 vCPU (especially 4 or more)? Just looking for info.

 

[ser_rhaegar]

Also what is in the log for FreeNAS?

 

[Kai Timmer]

In the attachment you can see the vm overview of the esxi host where my freenas instance is running.

 

[Kai Timmer]

From Freenas, which log do you need?

 

[ser_rhaegar]

/var/log/messages

 

[ser_rhaegar]

How many of those VMs have multiple vCPUs and how many with 4+ vCPUs?

 

[cyberjock]

I know there's serious problems with running FreeNAS in a VM up to and including the scheduling of CPU resources. The fact you are doing a ro for one directory followed by a rw for another directory in quick succession could be fubaring things.

 

[ser_rhaegar]

And post a screenshot of the command esxtop from your host please.

 

[Kai Timmer]

I know there's serious problems with running FreeNAS in a VM up to and including the scheduling of CPU resources. The fact you are doing a ro for one directory followed by a rw for another directory in quick succession could be fubaring things.

Unfortunately running on bare metal is not an option here. :(

 

[Kai Timmer]

/var/log/messages doesn't show anything interesting. The only thing I see when mounting/remounting the shares is a list of this:

Code:

May  9 23:00:01 freenas mountd[1760]: mount request succeeded from 172.16.1.37 for /mnt/backup/srv0
May  9 23:00:01 freenas mountd[1760]: mount request succeeded from 172.16.1.52 for /mnt/backup/srv1
[...]

Attached is a esxtop screenshot. I had to remove the actual vm names.

One of the vms is running with 8 cores most of the others run with 1 core and a few with 2 cores.

@cyberjock: do you think it could help if I do a complete umount and not only mount as "read-only"?