Replication failing after 9.2.1.5 to 9.2.1.9 upgrade

[nOw2]

Hi,

I've upgraded a pair of FreeNAS servers from 9.2.1.5 to 9.2.1.9.
One server is a 'live' machine serving ZVOLs over iSCSI to VMWare, called 'source-nas'.
The second server is replication only, called 'destination-nas'.

With both on 9.2.1.5 the replication was working okay.

'destination-nas' was upgraded last week, and replication still worked okay.

'source-nas' was upgraded yesterday and now replication is failing with the following error:

CRITICAL: Replication Source_Volume -> destination-nas.lan failed: cannot mount 'backups/source-nas/Source_Volume/.system/rrd-408aad2948aa4c99bb7d8466d281bcf7': File name too long cannot mount 'backups/source-nas/Source_Volume/.system/syslog-408aad2948aa4c99bb7d8466d281bcf7': File name too long

I have tried:

• Initialising the remote side in the replication - same error re-occurs.
• Switching off the System Dataset checkboxes in settings (though didn't reboot). No change.

Worthy of note is that there are two volumes/pools on the source NAS, and only the one set as the System Dataset pool has this error. Also, this is not an isolated problem - I actual have several pairs of FreeNAS servers and this has occurred on both of the two pairs that have been upgraded.

Is this a known problem? Is there a workaround or fix?

 

[cyberjock]

Can you post a screenshot of the window with the replication settings?

 

[nOw2]

Attached.

I had attempted some redacting above so:
Source = internal-nas-a-4.
Destination = internal-nas-a-5.

 

[cyberjock]

Can you post the output of "zfs list" and "zfs list -t snapshot -o name" on the source and destination system?

 

[nOw2]

I've had to redact to protect the names of the innocent.. but hopefully everything needed is shown.

Where you see "ZVOLs-FOR-iSCSI" assume that multiple ZVOLs are present.

Perhaps an important note: these machine are used only for iSCSI so host only ZVOLs (apart from a jail to run a Nagios monitoring agent).

Attached.

 

[cyberjock]

So your redacting was actually the info we needed. :P

So ZFS is limited to a path of 256 characters. A filename is limited to 4096 characters.

It just so happens that on your original server the path happens to be less than 256 characters, but the destination path is probably >256 characters, so that fails.

You have two options, both involve not snapshotting the .system dataset.

Option A: Change your snapshot task(s) to not include the .system data. Presumably, if you were snapshotting the whole pool with recursion you'd have to created multiple snapshot tasks that handle each dataset independently. You will probably have to destroy the snapshots yourself if you go with this route, on both sides.

Option B: If you upgrade to the latest 9.3 that came out on April 10th, there is a new snapshot setting. This setting is a checkbox to exclude the .system dataset. Check that checkbox and it should start working fine, assuming you don't have another path that goes over 256 characters. This will delete the snapshots for the .system dataset automatically when you disable the .system dataset snapshotting.

Hope this helps. Please let us know if that is not the problem and I will probably ask for a non-redacted debug file, but you can PM that if we need it.

 

[nOw2]

So your redacting was actually the info we needed. :p

Always the risk.. Sorry for that.

Your suggestion was correct and indeed the length of the filename on the destination was the problem.

I implemented a third option - I reduced the length of the name of the subfolder on the destination. This allowed replication to complete successfully. There will be no filenames longer than the syslog/rrd directories so this is a permanent solution for us.

Thanks for your help!