Trouble Migrating Dataset to New ZFS Pool

[wootonius]

I'm a bit stumped and am curious if you all have any suggestions on how I might better migrate data from one ZFS pool to another. I'd like to move my ix-applications dataset from a pool named "Stockpile" to a pool named "Containers". I tried selecting a new pool and migrating applications using the UI but got the following error:

Code:

Error: [EFAULT] Full ZFS replication failed to transfer all the children of the snapshot stockpile/ix-applications@ix-app-migrate-2022-09-11_10-02. The error was: cannot receive new filesystem stream: incremental send stream requires -L (--large-block), to match previous receive. Broken pipe. The snapshot containers/ix-applications/catalogs@ix-app-migrate-2022-09-11_10-02 was not transferred. Please run `zfs destroy -r containers/ix-applications@ix-app-migrate-2022-09-11_10-02` on the target system and run replication again.

What I interpreted the above to mean is that "-L" is needed in the zfs send command but the command run by Truenas didn't include that flag. After destroying the incomplete dataset in the desired destination, I ran the following command line from the GUI in the hopes that it would correctly complete the migration:

Code:

zfs send -vRL stockpile/ix-applications@migrate-2 | zfs recv containers/ix-applications

That command initially appears to work before yielding this error:

Code:

total estimated size is 85.4G
TIME        SENT   SNAPSHOT stockpile/ix-applications/docker@auto-2022-08-29_00-00
12:57:10   15.4M   stockpile/ix-applications/docker@auto-2022-08-29_00-00
12:57:11   38.0M   stockpile/ix-applications/docker@auto-2022-08-29_00-00
12:57:12   71.3M   stockpile/ix-applications/docker@auto-2022-08-29_00-00
12:57:13    132M   stockpile/ix-applications/docker@auto-2022-08-29_00-00
12:57:14    167M   stockpile/ix-applications/docker@auto-2022-08-29_00-00
12:57:15    202M   stockpile/ix-applications/docker@auto-2022-08-29_00-00
12:57:16    237M   stockpile/ix-applications/docker@auto-2022-08-29_00-00
12:57:17    259M   stockpile/ix-applications/docker@auto-2022-08-29_00-00
TIME        SENT   SNAPSHOT stockpile/ix-applications/docker@auto-2022-09-11_00-00
12:57:26   80.2M   stockpile/ix-applications/docker@auto-2022-09-11_00-00
cannot receive new filesystem stream: incremental send stream requires -L (--large-block), to match previous receive.

Do you all have any thoughts on a better way to do this? I'm kinda stumped and haven't been able to find a solution online in the zfs documentation which matches the error I'm getting. Any pointers would be greatly appreciated!

Thank you!

 

[morganL]

I'm a bit stumped and am curious if you all have any suggestions on how I might better migrate data from one ZFS pool to another. I'd like to move my ix-applications dataset from a pool named "Stockpile" to a pool named "Containers". I tried selecting a new pool and migrating applications using the UI but got the following error:

Code:

Error: [EFAULT] Full ZFS replication failed to transfer all the children of the snapshot stockpile/ix-applications@ix-app-migrate-2022-09-11_10-02. The error was: cannot receive new filesystem stream: incremental send stream requires -L (--large-block), to match previous receive. Broken pipe. The snapshot containers/ix-applications/catalogs@ix-app-migrate-2022-09-11_10-02 was not transferred. Please run `zfs destroy -r containers/ix-applications@ix-app-migrate-2022-09-11_10-02` on the target system and run replication again.

What I interpreted the above to mean is that "-L" is needed in the zfs send command but the command run by Truenas didn't include that flag. After destroying the incomplete dataset in the desired destination, I ran the following command line from the GUI in the hopes that it would correctly complete the migration:

Code:

zfs send -vRL stockpile/ix-applications@migrate-2 | zfs recv containers/ix-applications

That command initially appears to work before yielding this error:

Code:

total estimated size is 85.4G
TIME        SENT   SNAPSHOT stockpile/ix-applications/docker@auto-2022-08-29_00-00
12:57:10   15.4M   stockpile/ix-applications/docker@auto-2022-08-29_00-00
12:57:11   38.0M   stockpile/ix-applications/docker@auto-2022-08-29_00-00
12:57:12   71.3M   stockpile/ix-applications/docker@auto-2022-08-29_00-00
12:57:13    132M   stockpile/ix-applications/docker@auto-2022-08-29_00-00
12:57:14    167M   stockpile/ix-applications/docker@auto-2022-08-29_00-00
12:57:15    202M   stockpile/ix-applications/docker@auto-2022-08-29_00-00
12:57:16    237M   stockpile/ix-applications/docker@auto-2022-08-29_00-00
12:57:17    259M   stockpile/ix-applications/docker@auto-2022-08-29_00-00
TIME        SENT   SNAPSHOT stockpile/ix-applications/docker@auto-2022-09-11_00-00
12:57:26   80.2M   stockpile/ix-applications/docker@auto-2022-09-11_00-00
cannot receive new filesystem stream: incremental send stream requires -L (--large-block), to match previous receive.

Do you all have any thoughts on a better way to do this? I'm kinda stumped and haven't been able to find a solution online in the zfs documentation which matches the error I'm getting. Any pointers would be greatly appreciated!

Thank you!

Which version are you operating on?
Did you pause the applications first?

If you could capture the specific actions you performed in the WebUI... you can report- a-bug if no-one can see an issue with what you have done.

 

[wootonius]

Which version are you operating on?
Did you pause the applications first?

If you could capture the specific actions you performed in the WebUI... you can report- a-bug if no-one can see an issue with what you have done.

To answer your questions:
Which version are you operating on? TrueNAS-SCALE-22.02.3 (I should've provided this in my original post)
Did you pause the applications first? I did

As for the steps I followed, here they are in order:

1. I stopped all applications
2. I went to the settings menu and clicked "Choose Pool" - By the way, I can only get this menu option to work on the "Available Applications" tab and not the "Installed Applications" tab even though it "Choose Pool" is also visible in the Settings menu on the "Installed Application" tab
3. I chose the desired pool, clicked "Migrate applications to the new pool" and clicked the "Choose" button
4. The process begins and when I check back on it in the "Task Manager" view I get the below error (same text as above)

 

[Arwen]

Could you supply the output of the following commands?

zpool get all | grep large_blocks

It's possible that you are using the large_blocks feature in Stockpile but not Containers

 

[wootonius]

Could you supply the output of the following commands?

zpool get all | grep large_blocks

It's possible that you are using the large_blocks feature in Stockpile but not Containers

From what I can tell it is in use on both

Code:

boot-pool   feature@large_blocks                    enabled                                 local

containers  feature@large_blocks                    active                                  local

stockpile   feature@large_blocks                    active                                  local

 

[Arwen]

Okay, that was a shot in the dark. Sorry, I don't have any other suggestions.

 

[wootonius]

Okay, that was a shot in the dark. Sorry, I don't have any other suggestions.

No worries, I appreciate the suggestion

 

[morganL]

To answer your questions:
Which version are you operating on? TrueNAS-SCALE-22.02.3 (I should've provided this in my original post)
Did you pause the applications first? I did

As for the steps I followed, here they are in order:

1. I stopped all applications
2. I went to the settings menu and clicked "Choose Pool" - By the way, I can only get this menu option to work on the "Available Applications" tab and not the "Installed Applications" tab even though it "Choose Pool" is also visible in the Settings menu on the "Installed Application" tab
3. I chose the desired pool, clicked "Migrate applications to the new pool" and clicked the "Choose" button
4. The process begins and when I check back on it in the "Task Manager" view I get the below error (same text as above)

View attachment 58362

This is enough information to report-a-bug.... this process is probably not a normal test case and there may be an issue.

Thanks in advance for reporting and assisting the engineering team to replicate. Please report the bug ticket ID.

 

[wootonius]

This is enough information to report-a-bug.... this process is probably not a normal test case and there may be an issue.

Thanks in advance for reporting and assisting the engineering team to replicate. Please report the bug ticket ID.

Done, logged bug NAS-118126. Thank you!

 

[kevinmatt]

Done, logged bug NAS-118126. Thank you!

Sorry but I just can't see NAS-118126 on jira, is it visible? Did it get fixed?

 

[morganL]

NAS-118126 is Not resolved yet. May be a ZFS issue.