I did this.. when i went to import my volumes, "failed to decrypt" ( I usually have a 50% success rate re-importing after a change so I back up everything - no data loss)
stuck the USB stick back in, all shares/volumes/users look identical, but using same recovery key/passphrase, it mounts fine.
I have what appears to be an identical config on 2 mirrored X-25 SSD's.. but on my original deployment on the USB it works.. on the supposedly identical one migrated to the SSD's, it doesn't.
If i try to import volume using the raw devices, it does the Step 3: choose volume to import with the blank space in it deal, and also fails to import.
I also removed the 2 SSD's from the bays, it may have changed the names of the devices on the other disks, da0-da3 => dao-da1, if this is used in the encryption hash it may throw things off.. just a theory.
Typically I'd expect a change like this to be difficult and would just erase all the volumes and re-create them on a new install, but it would take a while to migrate all the data back on, so would be cool to get this backup config method to work.