Unable to migrate instance from encrypted CN

[blackwood821]

[root@headnode (us-west-agc) ~]# sdc-migrate migrate -n 44454c4c-5100-1059-8057-b9c04f4e5832 3d583590-c8ff-cd0d-e0b4-c3e439208fe6
# Migration begin running in job 93270a0c-4700-436c-84d0-493b70fe0ed1
 - reserving instance
 - syncing data
Failure: failed to sync migration instance - Error - Error: sync error: zfs snapshot error: Command failed: /usr/sbin/zfs send --parsable --dryrun --replicate zones/3d583590-c8ff-cd0d-e0b4-c3e439208fe6@vm-migration-1

Here is what I see on the source CN:

[root@cn04 (us-west-agc) /opt]# zfs list -t all -r zones/3d583590-c8ff-cd0d-e0b4-c3e439208fe6
NAME                                                             USED  AVAIL     REFER  MOUNTPOINT
zones/3d583590-c8ff-cd0d-e0b4-c3e439208fe6                       416M  99.6G     64.9M  /zones/3d583590-c8ff-cd0d-e0b4-c3e439208fe6
zones/3d583590-c8ff-cd0d-e0b4-c3e439208fe6@vm-migration-1           0      -     64.9M  -
zones/3d583590-c8ff-cd0d-e0b4-c3e439208fe6/data                  397M  99.6G      397M  /zones/3d583590-c8ff-cd0d-e0b4-c3e439208fe6/data
zones/3d583590-c8ff-cd0d-e0b4-c3e439208fe6/data@vm-migration-1      0      -      397M  -

And here is the underlying error:

[root@cn04 (us-west-agc) /opt]# /usr/sbin/zfs send --parsable --dryrun --replicate zones/3d583590-c8ff-cd0d-e0b4-c3e439208fe6@vm-migration-1 > 3d583590-c8ff-cd0d-e0b4-c3e439208fe6@vm-migration-1.zfs
cannot send zones/3d583590-c8ff-cd0d-e0b4-c3e439208fe6@vm-migration-1: encrypted dataset zones/3d583590-c8ff-cd0d-e0b4-c3e439208fe6 may not be sent with properties without the raw flag

Code of interest may be https://github.com/TritonDataCenter/sdc-cn-agent/blob/master/lib/backends/smartos/bin/machine-migrate-send.js#L945

[blackwood821]

It appears this has been an issue for over 2 years and possibly ever since the beginning so it's nothing new:
TritonDataCenter/sdc-cn-agent#37 (comment)

[blackwood821]

Since this has been a critical issue for us because we have many encrypted CNs, I have implemented my own version of sdc-migrate (https://github.com/vrcis/triton/blob/main/vrc-migrate) that works around this limitation. The main difference is that it does not use the -R zfs send argument and instead iterates over the datasets and handles the properties explicitly. I verified that the zfs clone aspect of the dataset is preserved by verifying files the zone root inherits from the origin image are still hard linked and have the same inode as the corresponding file in the image file system. This script is obviously not as ideal as sdc-migrate since it doesn't call any of the Triton APIs and instead does everything locally on the CN globals but it is working very well for us now. Maybe Triton will incorporate something similar into sdc-migrate one day.