I have successfully migrate 8.6 to 10.5.2 publisher and I got error message while sub migration.
tried again to install manually sub 4 times but all the time I got same error message.
Disabled the LRO setting as recommended by TAC but no luck.
I suppose u have opened a TAC case and did changes in LRO but did not help.
Just to share with you:
Even we faced the similar issue of LRO and did the workaround but did not solve the issue.
TAC engineer added the script in PUB to fix the issue.
In existing PUB, can u share show version active
I have done the fresh install with different IP addresses both Pub/Sub.
shutdown old cluster and re-address 10.5 with previous cluster IP address.
Observed only cooperate directory not accessible issue
reason - IP address didn't change on enterprise parameter after re-addressing
After change IP it worked.
I just wanted to dump the log file and also wanted to install without shut down running cm. this is the reason to use separate IPs.
It was success.