After a reboot of the server hosting our CSMS installation the application is now failing to start.
The systemd service csm-supervisor.service, which in turn attempts to start a docker container, is attempting to fetch components from Cisco's docker repository at devhub-docker.cisco.com, but sadly this fails with the following errors:
Jan 16 09:49:55 <hostname redacted> csm-supervisor[267908]: 23-01-12 08:49:55 ERROR (SyncWorker_3) [csmcloud.docker.interface] Can't authenticate devhub-docker.cisco.com -> 500 Server Error: Internal Server Error ("Get "https://devhub-docker.cisco.com/v2/": unknown: Bad props auth token: apiKey=AKCp5bBCwQFwXvfnC83HVsJVM77qedNm5m9bSsMGjhEVVFxtzjFuJa4us8ji6R8FUJF99yMrK").
Jan 16 09:49:55 <hostname redacted> csm-supervisor[267908]: 23-01-12 08:49:55 WARNING (MainThread) [csmcloud.plugins.cli] Error on install CLI plugin. Retry in 30sec
Jan 16 09:50:25 <hostname redacted> csm-supervisor[267908]: 23-01-12 08:50:25 ERROR (SyncWorker_6) [csmcloud.docker.interface] Can't authenticate devhub-docker.cisco.com -> 500 Server Error: Internal Server Error ("Get "https://devhub-docker.cisco.com/v2/": unknown: This request is blocked due to recurrent login failures, please try again in 2 seconds").
Jan 16 09:50:25 <hostname redacted> csm-supervisor[267908]: 23-01-12 08:50:25 WARNING (MainThread) [csmcloud.plugins.cli] Error on install CLI plugin. Retry in 30sec
now if you attempt to pull the image manually you get a similar error:
$ docker pull devhub-docker.cisco.com/csm-docker/amd64-csm-supervisor
Using default tag: latest
Error response from daemon: Head "https://devhub-docker.cisco.com/v2/csm-docker/amd64-csm-supervisor/manifests/latest": unknown: This request is blocked due to recurrent login failures, please try again in 4 seconds
After browsing the web a little bit, it seems like other projects hosted at devhub-docker.cisco.com no longer need authentication, and this seems to be correct, however it seems like the csm-docker project is completely missing from the repository:
$ docker logout devhub-docker.cisco.com
Removing login credentials for devhub-docker.cisco.com
$ docker pull devhub-docker.cisco.com/csm-docker/amd64-csm-supervisor
Using default tag: latest
Error response from daemon: manifest for devhub-docker.cisco.com/csm-docker/amd64-csm-supervisor:latest not found: manifest unknown: The named manifest is not known to the registry.
Furthermore, if one attempts to follow the official documentation for CSMS installation, you also get a rather peculiar result after executing:
curl -Ls https://devhub.cisco.com/artifactory/software-manager-install-group/install.sh -O
You end up with a file containing:
{
"errors" : [ {
"status" : 401,
"message" : "Unauthorized"
} ]
}
I have already tried contacting TAC regarding this issue, they basically just gave me the message, that CSM is not supported by TAC, however it is no longer functioning at all, it seems like Cisco should either stop promoting the tool or find a better solution for customers who are actually using it.
Without this tool, managing the software install base of our 600+ IOS-XR devices becomes a real chore.
I hope this post can get some attention from someone who can fix this docker repository and make the software function again.
