Hi gurus,
I have a single DNA Center server that does not want to upgrade. I have googled and read so many posts now, and tried to follow all the steps, but I can't figure it out.
So my server is running version 1.1.0.659.1, and it wants to update to version 1.1.0.754. I have tried both from the GUI and the CLI, and the output is really that it can't download.
When opening the Software Updates, it says that the connection to the Cisco Software Center is OK, and also it verifies the CCO credentials.
I am using a proxy-server for access to the Internet, but I do not have anything performing SSL inspection or SSL termination of any kind.
So, lets start with the error message:
$ maglev system_updater update_info
System update status:
Version successfully installed: 1.1.0.659.1
Updater State:
Currently processed version: 1.1.0.754
State: FAILED
Sub-State: DOWNLOADED_CATALOGSERVER
Details: Downloading systemUpdatePackage catalogserver:1.1.0.754 failed
Progress: 0%
I have validated the catalog settings:
[Mon Jun 29 09:56:32 UTC] maglev@10.x.x.x (maglev-master-1) ~
$ maglev catalog settings validate
Validating catalog server settings...
Parent catalog settings are valid
I also see that the packages are needed to be downloaded:
$ maglev catalog package display
maglev-1 [main - https://kong-frontend.maglev-system.svc.cluster.local:443]
NAME DISPLAY_NAME VERSION STATE INFO
--------------------------------------------------------------------------------------------------------
access-control-application Access Control Application 2.1.114.62044 PARTIAL Package needs to be pulled/downloaded
ai-network-analytics AI Network Analytics 2.1.18.0 PARTIAL Package needs to be pulled/downloaded
..... [Shortened for readability] .....
The catalog settings look good to me, even though I read somewhere that the repo should be different. Don't know if that might be wrong, just guessing it is a newer version:
$ maglev catalog settings display
maglev-1 [main - https://kong-frontend.maglev-system.svc.cluster.local:443]
SETTING VALUE
----------------------------------------------------------------------------
defaultRepository
exposedRegistry
httpsProxy http://172.x.x.x:8181
parentCatalogServer https://www.ciscoconnectdna.com:443
parentCatalogServerRepository dnac133
I have tried to delete packages from the catalog, as mentioned in one of the release notes of DNA Center, using the following loop:
$ for pkg in $(maglev package status -o json | jq -r '.[] | select(.available!="-") | [ .name,.available | tostring ] | join (":")'); do maglev catalog package delete $pkg 2>/dev/null; doneWhen looking into the systemlogs with journalctl I do see that it tries to lookup its own internal kube cluster IP from the domain DNS server. Don't know if that is normal, or if there is something wrong somewhere....
Jun 29 11:40:09 maglev-master-1 nodeupdater-service.sh[1600]: Error: client: etcd cluster is unavailable or misconfigured
Jun 29 11:40:09 maglev-master-1 nodeupdater-service.sh[1600]: error #0: dial tcp: lookup etcd.kube-system.svc.cluster.local on 10.x.x.x:53: no such host
And finally, when showing the service logs through the maglev tool, I do get quite a lot of these types of errors, stating HTTP 404 issues for the update files:
{"asctime": "2020-06-29 13:24:17,489", "timeMillis": 1593437057.48936, "filename": "system_update_orchestrator.py", "funcName": "download_update_package_sync_retry", "levelname": "ERROR", "levelno": 40, "lineno": 298, "module": "system_update_orchestrator", "msecs": 489.36009407043457, "message": "Exception while downloading catalogserver:1.1.0.754", "name": "system-updater", "pathname": "/opt/maglev/lib/python3.5/site-packages/system_updater/system_update_orchestrator.py", "process": 22, "processName": "MainProcess", "relativeCreated": 6470324.311971664, "thread": 140360487507712, "threadName": "Thread-23", "level": "ERROR", "exc_info": "Traceback (most recent call last):\n File \"/opt/maglev/lib/python3.5/site-packages/maglev_api/proxies/AbstractProxy.py\", line 231, in _process_response\n request.raise_for_status()\n File \"/opt/maglev/lib/python3.5/site-packages/requests/models.py\", line 909, in raise_for_status\n raise HTTPError(http_error_msg, response=self)\nrequests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://catalogserver.maglev-system.svc.cluster.local:8011/api/system/v1/catalog/system-update-package/catalogserver/1.1.0.754?force=true\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/opt/maglev/lib/python3.5/site-packages/system_updater/system_update_orchestrator.py\", line 295, in download_update_package_sync_retry\n force=True)\n File \"/opt/maglev/lib/python3.5/site-packages/maglev_api/proxies/system_update_package.py\", line 42, in pull_system_update_package\n r = self._do_put(url, query_string_builder=query_string_builder)\n File \"/opt/maglev/lib/python3.5/site-packages/maglev_api/decorators.py\", line 18, in inner_wrapper\n return func(*args, **kwargs)\n File \"/opt/maglev/lib/python3.5/site-packages/maglev_api/proxies/AbstractProxy.py\", line 37, in wrapped\n return func(*args, **kwargs)\n File \"/opt/maglev/lib/python3.5/site-packages/maglev_api/proxies/AbstractProxy.py\", line 210, in _do_put\n return self._process_response(r)\n File \"/opt/maglev/lib/python3.5/site-packages/maglev_api/proxies/AbstractProxy.py\", line 246, in _process_response\n raise self._get_error(request.status_code, error_message, error_code, response)\nmaglev_api.exceptions.ResourceNotFoundError: Could not download SystemUpdatePackage catalogserver:1.1.0.754 from parent catalog. Specified SystemUpdatePackage not found in repository 'dnac133': catalogserver:1.1.0.754"}So, now I have come to the end of what I can test and verify. Anyone has any ideas on what to do next?
Best regards
::Rune
