03-08-2018 04:17 AM - edited 03-20-2019 09:58 PM
Hi guys,
I struggled several days with upgrade of the DNA appliance from 1.1 to 1.1.2, but in the end I found a working workaround for this bug.
From the CLI I could see the system was failing upgrading:
$ maglev package status
maglev-1 [main - https://kong-frontend.maglev-system.svc.cluster.local:443]
NAME DEPLOYED AVAILABLE STATUS
-----------------------------------------------------------------------------------
application-policy - 2.1.4.170015 NOT_DEPLOYED
assurance 1.0.5.503 1.0.5.630 DEPLOYED
automation-core 2.1.1.60067 2.1.4.60025 DEPLOYED
base-provision-core - 2.1.4.60025 NOT_DEPLOYED
command-runner - 2.1.4.60025 NOT_DEPLOYED
device-onboarding - 2.1.4.60025 NOT_DEPLOYED
image-management 2.1.1.60067 2.1.4.60025 DEPLOYED
ncp-system 2.1.1.60067 2.1.4.60025 DEPLOYED
ndp-base-analytics 1.0.6.342 1.0.7.871 DEPLOYED
ndp-platform 1.0.6.246 1.0.7.803 DEPLOYED
ndp-ui 1.0.6.454 1.0.7.949 DEPLOYED
network-visibility 2.1.1.60067 2.1.4.60025 DEPLOYED
path-trace 2.1.1.60067 2.1.4.60025 DEPLOYED
sd-access - 2.1.4.60025 NOT_DEPLOYED
sensor-assurance - 1.0.5.326 NOT_DEPLOYED
sensor-automation - 2.1.4.60025 NOT_DEPLOYED
system 1.0.4.776 - UPGRADE_ERROR - maglev_workflow.workflow.exceptions.TaskCallableExecutionError: (1520330534.0461926, 1520330624.4249065, 'TimeoutError', 'Timeout of 90 seconds has expired while watching for k8s changes for telemetry-agent ')
I looked which parts were failing:
$ magctl appstack status | grep -v Run
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE
maglev-system agent-6kz9k 0/1 ImagePullBackOff 0 1m <none> 10.22.30.10
maglev-system remedycontroller-419825855-cb5sn 0/1 ImagePullBackOff 0 9m 10.22.27.69 10.22.30.10
maglev-system workflow-ui-2875804863-dxn68 0/1 ImagePullBackOff 0 9m 10.22.27.120 10.22.30.10
I decided to try to pull the failing packages manually one by one.
$ maglev catalog system_update_addon pull image-remedy-controller-1-0-4-776
SystemUpdateAddon pull initiated
When all packages were in the Running state I upgraded the system again:
$ maglev package upgrade system
Package will start getting upgraded momentarily
I had to repeat those steps several times but in the end I was up and running:
$ maglev package status
maglev-1 [main - https://kong-frontend.maglev-system.svc.cluster.local:443]
NAME DEPLOYED AVAILABLE STATUS
-----------------------------------------------------------------------------------
application-policy - 2.1.4.170015 NOT_DEPLOYED
assurance 1.0.5.503 1.0.5.630 DEPLOYED
automation-core 2.1.1.60067 2.1.4.60025 DEPLOYED
base-provision-core - 2.1.4.60025 NOT_DEPLOYED
command-runner - 2.1.4.60025 NOT_DEPLOYED
device-onboarding - 2.1.4.60025 NOT_DEPLOYED
image-management 2.1.1.60067 2.1.4.60025 DEPLOYED
ncp-system 2.1.1.60067 2.1.4.60025 DEPLOYED
ndp-base-analytics 1.0.6.342 1.0.7.871 DEPLOYED
ndp-platform 1.0.6.246 1.0.7.803 DEPLOYED
ndp-ui 1.0.6.454 1.0.7.949 DEPLOYED
network-visibility 2.1.1.60067 2.1.4.60025 DEPLOYED
path-trace 2.1.1.60067 2.1.4.60025 DEPLOYED
sd-access - 2.1.4.60025 NOT_DEPLOYED
sensor-assurance - 1.0.5.326 NOT_DEPLOYED
sensor-automation - 2.1.4.60025 NOT_DEPLOYED
system 1.0.4.776 - DEPLOYED
I hope this guide will keep you from having to reimage the server from the ISO.
Regards
Michal
03-08-2018 12:34 PM
Thanks mifi, it really works.
Regards
Martin
01-21-2019 01:25 AM
Hi All,
We are trying to install DNAC 1.1.8, we are facing issue in deploying automation-core version 2.1.12.60011. Automation-core throws an error stating "DEPLOYMENT_ERROR - maglev.workflow.workflow.exceptions.TaskCallableExecutionError: (1547186922.4240553, 1547191113.6825242, 'TimeoutError' , 'Maximum wait time 4170 exceeded for the following services to be ready: task-service, telemetry-service, distributed -cache-service, file-service')". We even tried to pull each package from the catalog which failed by giving "maglev catalog system_update_addon pull image telemetry-service:2.1.12.60011", that too throws error saying "An unexpected error occurred".
I have attached the screenshots for your reference, it would be great if anyone can give us some insight about the issue.
Regards
Akchaiah
04-02-2019 11:38 PM
Hi,
Did you ever solve the issue with the automation-core package not deploying correctly as I have the same issue?
Thanks
04-08-2019 09:16 PM
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide