CSCvh98912 - Git734 DNAC upgrade fails w/ sftp-service not found and pod updates may not add or remove containers

[mifi]

Hi guys,

I struggled several days with upgrade of the DNA appliance from 1.1 to 1.1.2, but in the end I found a working workaround for this bug.

From the CLI I could see the system was failing upgrading:

$ maglev package status

maglev-1 [main - https://kong-frontend.maglev-system.svc.cluster.local:443]

NAME DEPLOYED AVAILABLE STATUS
-----------------------------------------------------------------------------------
application-policy - 2.1.4.170015 NOT_DEPLOYED
assurance 1.0.5.503 1.0.5.630 DEPLOYED
automation-core 2.1.1.60067 2.1.4.60025 DEPLOYED
base-provision-core - 2.1.4.60025 NOT_DEPLOYED
command-runner - 2.1.4.60025 NOT_DEPLOYED
device-onboarding - 2.1.4.60025 NOT_DEPLOYED
image-management 2.1.1.60067 2.1.4.60025 DEPLOYED
ncp-system 2.1.1.60067 2.1.4.60025 DEPLOYED
ndp-base-analytics 1.0.6.342 1.0.7.871 DEPLOYED
ndp-platform 1.0.6.246 1.0.7.803 DEPLOYED
ndp-ui 1.0.6.454 1.0.7.949 DEPLOYED
network-visibility 2.1.1.60067 2.1.4.60025 DEPLOYED
path-trace 2.1.1.60067 2.1.4.60025 DEPLOYED
sd-access - 2.1.4.60025 NOT_DEPLOYED
sensor-assurance - 1.0.5.326 NOT_DEPLOYED
sensor-automation - 2.1.4.60025 NOT_DEPLOYED
system 1.0.4.776 - UPGRADE_ERROR - maglev_workflow.workflow.exceptions.TaskCallableExecutionError: (1520330534.0461926, 1520330624.4249065, 'TimeoutError', 'Timeout of 90 seconds has expired while watching for k8s changes for telemetry-agent ')

I looked which parts were failing:

$ magctl appstack status | grep -v Run
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE
maglev-system agent-6kz9k 0/1 ImagePullBackOff 0 1m <none> 10.22.30.10
maglev-system remedycontroller-419825855-cb5sn 0/1 ImagePullBackOff 0 9m 10.22.27.69 10.22.30.10
maglev-system workflow-ui-2875804863-dxn68 0/1 ImagePullBackOff 0 9m 10.22.27.120 10.22.30.10

 I decided to try to pull the failing packages manually one by one.

$ maglev catalog system_update_addon pull image-remedy-controller-1-0-4-776
SystemUpdateAddon pull initiated

When all packages were in the Running state I upgraded the system again:

$ maglev package upgrade system
Package will start getting upgraded momentarily

I had to repeat those steps several times but in the end I was up and running:

$ maglev package status

maglev-1 [main - https://kong-frontend.maglev-system.svc.cluster.local:443]

NAME DEPLOYED AVAILABLE STATUS
-----------------------------------------------------------------------------------
application-policy - 2.1.4.170015 NOT_DEPLOYED
assurance 1.0.5.503 1.0.5.630 DEPLOYED
automation-core 2.1.1.60067 2.1.4.60025 DEPLOYED
base-provision-core - 2.1.4.60025 NOT_DEPLOYED
command-runner - 2.1.4.60025 NOT_DEPLOYED
device-onboarding - 2.1.4.60025 NOT_DEPLOYED
image-management 2.1.1.60067 2.1.4.60025 DEPLOYED
ncp-system 2.1.1.60067 2.1.4.60025 DEPLOYED
ndp-base-analytics 1.0.6.342 1.0.7.871 DEPLOYED
ndp-platform 1.0.6.246 1.0.7.803 DEPLOYED
ndp-ui 1.0.6.454 1.0.7.949 DEPLOYED
network-visibility 2.1.1.60067 2.1.4.60025 DEPLOYED
path-trace 2.1.1.60067 2.1.4.60025 DEPLOYED
sd-access - 2.1.4.60025 NOT_DEPLOYED
sensor-assurance - 1.0.5.326 NOT_DEPLOYED
sensor-automation - 2.1.4.60025 NOT_DEPLOYED
system 1.0.4.776 - DEPLOYED

I hope this guide will keep you from having to reimage the server from the ISO.

Regards

Michal

[manob]

Thanks mifi, it really works.

Regards

Martin

[atsaya2512]

Hi All,

We are trying to install DNAC 1.1.8, we are facing issue in deploying automation-core version 2.1.12.60011. Automation-core throws an error stating "DEPLOYMENT_ERROR - maglev.workflow.workflow.exceptions.TaskCallableExecutionError: (1547186922.4240553, 1547191113.6825242, 'TimeoutError' , 'Maximum wait time 4170 exceeded for the following services to be ready: task-service, telemetry-service, distributed -cache-service, file-service')". We even tried to pull each package from the catalog which failed by giving "maglev catalog system_update_addon pull image telemetry-service:2.1.12.60011", that too throws error saying "An unexpected error occurred".

I have attached the screenshots for your reference, it would be great if anyone can give us some insight about the issue.

Regards

Akchaiah

[de1denta]

Hi,

Did you ever solve the issue with the automation-core package not deploying correctly as I have the same issue?

Thanks

[atsaya2512]

No we were not able to solve that issue