Buy or Renew
Buy or Renew
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
cancel
Start a conversation
11594
Views
15
Helpful
4
Replies

DNA Center Update Failed

jpdeboer1
Level 1
Level 1

‎09-27-2019 01:10 AM

Hello,

 

We are currently building a test setup with DNA Center and a couple of routers en switches. Now i was in the process of updating the DNA center with latest software step by step. But now one update failed and i have no longer access to the web interface of DNA Center. I still do have access to the maglev CLI on the same ip address via SSH.

 

The update was stuck for a long time at below:

 

$ sudo maglev system_updater update_info

System update status:
  Version successfully installed : 1.1.0.784
  Version currently processed    : 1.2.0.1013
    Update phase                 : Updating the host components
    Update details               : Updating the node 10.0.32.1 [phase 4]
    Progress                     : 48%

  Updater State:
    Currently processed version  : 1.2.0.1013
    State                        : INSTALLING_UPDATES
    Sub-State                    : INSTALLED_PREINSTALL_CLUSTER_UPDATES
    Details                      : Updating the host components
    Source                       : system-updater

Then i saw below:

$ maglev catalog package display

maglev-1 [main - https://kong-frontend.maglev-system.svc.cluster.local:443]

NAME                      DISPLAY_NAME                             VERSION         STATE      INFO
--------------------------------------------------------------------------------------------------------
application-policy        Application Policy                       2.1.29.170002   PARTIAL    Package needs to be pulled/downloaded
assurance                 Assurance - Base                         1.2.12.125      PARTIAL    Package needs to be pulled/downloaded
automation-core           NCP - Services                           2.1.31.60019    PARTIAL    Package needs to be pulled/downloaded
base-provision-core       Automation - Base                        2.1.31.60019    PARTIAL    Package needs to be pulled/downloaded
command-runner            Command Runner                           2.1.31.60019    PARTIAL    Package needs to be pulled/downloaded
device-onboarding         Device Onboarding                        2.1.31.60019.2  PARTIAL    Package needs to be pulled/downloaded
dnac-platform             DNAC Platform                            1.0.12.4        PARTIAL    Package needs to be pulled/downloaded
icap-automation           Automation - Intelligent Capture         2.1.31.60019    PARTIAL    Package needs to be pulled/downloaded
image-management          Image Management                         2.1.31.60019    PARTIAL    Package needs to be pulled/downloaded
ncp-system                NCP - Base                               2.1.31.60019    PARTIAL    Package needs to be pulled/downloaded
ndp-base-analytics        Network Data Platform - Base Analytics   1.1.11.15       PARTIAL    Package needs to be pulled/downloaded
ndp-platform              Network Data Platform - Core             1.1.11.121      PARTIAL    Package needs to be pulled/downloaded
ndp-ui                    Network Data Platform - Manager          1.1.11.12       PARTIAL    Package needs to be pulled/downloaded
network-visibility        Network Controller Platform              2.1.31.60019.2  PARTIAL    Package needs to be pulled/downloaded
path-trace                Path Trace                               2.1.31.60019    PARTIAL    Package needs to be pulled/downloaded
platform-ui               Cisco DNA Center UI                      1.3.1.248       READY
sd-access                 SD Access                                2.1.31.60019.2  PARTIAL    Package needs to be pulled/downloaded
sensor-assurance          Assurance - Sensor                       1.2.12.124      PARTIAL    Package needs to be pulled/downloaded
sensor-automation         Automation - Sensor                      2.1.31.60019    PARTIAL    Package needs to be pulled/downloaded
system                    System                                   1.2.0.1013      READY

when i check the status of one of the package:

$ maglev catalog package status application-policy:2.1.29.170002

maglev-1 [main - https://kong-frontend.maglev-system.svc.cluster.local:443]

KIND                 RESOURCE                                      STATE      MESSAGE
----------------------------------------------------------------------------------------------------------------------
Package              application-policy:2.1.29.170002              PARTIAL    One or more child resources are not available yet
ServiceBundleGroup   fusion/application-policy:2.1.29.170002       NOT_FOUND  Manifest not available yet
ServiceBundleGroup   maglev-system/application-policy-ui:2.1.29.170002 NOT_FOUND  Manifest not available yet

So i tried to delete them till they were at ready state:

$ maglev catalog package delete assurance:1.2.12.125
ok

After all was deleted

$ maglev catalog package display

maglev-1 [main - https://kong-frontend.maglev-system.svc.cluster.local:443]

NAME                      DISPLAY_NAME                             VERSION         STATE      INFO
--------------------------------------------------------------------------------------------------------
application-policy        Application Policy                       2.1.28.170011   READY
assurance                 Assurance - Base                         1.2.11.304      READY
automation-core           NCP - Services                           2.1.28.60244.9  READY
base-provision-core       Automation - Base                        2.1.28.60244.9  READY
command-runner            Command Runner                           2.1.28.60244    READY
device-onboarding         Device Onboarding                        2.1.28.60244    READY
dnac-platform             DNAC Platform                            1.0.8.8         READY
icap-automation           Automation - Intelligent Capture         2.1.28.60244    READY
image-management          Image Management                         2.1.28.60244    READY
ncp-system                NCP - Base                               2.1.28.60244    READY
ndp-base-analytics        Network Data Platform - Base Analytics   1.1.11.8        READY
ndp-platform              Network Data Platform - Core             1.1.11.77       READY
ndp-ui                    Network Data Platform - Manager          1.1.11.8        READY
network-visibility        Network Controller Platform              2.1.28.60244.9  READY
path-trace                Path Trace                               2.1.28.60244    READY
platform-ui               Cisco DNA Center UI                      1.3.1.248       READY
sd-access                 SD Access                                2.1.28.60244.9  READY
sensor-assurance          Assurance - Sensor                       1.2.10.254      READY
sensor-automation         Automation - Sensor                      2.1.28.60244    READY
system                    System                                   1.2.0.1013      READY

But about 3 / 4 hours later the update failed.

System update status:
  Version successfully installed : 1.1.0.784
  Version currently processed    : 1.2.0.1013
    Update phase                 : failed
    Update details               : Node update took longer to complete in 10.0.32.1 [ version 1.2.0.1013 phase 4 ]
    Progress                     : 48%

  Updater State:
    Currently processed version  : 1.2.0.1013
    State                        : FAILED
    Sub-State                    : INSTALLED_HOST_COMPONENTS
    Details                      : Node update took longer to complete in 10.0.32.1 [ version 1.2.0.1013 phase 4 ]
    Source                       : system-updater

I then tried to force it back to an older version.

$ maglev system_updater update_system 1.1.0.784 -f
System update download initiated

But also failed later

$ sudo maglev system_updater update_info

System update status:
  Version successfully installed : 1.1.0.784
  Version currently processed    : 1.1.0.784
    Update phase                 : failed
    Update details               : Updating node 10.0.32.1 failed
    Progress                     : 52%

  Updater State:
    Currently processed version  : 1.1.0.784
    State                        : FAILED
    Sub-State                    : INSTALLED_HOST_COMPONENTS
    Details                      : Updating node 10.0.32.1 failed
    Source                       : system-updater

Eventhough all packages are deployed

$  maglev package status

maglev-1 [main - https://kong-frontend.maglev-system.svc.cluster.local:443]

NAME                      DISPLAY_NAME                             DEPLOYED        AVAILABLE       STATUS                    PROGRESS
---------------------------------------------------------------------------------------------------------------------------------------
application-policy        Application Policy                       2.1.28.170011   2.1.42.170001   DEPLOYED
assurance                 Assurance - Base                         1.2.11.304      1.3.1.140       DEPLOYED
automation-core           NCP - Services                           2.1.28.60244.9  2.1.43.60042    DEPLOYED
base-provision-core       Automation - Base                        2.1.28.60244.9  2.1.43.60042    DEPLOYED
command-runner            Command Runner                           2.1.28.60244    2.1.43.60042    DEPLOYED
device-onboarding         Device Onboarding                        2.1.28.60244    2.1.43.60042    DEPLOYED
dnac-platform             DNAC Platform                            1.0.8.8         1.1.1.2         DEPLOYED
icap-automation           Automation - Intelligent Capture         2.1.28.60244    2.1.43.60042    DEPLOYED
image-management          Image Management                         2.1.28.60244    2.1.43.60042    DEPLOYED
ncp-system                NCP - Base                               2.1.28.60244    2.1.43.60042    DEPLOYED
ndp-base-analytics        Network Data Platform - Base Analytics   1.1.11.8        1.3.1.204       DEPLOYED
ndp-platform              Network Data Platform - Core             1.1.11.77       1.3.1.615       DEPLOYED
ndp-ui                    Network Data Platform - Manager          1.1.11.8        1.3.1.185       DEPLOYED
network-visibility        Network Controller Platform              2.1.28.60244.9  2.1.43.60046    DEPLOYED
path-trace                Path Trace                               2.1.28.60244    2.1.43.60042    DEPLOYED
platform-ui               Cisco DNA Center UI                      1.3.1.248       1.3.1.250       DEPLOYED
sd-access                 SD Access                                2.1.28.60244.9  2.1.43.60045    DEPLOYED
sensor-assurance          Assurance - Sensor                       1.2.10.254      1.3.0.1313      DEPLOYED
sensor-automation         Automation - Sensor                      2.1.28.60244    2.1.43.60042    DEPLOYED
system                    System                                   1.1.0.784       1.2.0.1013      DEPLOYED

State of the packages is ready

$  maglev catalog package display

maglev-1 [main - https://kong-frontend.maglev-system.svc.cluster.local:443]

NAME                      DISPLAY_NAME                             VERSION         STATE      INFO
--------------------------------------------------------------------------------------------------------
application-policy        Application Policy                       2.1.28.170011   READY
assurance                 Assurance - Base                         1.2.11.304      READY
automation-core           NCP - Services                           2.1.28.60244.9  READY
base-provision-core       Automation - Base                        2.1.28.60244.9  READY
command-runner            Command Runner                           2.1.28.60244    READY
device-onboarding         Device Onboarding                        2.1.28.60244    READY
dnac-platform             DNAC Platform                            1.0.8.8         READY
icap-automation           Automation - Intelligent Capture         2.1.28.60244    READY
image-management          Image Management                         2.1.28.60244    READY
ncp-system                NCP - Base                               2.1.28.60244    READY
ndp-base-analytics        Network Data Platform - Base Analytics   1.1.11.8        READY
ndp-platform              Network Data Platform - Core             1.1.11.77       READY
ndp-ui                    Network Data Platform - Manager          1.1.11.8        READY
network-visibility        Network Controller Platform              2.1.28.60244.9  READY
path-trace                Path Trace                               2.1.28.60244    READY
platform-ui               Cisco DNA Center UI                      1.3.1.248       READY
sd-access                 SD Access                                2.1.28.60244.9  READY
sensor-assurance          Assurance - Sensor                       1.2.10.254      READY
sensor-automation         Automation - Sensor                      2.1.28.60244    READY
system                    System                                   1.2.0.1013      READY

Now my question is if there is something else i can try to restore the DNAC via maglev? I did download the DNAC ISO from cisco website, but could not find the instructions on how to do a re-installation of the DNAC software.

 

Thanks in advance!

3 people had this problem
4 Replies 4

jpdeboer1
Level 1
Level 1

‎09-27-2019 02:14 AM

Ok, i seem to have access again after disabling maintenance mode via maglev:

$ maglev maintenance disable
Disable cluster maintenance mode.

now it seems that the server is trying to restore itself, ill give it a weekend and will see what state it is in Monday.

Automation and Assurance services have been temporarily disrupted. The system is working to restore this functionality.
More Info

vijha
Cisco Employee
Cisco Employee
In response to jpdeboer1

‎11-20-2019 07:53 AM

Hi,

Did your issue get resolved after you disabled the maintenance mode?

0 Helpful

jpdeboer1
Level 1
Level 1
In response to vijha

‎11-21-2019 12:29 AM

Hi,

 

After disabling maintenance mode i was able to access the web interface again, but still the system was not working properly and did not restore. I had to reinstall the software on the appliance. 

0 Helpful

Jacob Zartmann
Level 1
Level 1
In response to jpdeboer1

‎11-21-2019 01:13 PM

Hi,

 

Did you not wait the 90 seconds they say you must in the upgrade guide?

 

https://www.cisco.com/c/en/us/td/docs/cloud-systems-management/network-automation-and-management/dna-center/upgrade/b_cisco_dna_center_upgrade_guide/b_cisco_dna_center_upgrade_guide_w_parts_2_chapter_01.html#task_uyz_24r_q3b

 

It doesn't say anywhere how to fix this if the user doesn't wait :(

 

I have the similar upgrade issue.

Customers Also Viewed These Support Documents

Review Cisco Networking for a $25 gift card