cancel
Showing results for 
Search instead for 
Did you mean: 
cancel

Who Me Too'd this topic

DNA Center Update Failed

jpdeboer1
Level 1
Level 1

Hello,

 

We are currently building a test setup with DNA Center and a couple of routers en switches. Now i was in the process of updating the DNA center with latest software step by step. But now one update failed and i have no longer access to the web interface of DNA Center. I still do have access to the maglev CLI on the same ip address via SSH.

 

The update was stuck for a long time at below:

 

$ sudo maglev system_updater update_info

System update status:
  Version successfully installed : 1.1.0.784
  Version currently processed    : 1.2.0.1013
    Update phase                 : Updating the host components
    Update details               : Updating the node 10.0.32.1 [phase 4]
    Progress                     : 48%

  Updater State:
    Currently processed version  : 1.2.0.1013
    State                        : INSTALLING_UPDATES
    Sub-State                    : INSTALLED_PREINSTALL_CLUSTER_UPDATES
    Details                      : Updating the host components
    Source                       : system-updater

Then i saw below:

$ maglev catalog package display

maglev-1 [main - https://kong-frontend.maglev-system.svc.cluster.local:443]

NAME                      DISPLAY_NAME                             VERSION         STATE      INFO
--------------------------------------------------------------------------------------------------------
application-policy        Application Policy                       2.1.29.170002   PARTIAL    Package needs to be pulled/downloaded
assurance                 Assurance - Base                         1.2.12.125      PARTIAL    Package needs to be pulled/downloaded
automation-core           NCP - Services                           2.1.31.60019    PARTIAL    Package needs to be pulled/downloaded
base-provision-core       Automation - Base                        2.1.31.60019    PARTIAL    Package needs to be pulled/downloaded
command-runner            Command Runner                           2.1.31.60019    PARTIAL    Package needs to be pulled/downloaded
device-onboarding         Device Onboarding                        2.1.31.60019.2  PARTIAL    Package needs to be pulled/downloaded
dnac-platform             DNAC Platform                            1.0.12.4        PARTIAL    Package needs to be pulled/downloaded
icap-automation           Automation - Intelligent Capture         2.1.31.60019    PARTIAL    Package needs to be pulled/downloaded
image-management          Image Management                         2.1.31.60019    PARTIAL    Package needs to be pulled/downloaded
ncp-system                NCP - Base                               2.1.31.60019    PARTIAL    Package needs to be pulled/downloaded
ndp-base-analytics        Network Data Platform - Base Analytics   1.1.11.15       PARTIAL    Package needs to be pulled/downloaded
ndp-platform              Network Data Platform - Core             1.1.11.121      PARTIAL    Package needs to be pulled/downloaded
ndp-ui                    Network Data Platform - Manager          1.1.11.12       PARTIAL    Package needs to be pulled/downloaded
network-visibility        Network Controller Platform              2.1.31.60019.2  PARTIAL    Package needs to be pulled/downloaded
path-trace                Path Trace                               2.1.31.60019    PARTIAL    Package needs to be pulled/downloaded
platform-ui               Cisco DNA Center UI                      1.3.1.248       READY
sd-access                 SD Access                                2.1.31.60019.2  PARTIAL    Package needs to be pulled/downloaded
sensor-assurance          Assurance - Sensor                       1.2.12.124      PARTIAL    Package needs to be pulled/downloaded
sensor-automation         Automation - Sensor                      2.1.31.60019    PARTIAL    Package needs to be pulled/downloaded
system                    System                                   1.2.0.1013      READY

when i check the status of one of the package:

$ maglev catalog package status application-policy:2.1.29.170002

maglev-1 [main - https://kong-frontend.maglev-system.svc.cluster.local:443]

KIND                 RESOURCE                                      STATE      MESSAGE
----------------------------------------------------------------------------------------------------------------------
Package              application-policy:2.1.29.170002              PARTIAL    One or more child resources are not available yet
ServiceBundleGroup   fusion/application-policy:2.1.29.170002       NOT_FOUND  Manifest not available yet
ServiceBundleGroup   maglev-system/application-policy-ui:2.1.29.170002 NOT_FOUND  Manifest not available yet

So i tried to delete them till they were at ready state:

$ maglev catalog package delete assurance:1.2.12.125
ok

After all was deleted

$ maglev catalog package display

maglev-1 [main - https://kong-frontend.maglev-system.svc.cluster.local:443]

NAME                      DISPLAY_NAME                             VERSION         STATE      INFO
--------------------------------------------------------------------------------------------------------
application-policy        Application Policy                       2.1.28.170011   READY
assurance                 Assurance - Base                         1.2.11.304      READY
automation-core           NCP - Services                           2.1.28.60244.9  READY
base-provision-core       Automation - Base                        2.1.28.60244.9  READY
command-runner            Command Runner                           2.1.28.60244    READY
device-onboarding         Device Onboarding                        2.1.28.60244    READY
dnac-platform             DNAC Platform                            1.0.8.8         READY
icap-automation           Automation - Intelligent Capture         2.1.28.60244    READY
image-management          Image Management                         2.1.28.60244    READY
ncp-system                NCP - Base                               2.1.28.60244    READY
ndp-base-analytics        Network Data Platform - Base Analytics   1.1.11.8        READY
ndp-platform              Network Data Platform - Core             1.1.11.77       READY
ndp-ui                    Network Data Platform - Manager          1.1.11.8        READY
network-visibility        Network Controller Platform              2.1.28.60244.9  READY
path-trace                Path Trace                               2.1.28.60244    READY
platform-ui               Cisco DNA Center UI                      1.3.1.248       READY
sd-access                 SD Access                                2.1.28.60244.9  READY
sensor-assurance          Assurance - Sensor                       1.2.10.254      READY
sensor-automation         Automation - Sensor                      2.1.28.60244    READY
system                    System                                   1.2.0.1013      READY

But about 3 / 4 hours later the update failed.

System update status:
  Version successfully installed : 1.1.0.784
  Version currently processed    : 1.2.0.1013
    Update phase                 : failed
    Update details               : Node update took longer to complete in 10.0.32.1 [ version 1.2.0.1013 phase 4 ]
    Progress                     : 48%

  Updater State:
    Currently processed version  : 1.2.0.1013
    State                        : FAILED
    Sub-State                    : INSTALLED_HOST_COMPONENTS
    Details                      : Node update took longer to complete in 10.0.32.1 [ version 1.2.0.1013 phase 4 ]
    Source                       : system-updater

I then tried to force it back to an older version.

$ maglev system_updater update_system 1.1.0.784 -f
System update download initiated

But also failed later

$ sudo maglev system_updater update_info

System update status:
  Version successfully installed : 1.1.0.784
  Version currently processed    : 1.1.0.784
    Update phase                 : failed
    Update details               : Updating node 10.0.32.1 failed
    Progress                     : 52%

  Updater State:
    Currently processed version  : 1.1.0.784
    State                        : FAILED
    Sub-State                    : INSTALLED_HOST_COMPONENTS
    Details                      : Updating node 10.0.32.1 failed
    Source                       : system-updater

Eventhough all packages are deployed

$  maglev package status

maglev-1 [main - https://kong-frontend.maglev-system.svc.cluster.local:443]

NAME                      DISPLAY_NAME                             DEPLOYED        AVAILABLE       STATUS                    PROGRESS
---------------------------------------------------------------------------------------------------------------------------------------
application-policy        Application Policy                       2.1.28.170011   2.1.42.170001   DEPLOYED
assurance                 Assurance - Base                         1.2.11.304      1.3.1.140       DEPLOYED
automation-core           NCP - Services                           2.1.28.60244.9  2.1.43.60042    DEPLOYED
base-provision-core       Automation - Base                        2.1.28.60244.9  2.1.43.60042    DEPLOYED
command-runner            Command Runner                           2.1.28.60244    2.1.43.60042    DEPLOYED
device-onboarding         Device Onboarding                        2.1.28.60244    2.1.43.60042    DEPLOYED
dnac-platform             DNAC Platform                            1.0.8.8         1.1.1.2         DEPLOYED
icap-automation           Automation - Intelligent Capture         2.1.28.60244    2.1.43.60042    DEPLOYED
image-management          Image Management                         2.1.28.60244    2.1.43.60042    DEPLOYED
ncp-system                NCP - Base                               2.1.28.60244    2.1.43.60042    DEPLOYED
ndp-base-analytics        Network Data Platform - Base Analytics   1.1.11.8        1.3.1.204       DEPLOYED
ndp-platform              Network Data Platform - Core             1.1.11.77       1.3.1.615       DEPLOYED
ndp-ui                    Network Data Platform - Manager          1.1.11.8        1.3.1.185       DEPLOYED
network-visibility        Network Controller Platform              2.1.28.60244.9  2.1.43.60046    DEPLOYED
path-trace                Path Trace                               2.1.28.60244    2.1.43.60042    DEPLOYED
platform-ui               Cisco DNA Center UI                      1.3.1.248       1.3.1.250       DEPLOYED
sd-access                 SD Access                                2.1.28.60244.9  2.1.43.60045    DEPLOYED
sensor-assurance          Assurance - Sensor                       1.2.10.254      1.3.0.1313      DEPLOYED
sensor-automation         Automation - Sensor                      2.1.28.60244    2.1.43.60042    DEPLOYED
system                    System                                   1.1.0.784       1.2.0.1013      DEPLOYED

State of the packages is ready

$  maglev catalog package display

maglev-1 [main - https://kong-frontend.maglev-system.svc.cluster.local:443]

NAME                      DISPLAY_NAME                             VERSION         STATE      INFO
--------------------------------------------------------------------------------------------------------
application-policy        Application Policy                       2.1.28.170011   READY
assurance                 Assurance - Base                         1.2.11.304      READY
automation-core           NCP - Services                           2.1.28.60244.9  READY
base-provision-core       Automation - Base                        2.1.28.60244.9  READY
command-runner            Command Runner                           2.1.28.60244    READY
device-onboarding         Device Onboarding                        2.1.28.60244    READY
dnac-platform             DNAC Platform                            1.0.8.8         READY
icap-automation           Automation - Intelligent Capture         2.1.28.60244    READY
image-management          Image Management                         2.1.28.60244    READY
ncp-system                NCP - Base                               2.1.28.60244    READY
ndp-base-analytics        Network Data Platform - Base Analytics   1.1.11.8        READY
ndp-platform              Network Data Platform - Core             1.1.11.77       READY
ndp-ui                    Network Data Platform - Manager          1.1.11.8        READY
network-visibility        Network Controller Platform              2.1.28.60244.9  READY
path-trace                Path Trace                               2.1.28.60244    READY
platform-ui               Cisco DNA Center UI                      1.3.1.248       READY
sd-access                 SD Access                                2.1.28.60244.9  READY
sensor-assurance          Assurance - Sensor                       1.2.10.254      READY
sensor-automation         Automation - Sensor                      2.1.28.60244    READY
system                    System                                   1.2.0.1013      READY

Now my question is if there is something else i can try to restore the DNAC via maglev? I did download the DNAC ISO from cisco website, but could not find the instructions on how to do a re-installation of the DNAC software.

 

Thanks in advance!

Who Me Too'd this topic