cancel
Showing results for 
Search instead for 
Did you mean: 
cancel

Upgrade Cisco DCNM from version 11.2(1) to 11.3(1)

273
Views
0
Helpful
0
Comments

Introduction

The purpose of this document is to share my personal experience upgrading Cisco Data Center Network Manager (DCNM) from version 11.2(1) to 11.3(1) in my lab environment.

The details of this particular setup are:

  • DCNM 11.2(1) deployed using the OVA file on a VMware ESXi host
  • Two DCNM instances configured in Native HA mode (Active/Standby)
  • This Cisco DCNM setup is not in clustered mode
  • Network Insights - Resources (NIR) is not running in this setup
  • There are no DCNM Compute nodes in this setup
  • The upgrade process followed from the documentation was Inline Upgrade for DCNM Virtual Appliance in Native HA Mode

The outputs in this document were taken from devices in a specific lab environment. All devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

NOTE: It is advised to perform DCNM's Inline Upgrade using VMware's Web Console rather than via remote SSH Session. The reason is that the SSH session may timeout and result in an incomplete upgrade of DCNM.

Process

1. Download file dcnm-va.11.3.1.iso.zip from Cisco Software Download site.

Screen Shot 2020-06-02 at 2.03.25 PM.png

2. Confirm the MD5 hash from the downloaded ISO matches with the official MD5 hash indicated in the Cisco Software Download site. Hover the mouse over the file dcnm-va.11.3.1.iso.zip to make a table appear including this information.

Screen Shot 2020-06-02 at 2.09.59 PM.png

From my Mac OS X laptop, using the Terminal  (⌘ + Space bar and type "Terminal"), I moved to the directory where the ISO file was downloaded with the cd command and used  md5 command to generate the MD5 hash. Once confirmed the MD5 hash match with the one in Cisco.com we proceed.

$ md5 dcnm-va.11.3.1.iso.zip

Screen Shot 2020-06-02 at 5.15.28 PM.png

3. Log in to the Active DCNM and Standby DCNM instances via SSH using the root username and the password established during the DCNM installation.

You can have the IP addresses for the Active DCNM and Standby DCNM in the GUI via Administrator > DCNM Server > Native HA as seen in the next figure.

Screen Shot 2020-06-02 at 3.25.18 PM.png

Once logged in via SSH into the DCNM instances, run the command appmgr show ha-role to ensure the roles are correct.

Active DCNM Standby DCNM
Screen Shot 2020-06-02 at 2.35.07 PM.png Screen Shot 2020-06-02 at 2.35.29 PM.png
4. Unzip the dcnm-va.11.3.1.iso.zip file and upload the DCNM 11.3(1) .iso file to the /root/ folder in both, the Active DCNM and Standby DCNM.
Screen Shot 2020-06-02 at 5.18.23 PM.png
I simply used the utility SCP (Secure Copy) from Terminal to uploaded the ISO file to the Active DCNM. You need to use the DCNM root password. The command is show next.
HECSERRA-M-82J9:~ hecserra$ scp dcnm-va.11.3.1.iso.zip root@<ACTIVE_DCNM_IP_or_URL>:/root/
Screen Shot 2020-06-02 at 5.13.27 PM.png

Once the ISO file is in the Active DCNM node, you can use SCP from here to transfer it to the Standby DCNM instance. Since both instances are Layer 2 adjacent, it is likely the transfer will be faster.

Screen Shot 2020-06-02 at 5.21.34 PM.png

5. As a good practice, compare the MD5 hash of the ISO files that have been uploaded to the DCNM instances matches with the original one.

Screen Shot 2020-06-02 at 5.29.21 PM.png

Active DCNM Standby DCNM
[root@dcv-dcnm-vxlan-pod1-1 ~]# md5sum dcnm-va.11.3.1.iso
ece7dfbff034e3a85d8e7f79b49d3e71  dcnm-va.11.3.1.iso
[root@dcv-dcnm-vxlan-pod1-1 ~]#
[root@dcv-dcnm-vxlan-pod1-2 ~]# md5sum dcnm-va.11.3.1.iso
ece7dfbff034e3a85d8e7f79b49d3e71  dcnm-va.11.3.1.iso
[root@dcv-dcnm-vxlan-pod1-2 ~]#

6. Next, use the screen command on both DCNM instances. This is suggested to allow to execute commands that continue to run even if you get disconnected from the SSH session.

Active DCNM Standby DCNM
[root@dcv-dcnm-vxlan-pod1-1 ~]# screen
[root@dcv-dcnm-vxlan-pod1-1 ~]#
[root@dcv-dcnm-vxlan-pod1-2 ~]# screen
[root@dcv-dcnm-vxlan-pod1-2 ~]#

7. (Optional) It is highly advised to take a backup fo the DCNM deployment and retrieve it from DCNM before proceeding with the upgrade. Taking this backup took around 5 minutes in our lab environment. The size of the generated backup file was of 1.1 GB

Active DCNM
[root@dcv-dcnm-vxlan-pod1-1 ~]# appmgr backup
Backing up all Process...
=========================================================================
  DCNM Postgresql Environment
  . . .



<lots of output here>



************************************************************************************

Backup is available at /root/backup.06_07_2020__01_11_39.tar.gz

************************************************************************************

[root@dcv-dcnm-vxlan-pod1-1 ~]#
Standby DCNM
[root@dcv-dcnm-vxlan-pod1-2 ~]# appmgr backup
Backing up all Process...
=========================================================================
  DCNM Postgresql Environment
  . . .



<lots of output here>



************************************************************************************

Backup is available at /root/backup.06_07_2020__01_11_42.tar.gz

************************************************************************************

[root@dcv-dcnm-vxlan-pod1-2 ~]#

Using again SCP, we download the backup file to my laptop.

From my laptop, downloading backup file from the Active DCNM instance
HECSERRA-M-82J9:~ hecserra$ scp root@10.48.69.25:/root/backup.06_07_2020__01_11_39.tar.gz .
From my laptop, downloading backup file from the Standby DCNM instance
HECSERRA-M-82J9:~ hecserra$ scp root@10.48.69.32:/root/backup.06_07_2020__01_11_42.tar.gz .

8. We will now perform the upgraded on the Active DCNM instance. First step is to stop the HA applications on the Standby DCNM instance with the appmgr stop ha-apps command.

Standby DCNM
[root@dcv-dcnm-vxlan-pod1-2 ~]# appmgr stop ha-apps
Stopping AFW Applications...
Stopping AFW Server Processes
Stopping AFW Agent Processes
Stopped Application Framework...
Stopping High-Availability services: Done.

[root@dcv-dcnm-vxlan-pod1-2 ~]#

 

The next commands show how to mount the ISO file and run the upgrade script ./inline-upgrade.sh contained in the ISO file.

Run the script and when you are ready, type y to continue and notice that the scrip asks for a password for the the new sysadmin user that will be created.

  • The reason of this is that before DCNM version 11.3(1), the root Linux user is used to SSH into DCNM's Linux shell.
  • Starting DCNM version 11.3(1), the sysadmin user is now used to SSH into DCNM's Linux shell.

The root user is not allowed to SSH into 11.3(1) by default anymore improving security.

If you wish, you can modify this behavior with the appmgr root-access {permit|deny|without-password} command.

 

Active DCNM
[root@dcv-dcnm-vxlan-pod1-1 ~]# cd /root
[root@dcv-dcnm-vxlan-pod1-1 ~]# mkdir /mnt/iso
[root@dcv-dcnm-vxlan-pod1-1 ~]# mount -o loop dcnm-va.11.3.1.iso /mnt/iso
mount: /dev/loop0 is write-protected, mounting read-only
[root@dcv-dcnm-vxlan-pod1-1 ~]# cd /mnt/iso/packaged-files/scripts/
[root@dcv-dcnm-vxlan-pod1-1 scripts]# ./inline-upgrade.sh

=== Inline Upgrade to DCNM 11.3(1) ===

Current version: 11.2(1)
System type: HA

****************************************************************************************
WARNING: OS KERNEL WILL BE UPGRADED, THE SYSTEM WILL BE REBOOTED AFTER UPGRADE !!!
****************************************************************************************

****************************************************************************************
WARNING: AFTER THE UPGRADE SSH ROOT ACCESS WILL BE DENIED, USE sysadmin USER TO LOG IN
****************************************************************************************

Do you want to continue and perform the inline upgrade to 11.3(1)? [y/n]: y

Enter the password for the new sysadmin user:
Enter it again for verification:

==== Sun Jun 7 01:26:25 PDT 2020 - Inline upgrade started ====
Checking NIR is present or not...

 

Now sit and relax since this process may take around 30 minutes.

You will see a message similar to the following once the upgrade completes:

Active DCNM
Screen Shot 2020-06-07 at 11.03.21 AM.png

The Active DCNM instance should have reloaded by itself and you need to SSH again into it. This time use the sysadmin user.

Screen Shot 2020-06-07 at 12.34.52 PM.png

At this point, the Active DCNM instance should be now running version 11.3(1).

  • Ensure that on the Active DCNM instance, the output of the appmgr show ha-role command shows Current role: Active before proceeding with upgrading the Standby DCNM node.
  • Also, use appmgr status all to verify the status of DCNM at this point of the process.

Example of the output of these commands at this point of the process next:

Active DCNM Standby DCNM
[sysadmin@dcv-dcnm-vxlan-pod1-1 ~]$ appmgr show ha-role
Native HA enabled.
Deployed role: Active
Current role: Active
[sysadmin@dcv-dcnm-vxlan-pod1-1 ~]$
[root@dcv-dcnm-vxlan-pod1-2 ~]# appmgr show ha-role
Native HA enabled.
Deployed role: Standby
Current role: Stopped
[root@dcv-dcnm-vxlan-pod1-2 ~]#

Note that insecure access via HTTP is disabled. You must use HTTPS to log into the WebGUI.

Active DCNM
[sysadmin@dcv-dcnm-vxlan-pod1-1 ~]$ appmgr status all

DCNM v11 will only use HTTPS. Insecure access via HTTP is disabled.
Please use the url https://<DCNM-IP-ADDRESS> or https://<HOSTNAME> to launch the DCNM UI.

DCNM Status

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
=== ==== == == ======= ====== ===== = ==== ==== ======= =======
17629 fmserver 20 0 12.1g 3.0g 76440 S 6.7 12.9 4:17.26 java

Telemetry Manager Status

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
=== ==== == == ======= ====== ===== = ==== ==== ======= =======
17999 root 20 0 796236 5928 2892 S 0.0 0.0 0:00.14 telemetry-mgr.b

TFTP Status

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
=== ==== == == ======= ====== ===== = ==== ==== ======= =======
17992 root 20 0 27168 1068 812 S 0.0 0.0 0:00.00 xinetd

DHCP Status

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
=== ==== == == ======= ====== ===== = ==== ==== ======= =======
18002 dhcpd 20 0 103588 5748 3456 S 0.0 0.0 0:00.03 dhcpd

AMQP Status

Cluster status of node rabbit@dcv-dcnm-vxlan-pod1-1 ...
[{nodes,[{disc,[rabbit@dcv-dcnm-vxlan-pod1-1,rabbit@dcv-dcnm-vxlan-pod1-2]}]},
{running_nodes,[rabbit@dcv-dcnm-vxlan-pod1-2,rabbit@dcv-dcnm-vxlan-pod1-1]},
{cluster_name,<<"rabbit@dcv-dcnm-vxlan-pod1-1">>},
{partitions,[]},
{alarms,[{rabbit@dcv-dcnm-vxlan-pod1-2,[]},{rabbit@dcv-dcnm-vxlan-pod1-1,[]}]}]

[sysadmin@dcv-dcnm-vxlan-pod1-1 ~]$
Standby DCNM
[root@dcv-dcnm-vxlan-pod1-2 ~]# appmgr status all

DCNM v11 will only use HTTPS. Insecure access via HTTP is disabled.
Please use the url https://<DCNM-IP-ADDRESS> or https://<HOSTNAME> to launch the DCNM UI.

AMQP Status

Cluster status of node rabbit@dcv-dcnm-vxlan-pod1-2 ...
[{nodes,[{disc,[rabbit@dcv-dcnm-vxlan-pod1-1,rabbit@dcv-dcnm-vxlan-pod1-2]}]},
{running_nodes,[rabbit@dcv-dcnm-vxlan-pod1-1,rabbit@dcv-dcnm-vxlan-pod1-2]},
{cluster_name,<<"rabbit@dcv-dcnm-vxlan-pod1-1">>},
{partitions,[]},
{alarms,[{rabbit@dcv-dcnm-vxlan-pod1-1,[]},{rabbit@dcv-dcnm-vxlan-pod1-2,[]}]}]

Following applications are not running...
dcnm eplc epls telemetry-mgr tftp dhcp

[In a HA setup, dhcpd will be down until the IP ranges are entered in the default DHCPScopes in DCNM Web UI -> Config -> POAP]
[Note : In a Native HA setup, dcnm, dhcp, tftp and telemetry-mgr will run only on the node that is currently active.]
[Note : epls and eplc will run only when End Point Locator is enabled.]

[root@dcv-dcnm-vxlan-pod1-2 ~]#

 

9. If the previous outputs looks correct, proceed with upgrading the Standby DCNM instance before proceeding with the normal operation of DCNM. Note the same commands used to mount the ISO file and run the upgrade script but this time with the --standby parameter is used as seen in the ./inline-upgrade.sh --standby line next:

Standby DCNM
[root@dcv-dcnm-vxlan-pod1-2 ~]# cd /root
[root@dcv-dcnm-vxlan-pod1-2 ~]# mkdir /mnt/iso
[root@dcv-dcnm-vxlan-pod1-2 ~]# mount -o loop dcnm-va.11.3.1.iso /mnt/iso
mount: /dev/loop0 is write-protected, mounting read-only
[root@dcv-dcnm-vxlan-pod1-2 ~]# cd /mnt/iso/packaged-files/scripts/
[root@dcv-dcnm-vxlan-pod1-2 scripts]# ./inline-upgrade.sh --standby

=== Inline Upgrade to DCNM 11.3(1) ===

Current version: 11.2(1)
System type: HA

****************************************************************************************
WARNING: OS KERNEL WILL BE UPGRADED, THE SYSTEM WILL BE REBOOTED AFTER UPGRADE !!!
****************************************************************************************

****************************************************************************************
WARNING: AFTER THE UPGRADE SSH ROOT ACCESS WILL BE DENIED, USE sysadmin USER TO LOG IN
****************************************************************************************

Do you want to continue and perform the inline upgrade to 11.3(1)? [y/n]: y

Enter the password for the new sysadmin user:
Enter it again for verification:

==== Sun Jun 7 02:35:08 PDT 2020 - Inline upgrade started ====
This is Standby and hence nothing to cleanup for telemetry-mgr.

Again, sit and relax since this process may take around 10 minutes. Yes, faster than the upgrade of the Active DCNM instance.

When done, the Standby DCNM will reboot by itself as well.

Standby DCNM
Screen Shot 2020-06-07 at 11.48.23 AM.png

SSH again into the Standby DCNM instance using the sysadmin user.
Screen Shot 2020-06-07 at 12.38.58 PM.png

For comparison, here the output of appmgr show ha-role and appmgr status all at this point of the process:

Active DCNM Standby DCNM
[sysadmin@dcv-dcnm-vxlan-pod1-1 ~]$ appmgr show ha-role
Native HA enabled.
Deployed role: Active
Current role: Active
[sysadmin@dcv-dcnm-vxlan-pod1-1 ~]$
[sysadmin@dcv-dcnm-vxlan-pod1-2 ~]$ appmgr show ha-role
Native HA enabled.
Deployed role: Standby
Current role: Standby
[sysadmin@dcv-dcnm-vxlan-pod1-2 ~]$

 

Active DCNM
[sysadmin@dcv-dcnm-vxlan-pod1-1 ~]$ appmgr status all


DCNM v11 will only use HTTPS. Insecure access via HTTP is disabled.
Please use the url https://<DCNM-IP-ADDRESS> or https://<HOSTNAME> to launch the DCNM UI.

DCNM Status

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
=== ==== == == ======= ====== ===== = ==== ==== ======= =======
17629 fmserver 20 0 12.1g 3.2g 76464 S 0.0 13.8 4:50.54 java

Telemetry Manager Status

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
=== ==== == == ======= ====== ===== = ==== ==== ======= =======
17999 root 20 0 797836 6376 2944 S 0.0 0.0 0:00.60 telemetry-mgr.b

TFTP Status

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
=== ==== == == ======= ====== ===== = ==== ==== ======= =======
17992 root 20 0 27168 1068 812 S 0.0 0.0 0:00.00 xinetd

DHCP Status

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
=== ==== == == ======= ====== ===== = ==== ==== ======= =======
18002 dhcpd 20 0 103588 5748 3456 S 0.0 0.0 0:00.08 dhcpd

AMQP Status

Cluster status of node rabbit@dcv-dcnm-vxlan-pod1-1 ...
[{nodes,[{disc,[rabbit@dcv-dcnm-vxlan-pod1-1,rabbit@dcv-dcnm-vxlan-pod1-2]}]},
{running_nodes,[rabbit@dcv-dcnm-vxlan-pod1-2,rabbit@dcv-dcnm-vxlan-pod1-1]},
{cluster_name,<<"rabbit@dcv-dcnm-vxlan-pod1-1">>},
{partitions,[]},
{alarms,[{rabbit@dcv-dcnm-vxlan-pod1-2,[]},{rabbit@dcv-dcnm-vxlan-pod1-1,[]}]}]

[sysadmin@dcv-dcnm-vxlan-pod1-1 ~]$
Standby DCNM
[sysadmin@dcv-dcnm-vxlan-pod1-2 ~]$ appmgr status all


DCNM v11 will only use HTTPS. Insecure access via HTTP is disabled.
Please use the url https://<DCNM-IP-ADDRESS> or https://<HOSTNAME> to launch the DCNM UI.


AMQP Status

Cluster status of node rabbit@dcv-dcnm-vxlan-pod1-2 ...
[{nodes,[{disc,[rabbit@dcv-dcnm-vxlan-pod1-1,rabbit@dcv-dcnm-vxlan-pod1-2]}]},
{running_nodes,[rabbit@dcv-dcnm-vxlan-pod1-1,rabbit@dcv-dcnm-vxlan-pod1-2]},
{cluster_name,<<"rabbit@dcv-dcnm-vxlan-pod1-1">>},
{partitions,[]},
{alarms,[{rabbit@dcv-dcnm-vxlan-pod1-1,[]},{rabbit@dcv-dcnm-vxlan-pod1-2,[]}]}]

Following applications are not running...
dcnm telemetry-mgr tftp dhcp

[Note : In a Native HA setup, dcnm, dhcp, tftp and telemetry-mgr will run only on the node that is currently active.]

[sysadmin@dcv-dcnm-vxlan-pod1-2 ~]$

10. Login to DCNM via WebGUI, it is now running version 11.3(1)
https://<DCNM_VIP>. Ensure it is https and not http.

Screen Shot 2020-06-07 at 11.08.09 AM.png

Once logged into the WebGUI, go to Administrator > DCNM Server > Native HA to confirm the Native HA is OK.
Screen Shot 2020-06-07 at 12.02.32 PM.png

Conclusion

Following the documented Inline Upgrade for DCNM Virtual Appliance in Native HA Mode process (link below) was simple and straightforward. As any process of this nature, it is advised to perform it during a maintenance window. While the upgrade process itself can take around 1 hour, additional time must be accounted for downloading the ISO file from Cisco.com, transferring it to DCNM and downloading the backup files that are advised to be taken on DCNM before proceeding with the upgrade.

If your DCNM setup is in clustered mode and includes DCNM Compute nodes and/or Network Insights, there are a couple of additional steps that can be found in the official documentation guide.

Cisco DCNM Installation and Upgrade Guide for Classic LAN Deployment, Release 11.3(1)
Chapter: Upgrading Cisco DCNM
Inline Upgrade for DCNM Virtual Appliance in Native HA Mode 

The steps to upgrade the DCNM Standalone Mode are part of this same document and are pretty much the same.

Finally, you may need to restore the certificates after upgrading. This is a disruptive process. More information via:

Cisco DCNM Installation and Upgrade Guide for Classic LAN Deployment, Release 11.3(1)
Chapter: Certificates
Restoring the certificates after an upgrade

CreatePlease to create content
Content for Community-Ad

Cisco COVID-19 Survey

This widget could not be displayed.