Nexus Dashboard Fabric Controller image management issue

[Sunil Gajjar]

This is about VMware ESX (.ova) based deployment with 3 Data Nodes and 3 App Nodes. Nexus Dashboard v2.1(2d), Fabric Controller v12.0.2f.

I am trying to upgrade image using Nexus Dashboard Fabric Controller but it keeps failing with below error: any idea as to how to fix it?

Starting image stage.
Staging image nxos.9.3.8.bin
ssh: connect to host 10.192.162.207 port 22: Connection timed out
Stage image status - Failed.

[Gilad Meirovich]

Hi @Sunil Gajjar 

Are You familiar with "10.192.162.207" IP address?
It might be pod's inner IP on the cluster (if You dont recognize this address).
Go to Nexus Dashboard > Admin Console > System Resources > Pods
type the filter: IP Address == 10.192.162.207

[Sunil Gajjar · Accepted solution]

The issue was fixed only after rebooting all 6 VMs associated with 6 nodes (just rebooting nodes did not help).

It looks management service named as "cisco-ndfc-dcnm-poap-mgmt-http-ssh" takes care of the upgrade and it was not able to communicate with the network before as 10.192.162.207 was the IP assigned to it, rebooting all VMs fixed communication issue and so the firmware upgrade started working.

[Guorong Kelvin Wang]

Hi, I have the same issue. I only have 1 VM, I rebooted it but there is no luck in the staging. However, my error message is this:
sim@10.x.x.x 's password:

Permission denied, please try again. sim@10.x.x.x's password: (where 10.x.x.x is the external pool IP)

Stage image status - Failed.

Any idea?

[Sunil Gajjar]

I have not seen this error but I guess it could be related to cisco-ndfc-dcnm-poap-mgmt-http-ssh service. 

To verify it, please go to Admin Console> Admin > System Settings and verify under External Service Pool table that assignment for "cisco-ndfc-dcnm-poap-mgmt-http-ssh" is showing as In Use.

If it is alright then I guess next step would be to check device credentials as it is throwing permission related error to ensure device is accessible via provided credentials under NDFC > Settings > LAN Credentials Management, as NDFC tries to use individual switch credentials in the Devices, to begin with. If the credentials (username/password) columns are empty in the Devices, the default credentials will be used.

[bssmith10]

I have the same issue, same error message, only rebooting the VMs did not resolve the problem.  I see the IP address configured for the service "cisco-ndfc-dcnm-poap-mgmt-http-ssh" in the Admin Console/Infrastructure/Cluster configuration/External Service Pools.  I have no idea how that was configured and it won't allow me to change it.  The ip address is not pingable so I assume that is why the stage is failing.

[Sunil Gajjar]

NDFC assigns IP to the service "cisco-ndfc-dcnm-poap-mgmt-http-ssh" from External Service Pools, and it is pinging so please check those IP are routable and VM configuration is done properly to allow VLAN for the subnet configured for External Service Pools.