vManage unable to talk to devices after poweron

[bmcgahan]

I'm running vManage 20.9.4.1 in EVE-NG, and every time I power-on my virtual topology, the vManage doesn't work after it boots.

DTLS tunnels do form from vManage to the vSmart/vBond and Edge Routers, but vManage can't actually send any commands to them. The attached error is what I get when I try to view the Running Config from vManage of any device. It looks like it can't connect via NETCONF

I've tried clearing the control-connections, restarting the nms services on the vManage, but the only thing that works is a reload.

I don't understand why a reboot is needed, vs. vManage just powering on in the first place. Has anyone seen this issue before, and is there any workaround other than a reload?

TIA!

[MHM Cisco World]

Cedge# show control local-properties

Check the serial number of cedge before and after power-on/off

Also check cert. Validate

MHM

[bmcgahan]

Good idea, but I checked and the chassis/serial numbers stay the same between booting the instances, and the certificates are valid.

[balaji.bandi · Accepted solution]

i run same setup - i do not see any issue,

my boot order vbond and vsmart- then vmanage.

what device IP you showing error on screenshot ?

[bmcgahan]

In the screenshot above, that IP is the vSmart, but I get the same error if I try to send commands to any of the Edge Routers as well. What's strange is that the vManage shows 100% health to all devices:

show control connections looks good from the vManage also:

From vManage if I goto Configuration > Devices , then click the 3 dots on the right of any device and choose Running Configuration it hangs for a minute then returns this:

Device data error
jave.io.IOException: Unable to connect to vmanage-admin@172.17.1.1:830

[MHM Cisco World]

the error is different than your original post? the IP is different 

MHM

[bmcgahan]

It's for every IP in the PEER SYSTEM IP list in show control connections.

[MHM Cisco World]

 try this without adjust delay

Show sdwan peer <- in cedge

Do you see vbond connection OK 

MHM

[bmcgahan]

I added a 60 second delay for vBond to boot, and 120 second delay for vManage. I think this sorted the problem out for now. Thanks for the suggestion!

[Rajeev Sharma]

Could you try clearing control connections on the others instead, i mean vSmart, vBond and cEgde?

HTH.

[bmcgahan]

Yes, thanks, I tried that. The DTLS tunnels re-establish after a few seconds, but vManage still gets the same error when sending commands to any Device.  Also there are no errors in show control connections-history on any of the controllers or edges.