cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
2641
Views
5
Helpful
4
Replies

Unable to repeat ZTP

Martin Kyrc
Level 3
Level 3

Hello,

I'm playing in dCloud lab. ZTP is working for the first time (new lab), but after decommission vEdge device is another attempt of ZTP not working again. I'm not sure how can I "reset" device to "default setting" (I can't reset whole device, because I lost IP connectivity to it).

 

I have this message in the log:

Event Name : device-template-attached-during-ztp
Event Details : host-name=vManage; uuid=ddd801b2-8cbe-4394-abd1-3b71e39886e3; peer-type=vedge

Event Name : vbond-reject-vedge-connection
Event Details : host-name=vBond-2; uuid=ddd801b2-8cbe-4394-abd1-3b71e39886e3; organization-name=Cisco Sy1 - 19968; sp-organization-name=Cisco Sy1 - 19968; reason=ERR_BID_NOT_VERIFIED

I can't find description of error message "ERR_BID_NOT_VERIFIED". I think, it's connected with device validation against vbond but I'm not sure what else can I verify.

My troubleshooting steps:

vedge# show control connections-history
<cut>
CRTREJSER - Challenge response rejected by peer. RXTRDWN - Received Teardown. CRTVERFL - Fail to verify Peer Certificate. RDSIGFBD - Read Signature from Board ID failed. CTORGNMMIS - Certificate Org name mismatch. SSLNFAIL - Failure to create new SSL context. DCONFAIL - DTLS connection failure. SERNTPRES - Serial Number not present. NOERR - No Error. VS_TMO - Peer vSmart Timed out. <cut> PEER PEER PEER PEER PEER SITE DOMAIN PEER PRIVATE PEER PUBLIC LOCAL REMOTE REPEAT TYPE PROTOCOL SYSTEM IP ID ID PRIVATE IP PORT PUBLIC IP PORT LOCAL COLOR STATE ERROR ERROR COUNT DOWNTIME ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ vbond dtls 0.0.0.0 0 0 198.18.1.11 12346 198.18.1.11 12346 default tear_down CTORGNMMIS NOERR 0 2019-05-27T11:32:27+0000 vbond dtls 0.0.0.0 0 0 198.18.1.13 12346 198.18.1.13 12346 default challenge_resp RXTRDWN SERNTPRES 12 2019-05-27T11:32:15+0000

It looks like cert name mismatch. Which one? Manual attempt of "vedge-cloud activate" is not successful.

Has anybody experience with ZTP in the lab environment?

martin

 

 

4 Replies 4

Danny De Ridder
Cisco Employee
Cisco Employee

Hello,

 

to get more details as to what exactly is wrong with the certifciate, you can enable debugs on the vedge.

To catch this type of errors one would use "debug vdaemon misc high".

Data will then be capture in the log named : /var/log/tmplog/vdebug.

You can tail the log like this :

show log /var/log/tmplog/vdebug tail -f

To stop the tail, you can do <CNTRL>C.

 

This should print more info with regards to the error event you are seeing.

 

Regards,

 

Danny.

 

 

 

Hi Danny,

thank you for log file where can I read some troubleshooting records ("vdaemon"):

vedge# debug vdaemon misc high

and then

vedge# vshell 
vedge:~$ less/tail/grep/... /var/log/tmplog/vdebug

I can find in the log file these messages connected to (un-successful) ZTP (with my comments):

!-- ge0/0 is my Internet interface (dhcp with connection to "ztp server", in this case vbond.cisco.com/<lab-ip-address>)
local7.debug: May 27 13:01:13 vedge VDAEMON[655]: vdaemon_find_next_active_wan_intf[1422]: %VDAEMON_DBG_MISC-1: Next wan interface to connect to vmanage = ge0_0 local7.debug: May 27 13:01:13 vedge VDAEMON[655]: vdaemon_set_confd_ztp_status[6293]: %VDAEMON_DBG_MISC-1: Setting ztp-status to 0 local7.debug: May 27 13:01:13 vedge stray: setsockopt: Bad file descriptor local7.debug: May 27 13:01:13 vedge VDAEMON[655]: vdaemon_cfg_confd_params_threaded_set[5747]: %VDAEMON_DBG_CONFD-1: Setting ztp status to 0 local7.debug: May 27 13:01:13 vedge VDAEMON[655]: vdaemon_cfg_confd_params_threaded_set[5769]: %VDAEMON_DBG_CONFD-1: ztp status = 0 local7.debug: May 27 13:01:13 vedge VDAEMON[655]: vdaemon_cfg_confd_params_threaded_set[6159]: %VDAEMON_DBG_MISC-1: Applying configuration after 0 retries to 0.. local7.debug: May 27 13:01:13 vedge VDAEMON[655]: vdaemon_dtls_verify_vbond_cert[1014]: %VDAEMON_DBG_MISC-1: No SP Org name set .. bailing out local7.debug: May 27 13:01:13 vedge VDAEMON[655]: vbond_proc_challenge[3445]: %VDAEMON_DBG_MISC-1: Unable to verify v Server's certificate .. bailing out local7.debug: May 27 13:01:13 vedge VDAEMON[655]: vdaemon_find_next_active_wan_intf[1422]: %VDAEMON_DBG_MISC-1: Next wan interface to connect to vmanage = none local7.info: May 27 13:01:13 vedge VDAEMON[655]: %Viptela-vedge-vdaemon-6-INFO-1400002: Notification: 5/27/2019 13:1:13 control-connection-auth-fail severity-level:major host-name:"vedge" system-ip::: personality:vedge peer-type:vbond peer-system-ip::: local-system-ip:0.0.0.0 local-color:default reason:"ERR_CERT_ORG_NAME_MISMCH"

In other words (correct me if I'm wrong):

  • vbond cert is not verified. Right?
  • I'm not sure, that the reason is "missing org name in cert", but last row in the log confirm it: "reason:ERR_CERT_ORG_NAME_MISMCH"

I tried to verify "server's cert" (in this lab: ztp.viptela.com or vbond.cisco.com) but no success:

vedge:~$ openssl s_client -showcerts -connect vbond.cisco.com:12346
connect: Connection timed out
connect:errno=110

it looks like "filtered port" (fw/acl?):

vedge:~$ nmap vbond.cisco.com -Pn -p 12346
Nmap scan report for vbond.cisco.com (198.18.1.11)
Host is up.
Other addresses for vbond.cisco.com (not scanned): 198.18.1.21
PORT      STATE    SERVICE
12346/tcp filtered netbus

I checked configuration and no specific port is defined - it means port 12346 (or 12347 for nat) is used. Correct?

Are my troubleshooting steps correct? Forgot I something to verify? In other words, I'm not closer to solving this issue. I have no remote access to ZTP and vBond server in this scenario (because dcloud lab) and I'm not able to verify "server's cert".

Any ideas?

martin

 

 

 

 

Hello,

 

for the nmap test, I think you need to use UDP to test, not TCP.

 

EDGE:~# nmap 192.168.0.231 -sU -Pn -p 12346 --system-dns

Starting Nmap 6.47 ( http://nmap.org ) at 2019-05-29 14:57 CEST
Nmap scan report for 192.168.0.231
Host is up.
PORT      STATE         SERVICE
12346/udp open|filtered unknown

Nmap done: 1 IP address (1 host up) scanned in 2.80 seconds
EDGE:~#

Do you have both org-name and sp-org-name set?

Is setting under system.

The error seems to come from sp-org-name missing on the vEdge?

 

Danny.

Martin,

I had this error today as well. The step I took that I believed resolved it was sending my WAN Edge List of certificates to the Controllers. I could be wrong though. 

Secondly, make sure your org name you specified is spelled exactly correct, including case-sensitivity

Hope this helps!

Phil

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Innovations in Cisco Full Stack Observability - A new webinar from Cisco