This article describes control connection problems that might arise between SD-WAN routers (cEdge / vEdge) and vSmart controllers and between SDWAN routers (cEdge / vEdge) and vManage NMSs when you are bringing up these devices.  All the troubleshooting steps below apply to both vEdge and cEdge routers, but all the captures are taken from vEdge Router.   Check Control Connection Status To check the status of the control connections of all SD-WAN routers, in the vManage Dashboard, view the Control Status pane. Click any row to display a table with device details. To check the status of a single vEdge router's control connections, in vManage NMS, select Monitor ► Network, locate the desired vEdge router, and click its hostname. In the left pane, click Control Connections. To display active control connections from the CLI, issue the   show control connections  [vEdge] or show sdwan control connections [cEdge]  command. If a control connection is not listed in the command output, that connection is not operational. If the vManage NMS screens or the command output indicates connection problems between a vEdge router and a vSmart controller or vManage NMS, see the sections below to troubleshoot the problem. General checks Before starting to troubleshoot, make sure to confirm the SD-WAN router in question has been configured properly. It includes: Valid Certificate has been installed. Following configs put in place under “system” block System-IP Site-ID, Organization-Name vBond address VPN 0 Transport interface is configured with Tunnel option and IP address System Clock is configured correctly on the vEdge matching with other devices/controllers show clock  will confirm the current time set Use  clock set  to set the right time on the device. If the portion of the vEdge router's configuration that establishes control connections is correct such that control connections are up, the   show control local-properties   command output looks similar to this example, in Releases 16.3 and later:   vEdge# show control local-properties personality vedge sp-organization-name Viptela, Inc. organization-name Viptela, Inc. certificate-status Installed root-ca-chain-status Installed certificate-validity Valid certificate-not-valid-before Sep 06 22:39:01 2016 GMT certificate-not-valid-after Sep 06 22:39:01 2017 GMT dns-name trainingvbond.viptela.com site-id 10 domain-id 1 protocol dtls tls-port 0 system-ip 172.1.10.1 chassis-num/unique-id 66cb2a8b-2eeb-479b-83d0-0682b64d8190 serial-num 12345718 vsmart-list-version 0 keygen-interval 1:00:00:00 retry-interval 0:00:00:17 no-activity-exp-interval 0:00:00:12 dns-cache-ttl 0:00:02:00 port-hopped TRUE time-since-last-port-hop 20:16:24:43 number-vbond-peers 0 number-active-wan-interfaces 1 NAT TYPE: E -- indicates End-point independent mapping A -- indicates Address-port dependent mapping N -- indicates Not learned Note: Requires minimum two vbonds to learn the NAT type PUBLIC PUBLIC PRIVATE PRIVATE PRIVATE MAX RESTRICT/ LAST SPI TIME NAT VM INTERFACE IPv4 PORT IPv4 IPv6 PORT VS/VM COLOR STATE CNTRL CONTROL/ LR/LB CONNECTION REMAINING TYPE CON STUN PRF -------------------------------------------------------------------------------------------------------------------------------------------------------------- ge0/4 73.241.233.20 12386 192.168.0.20 2601:647:4380:ca75::c2 12386 2/1 public-internet up 2 no/yes/no No/Yes 0:10:34:16 0:03:03:26 E 5   If the control connections are down, debug the vEdge router's configuration as discussed in the sections below. Check that a Valid Certificate Is Installed on the vEdge Router A valid certificate must be installed on the vEdge router. To view the status of the router's certificate, in vManage NMS, select the Configuration ► Certificates screen, and select the vEdge List tab. In the table, the Validate column should be green, to indicate that the certificate is valid, and the State column should show a green icon. From the CLI, use the   show control local-properties   command.  Check the vEdge Router Configuration The vEdge router configuration must include a system IP address, a site ID, an organization name, and a vBond orchestrator IP address or DNS name. To view the device's running configuration in vManage NMS: Select the Configuration ► Devices screen. Select the vEdge List tab. For the desired vEdge router, click the More Actions icon to the right of the row. Click Running Configuration. To view the device's running configuration from the CLI, use the   show   running-config   [vEdge] or show sdwan running-config [cEdge] command. Check the Clock Time The vEdge router's clock must have the same time configured as other devices in the overlay network. To view the system time on the device, use the CLI command   show clock.  Check for Routing Issues Control connections might not come up if the overlay network has routing issues. If this is the case, the State column in the   show control connections   command output has a value of "connect", which indicates that a connection attempt is in progress. If the control connection is up, the State column shows a value of "up". PEER PEER CONTROLLER PEER PEER PEER SITE DOMAIN PEER PRIVATE PEER PUBLIC GROUP TYPE PROTOCOL SYSTEM IP ID ID PRIVATE IP PORT PUBLIC IP PORT LOCAL COLOR STATE UPTIME ID ------------------------------------------------------------------------------------------------------------------------------------------------------------------- vbond dtls - 0 0 1.3.25.25 12346 1.3.25.25 12346 gold connect 0 vbond dtls - 0 0 1.3.25.25 12346 1.3.25.25 12346 silver connect 0 Check for routing issues as discussed in the sections below. Check the Route Table for a Valid Route to the Next Hop The route table (RIB) might not contain a valid route to the correct next hop. To verify the entries in the route table in vManage NMS: Select the Monitor ► Network screen. Select the vEdge router by clicking its IP address. In the screen that opens, select Real Time in the left pane. In the Device Options drop-down, select IP Routes. To verify the entries in the route table from the CLI, use the command   show ip routes vpn 0   or   show ip routes vpn 0 prefix/length. Check for a Leaked TLOC IP Address The TLOC IP address might be being leaked between upstream ISPs. To verify connectivity, ping the default gateway. Check for correct distance values and protocols for the IP prefix. Use Error Codes to Troubleshoot Control Connection Problems If control connections are down, issue the   show control connections-history [vEdge] or show sdwan control local-properties [cEdge] on CLI command on the vEdge router to display information about control plane connection attempts initiated by the router. The Local Error or Remote Error columns in the output report any errors that occurred with the connection initialization attempts. The following errors are related to issues related to configuration and establishing control tunnels:   BIDNTVRFD, CRTREJSER, SERNTPRES—Missing serial number CTORGNMMIS—Organization name mismatch in the certificate CRTVERFL—Certificate verification failure DCONFAIL—DLTS connection failure DISCVBD or SYSIPCHNG—Transient error conditions DISTLOC—TLOC is disabled LISFD—Socket errors NOVMCFG—vEdge router template is not attached in vManage NMS RDSIGFBD, TXCHTOBD—For a hardware vEdge router, the board identifier did not initialize VB_TMO, VM_TMO, VP_TMO, VS_TMO—Peer timeout VSCRTREV—Revoked or invalidated certificate   BIDNTVRFD, CRTREJSER, SERNTPRES: Missing Serial Number Problem Statement A device's serial number is missing from the vSmart controllers. Identify the Problem Issue the   show control connections-history   command. In the Local Error column of the output, the values BIDNTVRFD, CRTREJSER, and SERNTPRES indicate a missing serial number. BIDNTVRFD indicates a missing serial number for vBond orchestrators. CRTREJSER indicates a missing serial number for vEdge routers and vSmart controllers. SERNTPRES on a vBond orchestrator indicates a serial number mismatch between vSmart controllers. PEER PEER PEER PEER PEER SITE DOMAIN PEER PRIVATE PEER PUBLIC REPEAT TYPE PROTOCOL SYSTEM IP ID ID PRIVATE IP PORT PUBLIC IP PORT LOCAL COLOR STATE LOCAL/REMOTE COUNT DOWNTIME --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- vbond dtls - 0 0 1.4.30.30 12346 1.4.30.30 12346 mpls tear_down CRTREJSER NOERR 161 2016-10-14T11:19:39-0700 vbond dtls - 0 0 1.4.30.30 12346 1.4.30.30 12346 silver tear_down CRTREJSER NOERR 161 2016-10-14T11:19:38-0700 vbond dtls - 0 0 1.3.25.25 12346 1.3.25.25 12346 mpls tear_down CRTREJSER NOERR 160 2016-10-14T11:19:22-0700 vbond dtls - 0 0 1.3.25.25 12346 1.3.25.25 12346 silver tear_down CRTREJSER NOERR 160 2016-10-14T11:19:22-0700 Resolve the Problem Send the device's serial number to the controllers: In vManage NMS, select the Configuration ► Certificates screen. In the vEdge List tab, select the device whose serial number is missing. Click Send to Controllers. CTORGNMMIS: Organization Name Mismatch Problem Statement The organization name is not identical among all devices in the overlay network. Identify the Problem Issue the   show control connections-history   command. In the Local Error column of the output, the value CTORGNMMIS indicates an organization name mismatch in the overlay network. PEER PEER PEER PEER PEER SITE DOMAIN PEER PRIVATE PEER PUBLIC LOCAL REMOTE REPEAT TYPE PROTOCOL SYSTEM IP ID ID PRIVATE IP PORT PUBLIC IP PORT LOCAL COLOR STATE ERROR ERROR COUNT DOWNTIME ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- vbond dtls - 0 0 1.3.25.25 12346 1.3.25.25 12346 mpls tear_down CTORGNMMIS NOERR 19 2016-10-06T00:39:37+0000 vbond dtls - 0 0 1.3.25.25 12346 1.3.25.25 12346 gold tear_down CTORGNMMIS NOERR 28 2016-10-06T10:39:20-0000 Resolve the Problem To configure the correct organization name on every device in the overlay network, use the   organization-name   command.   CRTVERFL: Certificate Verification Failure Problem Statement Verification of the vEdge router's certificate failed. Identify the Problem Issue the   show control connections-history   command. In the Local Error column of the output, the value CRTVERFL indicates certificate verification failure. PEER PEER PEER PEER PEER SITE DOMAIN PEER PRIVATE PEER PUBLIC LOCAL REMOTE REPEAT TYPE PROTOCOL SYSTEM IP ID ID PRIVATE IP PORT PUBLIC IP PORT LOCAL COLOR STATE ERROR ERROR COUNT DOWNTIME ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- vbond dtls - 0 0 1.3.25.25 12346 1.3.25.25 12346 mpls tear_down CRTVERFL NOERR 142 2016-10-03T00:39:37+0000 vbond dtls - 0 0 1.3.25.25 12346 1.3.25.25 12346 gold tear_down CRTVERFL NOERR 213 2016-10-03T10:39:20-0000 Resolve the Problem   DCONFAIL: DTLS Connection Failure Problem Statement The vEdge router does not establish DTLS connections to controllers in the overlay network. Identify the Problem Issue the   show control connections-history   command. In the Local Error column of the output, the value DCONFAIL indicates DTLS connection failure. PEER PEER PEER PEER PEER SITE DOMAIN PEER PRIVATE PEER PUBLIC LOCAL REMOTE REPEAT TYPE PROTOCOL SYSTEM IP ID ID PRIVATE IP PORT PUBLIC IP PORT LOCAL COLOR STATE ERROR ERROR COUNT DOWNTIME -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- vbond dtls - 0 0 1.3.25.25 12346 1.3.25.25 12346 mpls connect DCONFAIL NOERR 1 2016-09-22T10:49:04-0700 vbond dtls - 0 0 1.3.25.25 12346 1.3.25.25 12346 gold connect DCONFAIL NOERR 1 2016-09-22T10:49:03-0700 Resolve the Problem Verify that the next hop is reachable: Issue the   show ip routes vpn 0   command. In the Next Hop Address column of the output, make sure the next-hop address is present. If the next-hop address is missing, configure a   static route   to it in VPN 0. If the next-hop address is present, ping it. If the ping fails, troubleshoot connectivity to the next hop. Verify that a default gateway is installed in the route table: Issue the   show ip routes vpn 0   command. In the Prefix column of the output, make sure the default gateway, 0.0.0.0/0, is present If the default gateway is missing, configure a static route for it in VPN 0. Verify that the DTLS ports are open in any firewall you use. For more information, see   Firewall Ports for Viptela Deployments. To verify that the default gateway is correctly mapping the IP address to the MAC address correctly, use the   show arp   command. Verify reachability: Ping the default gateway. Ping the Google DNS (8.8.8.8 or 8.8.4.4). If the ICMP service is allowed on the vEdge router, ping the vBond orchestrator. Run a   traceroute   command to the vBond orchestrator's DNS address.   DISCVBD or SYSIPCHNG: Transient Conditions Problem Statement The vEdge router experiences transient control connection errors. Identify the Problem Issue the   show control connections-history   command. In the Local Error column of the output, the value DISCVBD indicates that the vEdge router's connection to the vBond orchestrator has been taken down. This is normal behavior. The value SYSIPCHNG indicates a change in the vEdge router's system IP address. PEER PEER PEER PEER PEER SITE DOMAIN PEER PRIVATE PEER PUBLIC REPEAT TYPE PROTOCOL SYSTEM IP ID ID PRIVATE IP PORT PUBLIC IP PORT LOCAL COLOR STATE LOCAL/REMOTE COUNT DOWNTIME --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- vbond dtls - 0 0 1.3.25.25 12346 1.3.25.25 12346 lte tear_down DISCVBD /NOERR 0 2016-09-27T19:06:16+0000 vmanage tls 172.1.0.18 200 0 1.4.28.28 12346 1.3.25.25 12346 lte tear_down SYSIPCHNG /NOERR 0 2016-09-27T19:05:30+0000 vsmart tls 172.1.0.19 103 1 1.3.26.26 12346 1.3.25.25 12346 lte tear_down SYSIPCHNG /NOERR 0 2016-09-27T19:05:30+0000 vsmart tls 172.1.0.16 104 1 1.3.29.29 12346 1.3.25.25 12346 lte tear_down SYSIPCHNG /NOERR 0 2016-09-27T19:05:30+0000 vbond dtls - 0 0 1.4.30.30 12346 1.3.25.25 12346 lte tear_down DISCVBD /NOERR 0 2016-09-27T17:56:30+0000 Resolve the Problem These issues are part of normal operation of the overlay network. They have no impact on production traffic, and they resolve by themselves, with no action required. DISTLOC: Disabled TLOC Problem Statement A TLOC, or transport location, is disabled on the vEdge router. A TLOC identifies the physical interface where a vEdge router connects to the WAN transport network or to a NAT gateway.    Identify the Problem Issue the   show control connections-history   command. In the Local Error column of the output, the value DISTLOC indicates that a TLOC is disabled. PEER PEER PEER PEER PEER SITE DOMAIN PEER PRIVATE PEER PUBLIC LOCAL REMOTE REPEAT TYPE PROTOCOL SYSTEM IP ID ID PRIVATE IP PORT PUBLIC IP PORT LOCAL COLOR STATE ERROR ERROR COUNT DOWNTIME -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- vmanage dtls 172.1.0.18 1001 0 1.4.28.28 12346 1.4.28.28 12346 gold tear_down DISTLOC NOERR 0 2016-09-25T18:00:41-0700 vsmart dtls 172.1.0.19 1013 1 1.3.29.29 12346 1.3.29.29 12346 gold tear_down DISTLOC NOERR 0 2016-09-25T18:00:41-0700 vsmart dtls 172.1.0.16 1011 1 1.3.26.26 12346 1.3.26.26 12346 gold tear_down DISTLOC NOERR 0 2016-09-25T18:00:41-0700 Resolve the Problem A TLOC might be disabled if: Control connections have been cleared. The TLOC color has changed. The system IP has changed. Use the   show running-config   command to check the configuration. Reconfigure any attributes necessary.   LISFD: Socket Errors Problem Statement Socket error messages occur. Identify the Problem Issue the   show control connections-history   command. In the Local Error column of the output, the value LISFD indicates a socket error. PEER PEER PEER PEER PEER SITE DOMAIN PEER PRIVATE PEER PUBLIC LOCAL REMOTE REPEAT TYPE PROTOCOL SYSTEM IP ID ID PRIVATE IP PORT PUBLIC IP PORT LOCAL COLOR STATE ERROR ERROR COUNT DOWNTIME ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- vsmart dtls 1.1.1.8 7 0 1.4.28.28 12447 1.4.28.28 12447 gold up LISFD NOERR 45 2016-07-26T23:53:32-0700 vmanage dtls 1.1.1.7 7 1 1.3.29.29 12647 1.3.29.29 12647 gold up LISFD NOERR 70 2016-07-26T23:53:32-0700 vmanage dtls 1.1.1.7 7 1 1.3.26.26 12747 1.3.26.26 12867 gold up LISFD NOERR 69 2016-07-26T23:53:32-0700 Resolve the Problem Socket errors might occur if: The overlay network contains duplicate IP addresses, especially duplicate transport addresses. Packets have been corrupted. The vEdge router receives a reset request from the remote device. The vEdge router and the vSmart controller are not both using DTLS or TLS ports, but one device is using TLS while the other is using DTLS. Forwarding ports are not open.   NOVMCFG: vEdge Router Template Not Attached in vManage NMS Problem Statement The vEdge router's configuration template was not attached to the router during the bringup of the router. Identify the Problem Issue the   show control connections-history   command. In the Remote Error column of the output, the value NOVMCFG indicates that the vEdge router's template was not attached during bringup. PEER PEER PEER PEER PEER SITE DOMAIN PEER PRIVATE PEER PUBLIC REPEAT TYPE PROTOCOL SYSTEM IP ID ID PRIVATE IP PORT PUBLIC IP PORT LOCAL COLOR STATE LOCAL/REMOTE COUNT DOWNTIME ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- vmanage tls 172.1.0.18 200 0 1.4.28.28 12346 1.3.25.25 12346 lte tear_down RXTRDWN/ NOVMCFG 78 2016-10-06T08:10:06+0000 Resolve the Problem During bringup from ZTP, if the device is not attached with a template on the vManage, then you will see “No Config. in vManage for device”.    Make sure that the template is assigned on the vManage for the device in question. RDSIGFBD, TXCHTOBD: Board ID Not Initialized Problem Statement In an unstable network, when connections are frequently going down and coming up, the Trusted Board ID chip on a hardware vEdge router might not initialize. Identify the Problem Issue the   show control connections-history   command. In the Local Error column of the output, the values RDSIGFBD and TXCHTOBD indicate a problem with initializing the Trusted Board ID chip. RDSIGFBD indicates that the network failed to read the signature from board ID. TXCHTOBD indicates that the network failed to send a challenge to board ID. PEER PEER PEER PEER PEER SITE DOMAIN PEER PRIVATE PEER PUBLIC LOCAL REMOTE REPEAT TYPE PROTOCOL SYSTEM IP ID ID PRIVATE IP PORT PUBLIC IP PORT LOCAL COLOR STATE ERROR ERROR COUNT DOWNTIME ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ vbond dtls 0 0 172.16.1.1 12346 172.16.1.1 12346 publicinternet challenge TXCHTOBD NOERR 0 2016­07­01T15:47:40+000 vbond dtls ­ 0 0 172.16.1.1 12346 172.16.1.1 12346 publicinternet challenge TXCHTOBD NOERR 0 2016­07­01T15:47:40+0000 Resolve the Problem  Sometimes due to locking issues, sending challenge to board-id fails and when that happens, we reset the board-ID and try again.    It shouldn’t happen often, it delays the bring up of control connections. This should be fixed in newer versions. VB_TMO, VM_TMO, VP_TMO, VS_TMO: Peer Timeout Problem Statement A peer timeout occurs if the vEdge router loses reachability to a controller in the overlay network. Identify the Problem Issue the   show control connections-history   command. In the Local Error column of the output, the values VB_TMO, VM_TMO, VP_TMO, and VS_TMO indicate a peer timeout error for vBond orchestrator, vManage NMS, peer vEdge routers, and vSmart controllers, respectively. PEER PEER PEER PEER PEER SITE DOMAIN PEER PRIVATE PEER PUBLIC REPEAT TYPE PROTOCOL SYSTEM IP ID ID PRIVATE IP PORT PUBLIC IP PORT LOCAL COLOR STATE LOCAL/REMOTE COUNT DOWNTIME ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- vmanage tls 172.1.0.18 200 0 1.4.28.28 12346 1.3.25.25 12346 default tear_down VM_TMO /NOERR 0 2016-10-01T10:54:20-0700 Issue the   show control connections-history detail   command to check the hello counters. A discrepancy between the transfer and receive Hello packet counters indicates packet loss between the vEdge router and the controller. Tx Statistics- -------------- hello 1467659 connects 0 registers 0 register-replies 0 challenge 0 challenge-response 1 challenge-ack 0 teardown 1 teardown-all 0 vmanage-to-peer 0 register-to-vmanage 0 Rx Statistics- -------------- hello 1467279 connects 0 registers 0 register-replies 0 challenge 1 challenge-response 0 challenge-ack 1 teardown 0 vmanage-to-peer 0 register-to-vmanage 0 Resolve the Problem Troubleshoot reachability to the controller using ping, traceroute, and rapid ping. Increase the   hello-interval   and   hello-tolerance   values on the interface to prevent packet loss.   VECRTREV, VSCRTREV: Revoked Certificate Problem Statement The certificate for the vEdge router or vSmart controller has been revoked. Identify the Problem Issue the   show control connections-history   command. In the Local Error column of the output, the value VECRTREV indicates a revoked certificate on a vEdge router. The value VSCRTREV indicates a revoked certificate on a remote vSmart controller. If a certificate is revoked on a local vSmart controller, the value VSCRTREV displays in the Remote Error column. PEER PEER PEER PEER PEER SITE DOMAIN PEER PRIVATE PEER PUBLIC LOCAL REMOTE REPEAT INSTANCE TYPE PROTOCOL SYSTEM IP ID ID PRIVATE IP PORT PUBLIC IP PORT REMOTE COLOR STATE ERROR ERROR COUNT DOWNTIME -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0 vsmart dtls 172.1.0.16 1011 1 1.3.26.26 12346 1.3.26.26 12346 default tear_down VSCRTREV NOERR 0 2016-10-14T11:59:13-0700 Resolve the Problem Certification verification failure is when certificate cannot be verified with the root cert installed: 1) Check time - it should be at least  within vBond's certificate validity range. show clock   2) This can be caused by root cert corruption on vEdge Open a Cisco / Viptela Support case to resolve the issue. ... View more