I'm trying to build a lab based on EVE-NG, viptela controllers and CSR1000v images and It looks like i'm unable to overcome this issue.
I'm using my own enterprise root CA. All vManage, Vbond and vSmart are successfully up and running. Tried on different versions, the problem can be recreated on any available controllers and CSR1000v images.
what has been done:
0. I have all 3 controllers up and running, no issues.
1. I created a list of CSRv devices in the cisco PnP portal and downloaded the file list.
2. The list was uploded into vManage.
3. Bootstrap was generated, and uploaded into CSRv and then request platform software sdwan software reset used
4. L3 connectivity was provided and working, tunnels created CSRv was visible in vmanage.
5. The widely described way to use the command "request platform software sdwan vedge_cloud activate chassis-number" did not work. the CSRv simply were not able to communicate to vManage/vBond after that having DISTLOC.
6. I decided to use manual signing the devices certificates. Cert signed, new certificate successfully uploaded on CSRv. Once it's done, the CSRv is never able to make connection to vBond having the error as in the subject
some snippets:
CSRv:
HQ-R1#sh sdwan run
system
system-ip 172.16.0.12
domain-id 1
site-id 1
admin-tech-on-failure
organization-name "OVEL Lab"
vbond 1.1.0.12
ntp peer 1.1.0.1
interface GigabitEthernet1
no shutdown
ip address 1.1.1.2 255.255.255.0
no mop enabled
no mop sysid
negotiation auto
interface Tunnel1
no shutdown
ip unnumbered GigabitEthernet1
tunnel source GigabitEthernet1
tunnel mode sdwan
exit
clock timezone AEST 10 0
sdwan
interface GigabitEthernet1
tunnel-interface
encapsulation ipsec
allow-service all
no allow-service bgp
allow-service dhcp
allow-service dns
allow-service icmp
no allow-service sshd
no allow-service netconf
no allow-service ntp
no allow-service ospf
no allow-service stun
no allow-service snmp
exit
HQ-R1#sh sdwan control local-properties
personality vedge
sp-organization-name OVEL Lab
organization-name OVEL Lab
root-ca-chain-status Installed
certificate-status Installed
certificate-validity Valid
certificate-not-valid-before Apr 16 14:34:48 2021 GMT
certificate-not-valid-after Apr 16 14:34:48 2023 GMT
enterprise-cert-status Not-Applicable
enterprise-cert-validity Not Applicable
enterprise-cert-not-valid-before Not Applicable
enterprise-cert-not-valid-after Not Applicable
dns-name 1.1.0.12
site-id 1
domain-id 1
protocol dtls
tls-port 0
system-ip 172.16.0.12
chassis-num/unique-id CSR-16711BCD-4B91-4A39-E3BD-C63E6A5EC003
serial-num 100000001D3D436E39E4A99B3300000000001D
token Invalid
keygen-interval 1:00:00:00
retry-interval 0:00:00:16
no-activity-exp-interval 0:00:00:20
dns-cache-ttl 0:00:02:00
port-hopped TRUE
time-since-last-port-hop 0:00:24:21
embargo-check success
number-vbond-peers 1
INDEX IP PORT
-----------------------------------------------------
0 1.1.0.12 12346
number-active-wan-interfaces 1
NAT TYPE: E -- indicates End-point independent mapping
A -- indicates Address-port dependent mapping
N -- indicates Not learned
Note: Requires minimum two vbonds to learn the NAT type
PUBLIC PUBLIC PRIVATE PRIVATE PRIVATE MAX RESTRICT/ LAST SPI TIME NAT VM
INTERFACE IPv4 PORT IPv4 IPv6 PORT VS/VM COLOR STATE CNTRL CONTROL/ LR/LB CONNECTION REMAINING TYPE CON
STUN PRF
--------------------------------------------------------------------------------------------------------------------------------- --------------------------------------------------------------------------
GigabitEthernet1 1.1.1.2 12386 1.1.1.2 :: 12386 0/0 default up 2 no/yes/no No/No 0:00:00:00 0:10:42:56 N 5
Root CA is imported
HQ-R1#sh sdwan certificate root-ca-cert | i OVEL
Issuer: DC=LOCAL, DC=OVEL, CN=OVEL-WINSERVER-CA
Subject: DC=LOCAL, DC=OVEL, CN=OVEL-WINSERVER-CAconnection-history
PEER PEER PEER SITE DOMAIN PEER PRIVATE PEER PUBLIC LOCAL REMOTE REPEAT TYPE PROTOCOL SYSTEM IP ID ID PRIVATE IP PORT PUBLIC IP PORT LOCAL COLOR STATE ERROR ERROR COUNT DOWNTIME ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ vbond dtls 0.0.0.0 0 0 1.1.0.12 12346 1.1.0.12 12346 default challenge_resp RXTRDWN BIDNTVRFD 38 2021-04-17T02:33:29+1000
obviously, L3 is up and everything is pingable
Vbond log messages:
Apr 17 02:34:20 OVEL-VBOND1 VBOND[10509]: %Viptela-OVEL-VBOND1-vbond_0-6-INFO-1400002: Notification: vbond-reject-vedge-connection severity-level:major host-name:"OVEL-VBOND1" system-ip:1.1.255.12 uuid:"CSR-16711BCD-4B91-4A39-E3BD-C63E6A5EC003" organization-name:"OVEL Lab" sp-organization-name:"OVEL Lab" reason:"ERR_BID_NOT_VERIFIED"
Apr 17 02:34:20 OVEL-VBOND1 confd[992]: netconf id=21 sending notification {http://viptela.com/security}vbond-reject-vedge-connection
Apr 17 02:34:20 OVEL-VBOND1 VBOND[10509]: %Viptela-OVEL-VBOND1-vbond_0-6-INFO-1400002: Notification: control-connection-auth-fail severity-level:major host-name:"OVEL-VBOND1" system-ip:1.1.255.12 personality:vbond peer-type:vedge peer-system-ip::: local-system-ip:1.1.255.12 local-color:default reason:"ERR_BID_NOT_VERIFIED"
Apr 17 02:34:20 OVEL-VBOND1 confd[992]: netconf id=21 sending notification {http://viptela.com/security}control-connection-auth-failValid vEdge:
OVEL-VBOND1# show orchestrator valid-vedges orchestrator valid-vedges CSR-16711BCD-4B91-4A39-E3BD-C63E6A5EC003 serial-number 100000001D3D436E39E4A99B3300000000001D validity valid org "OVEL Lab" hardware-installed-serial-number N/A subject-serial-number CSR-16711BC
not sure where subject-serial-number is coming from. But I also tried to create vEdge manually on vBond without the subject-serial-number, it didn't help
OVEL-VBOND1# show orchestrator connections
PEER PEER
PEER PEER PEER SITE DOMAIN PEER PRIVATE PEER PUBLIC ORGANIZATION
INSTANCE TYPE PROTOCOL SYSTEM IP ID ID PRIVATE IP PORT PUBLIC IP PORT REMOTE COLOR STATE NAME UPTIME
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
0 vsmart dtls 1.1.255.13 255 1 1.1.0.13 12346 1.1.0.13 12346 default up OVEL Lab 0:00:26:17
0 vsmart dtls 1.1.255.13 255 1 1.1.0.13 12446 1.1.0.13 12446 default up OVEL Lab 0:00:26:17
0 vmanage dtls 1.1.255.11 255 0 1.1.0.11 12346 1.1.0.11 12346 default up OVEL Lab 0:00:26:19
0 vmanage dtls 1.1.255.11 255 0 1.1.0.11 12446 1.1.0.11 12446 default up OVEL Lab 0:00:26:21
0 vmanage dtls 1.1.255.11 255 0 1.1.0.11 12546 1.1.0.11 12546 default up OVEL Lab 0:00:26:21
0 vmanage dtls 1.1.255.11 255 0 1.1.0.11 12646 1.1.0.11 12646 default up OVEL Lab 0:00:26:21
connection-history
PEER PEER PEER
PEER PEER PEER SITE DOMAIN PEER PRIVATE PEER PUBLIC REPEAT
INSTANCE TYPE PROTOCOL SYSTEM IP ID ID PRIVATE IP PORT PUBLIC IP PORT REMOTE COLOR STATE LOCAL/REMOTE COUNT DOWNTIME
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
0 unknown dtls - 0 0 :: 0 1.1.1.2 12386 default tear_down BIDNTVRFD/NOERR 45 2021-04-17T02:37:17+1000
vBond config:
OVEL-VBOND1# sh run system host-name OVEL-VBOND1 system-ip 1.1.255.12 site-id 255 admin-tech-on-failure no route-consistency-check no vrrp-advt-with-phymac organization-name "OVEL Lab" clock timezone Australia/Brisbane vbond 1.1.0.12 local aaa auth-order local radius tacacs usergroup basic task system read write task interface read write ! usergroup netadmin ! usergroup operator task system read task interface read task policy read task routing read task security read ! usergroup tenantadmin ! user admin password $6$54082b12c893a22c$x.4TxtWCjpCqKZV8TYSbC6.5P/G8pST1LxlBK/6tBGkC9jk0rlyF5StAukHx8OmX4x/zV.b/Ekb32cg5kGezI0 ! user ciscotacro description CiscoTACReadOnly group operator status enabled ! user ciscotacrw description CiscoTACReadWrite group netadmin status enabled ! ! logging disk enable ! ! ntp parent no enable stratum 5 exit server 1.1.0.1 version 4 prefer exit ! support zbfw-tcp-finwait-time 30 zbfw-tcp-idle-time 3600 zbfw-tcp-synwait-time 30 zbfw-udp-idle-time 30 ! ! omp no shutdown graceful-restart advertise connected advertise static ! security ipsec authentication-type ah-sha1-hmac sha1-hmac ! ! vpn 0 dns 8.8.8.8 primary interface ge0/0 ip address 1.1.0.12/24 ipv6 dhcp-client tunnel-interface encapsulation ipsec allow-service all no allow-service bgp allow-service dhcp allow-service dns allow-service icmp no allow-service sshd no allow-service netconf no allow-service ntp no allow-service ospf no allow-service stun allow-service https ! no shutdown ! ip route 0.0.0.0/0 1.1.0.1 ! vpn 512 interface eth0 ip dhcp-client ipv6 dhcp-client shutdown ! !
root cert is installed:
OVEL-VBOND1# show certificate root-ca-cert | i OVEL
Issuer: DC=LOCAL, DC=OVEL, CN=OVEL-WINSERVER-CA
Subject: DC=LOCAL, DC=OVEL, CN=OVEL-WINSERVER-CA
OVEL-VBOND1#I also tried to play with invalid/staging/valid in vmanage gui. after moving the CSRv to invalid, vmanage is no longer able to get CSRv details (because it cant make the connection to vBond). So this ste didn't help
Thank you for your help! Spent 3 days, tried everything. looks like something wrong with vBond or i'm missing somethign crucial.
Accepted Solutions
