04-16-2021 09:53 AM
I'm trying to build a lab based on EVE-NG, viptela controllers and CSR1000v images and It looks like i'm unable to overcome this issue.
I'm using my own enterprise root CA. All vManage, Vbond and vSmart are successfully up and running. Tried on different versions, the problem can be recreated on any available controllers and CSR1000v images.
what has been done:
0. I have all 3 controllers up and running, no issues.
1. I created a list of CSRv devices in the cisco PnP portal and downloaded the file list.
2. The list was uploded into vManage.
3. Bootstrap was generated, and uploaded into CSRv and then request platform software sdwan software reset used
4. L3 connectivity was provided and working, tunnels created CSRv was visible in vmanage.
5. The widely described way to use the command "request platform software sdwan vedge_cloud activate chassis-number" did not work. the CSRv simply were not able to communicate to vManage/vBond after that having DISTLOC.
6. I decided to use manual signing the devices certificates. Cert signed, new certificate successfully uploaded on CSRv. Once it's done, the CSRv is never able to make connection to vBond having the error as in the subject
some snippets:
CSRv:
HQ-R1#sh sdwan run
system
system-ip 172.16.0.12
domain-id 1
site-id 1
admin-tech-on-failure
organization-name "OVEL Lab"
vbond 1.1.0.12
ntp peer 1.1.0.1
interface GigabitEthernet1
no shutdown
ip address 1.1.1.2 255.255.255.0
no mop enabled
no mop sysid
negotiation auto
interface Tunnel1
no shutdown
ip unnumbered GigabitEthernet1
tunnel source GigabitEthernet1
tunnel mode sdwan
exit
clock timezone AEST 10 0
sdwan
interface GigabitEthernet1
tunnel-interface
encapsulation ipsec
allow-service all
no allow-service bgp
allow-service dhcp
allow-service dns
allow-service icmp
no allow-service sshd
no allow-service netconf
no allow-service ntp
no allow-service ospf
no allow-service stun
no allow-service snmp
exit
HQ-R1#sh sdwan control local-properties personality vedge sp-organization-name OVEL Lab organization-name OVEL Lab root-ca-chain-status Installed certificate-status Installed certificate-validity Valid certificate-not-valid-before Apr 16 14:34:48 2021 GMT certificate-not-valid-after Apr 16 14:34:48 2023 GMT enterprise-cert-status Not-Applicable enterprise-cert-validity Not Applicable enterprise-cert-not-valid-before Not Applicable enterprise-cert-not-valid-after Not Applicable dns-name 1.1.0.12 site-id 1 domain-id 1 protocol dtls tls-port 0 system-ip 172.16.0.12 chassis-num/unique-id CSR-16711BCD-4B91-4A39-E3BD-C63E6A5EC003 serial-num 100000001D3D436E39E4A99B3300000000001D token Invalid keygen-interval 1:00:00:00 retry-interval 0:00:00:16 no-activity-exp-interval 0:00:00:20 dns-cache-ttl 0:00:02:00 port-hopped TRUE time-since-last-port-hop 0:00:24:21 embargo-check success number-vbond-peers 1 INDEX IP PORT ----------------------------------------------------- 0 1.1.0.12 12346 number-active-wan-interfaces 1 NAT TYPE: E -- indicates End-point independent mapping A -- indicates Address-port dependent mapping N -- indicates Not learned Note: Requires minimum two vbonds to learn the NAT type PUBLIC PUBLIC PRIVATE PRIVATE PRIVATE MAX RESTRICT/ LAST SPI TIME NAT VM INTERFACE IPv4 PORT IPv4 IPv6 PORT VS/VM COLOR STATE CNTRL CONTROL/ LR/LB CONNECTION REMAINING TYPE CON STUN PRF --------------------------------------------------------------------------------------------------------------------------------- -------------------------------------------------------------------------- GigabitEthernet1 1.1.1.2 12386 1.1.1.2 :: 12386 0/0 default up 2 no/yes/no No/No 0:00:00:00 0:10:42:56 N 5
Root CA is imported
HQ-R1#sh sdwan certificate root-ca-cert | i OVEL Issuer: DC=LOCAL, DC=OVEL, CN=OVEL-WINSERVER-CA Subject: DC=LOCAL, DC=OVEL, CN=OVEL-WINSERVER-CA
connection-history
PEER PEER PEER SITE DOMAIN PEER PRIVATE PEER PUBLIC LOCAL REMOTE REPEAT TYPE PROTOCOL SYSTEM IP ID ID PRIVATE IP PORT PUBLIC IP PORT LOCAL COLOR STATE ERROR ERROR COUNT DOWNTIME ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ vbond dtls 0.0.0.0 0 0 1.1.0.12 12346 1.1.0.12 12346 default challenge_resp RXTRDWN BIDNTVRFD 38 2021-04-17T02:33:29+1000
obviously, L3 is up and everything is pingable
Vbond log messages:
Apr 17 02:34:20 OVEL-VBOND1 VBOND[10509]: %Viptela-OVEL-VBOND1-vbond_0-6-INFO-1400002: Notification: vbond-reject-vedge-connection severity-level:major host-name:"OVEL-VBOND1" system-ip:1.1.255.12 uuid:"CSR-16711BCD-4B91-4A39-E3BD-C63E6A5EC003" organization-name:"OVEL Lab" sp-organization-name:"OVEL Lab" reason:"ERR_BID_NOT_VERIFIED" Apr 17 02:34:20 OVEL-VBOND1 confd[992]: netconf id=21 sending notification {http://viptela.com/security}vbond-reject-vedge-connection Apr 17 02:34:20 OVEL-VBOND1 VBOND[10509]: %Viptela-OVEL-VBOND1-vbond_0-6-INFO-1400002: Notification: control-connection-auth-fail severity-level:major host-name:"OVEL-VBOND1" system-ip:1.1.255.12 personality:vbond peer-type:vedge peer-system-ip::: local-system-ip:1.1.255.12 local-color:default reason:"ERR_BID_NOT_VERIFIED" Apr 17 02:34:20 OVEL-VBOND1 confd[992]: netconf id=21 sending notification {http://viptela.com/security}control-connection-auth-fail
Valid vEdge:
OVEL-VBOND1# show orchestrator valid-vedges orchestrator valid-vedges CSR-16711BCD-4B91-4A39-E3BD-C63E6A5EC003 serial-number 100000001D3D436E39E4A99B3300000000001D validity valid org "OVEL Lab" hardware-installed-serial-number N/A subject-serial-number CSR-16711BC
not sure where subject-serial-number is coming from. But I also tried to create vEdge manually on vBond without the subject-serial-number, it didn't help
OVEL-VBOND1# show orchestrator connections PEER PEER PEER PEER PEER SITE DOMAIN PEER PRIVATE PEER PUBLIC ORGANIZATION INSTANCE TYPE PROTOCOL SYSTEM IP ID ID PRIVATE IP PORT PUBLIC IP PORT REMOTE COLOR STATE NAME UPTIME ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0 vsmart dtls 1.1.255.13 255 1 1.1.0.13 12346 1.1.0.13 12346 default up OVEL Lab 0:00:26:17 0 vsmart dtls 1.1.255.13 255 1 1.1.0.13 12446 1.1.0.13 12446 default up OVEL Lab 0:00:26:17 0 vmanage dtls 1.1.255.11 255 0 1.1.0.11 12346 1.1.0.11 12346 default up OVEL Lab 0:00:26:19 0 vmanage dtls 1.1.255.11 255 0 1.1.0.11 12446 1.1.0.11 12446 default up OVEL Lab 0:00:26:21 0 vmanage dtls 1.1.255.11 255 0 1.1.0.11 12546 1.1.0.11 12546 default up OVEL Lab 0:00:26:21 0 vmanage dtls 1.1.255.11 255 0 1.1.0.11 12646 1.1.0.11 12646 default up OVEL Lab 0:00:26:21
connection-history
PEER PEER PEER PEER PEER PEER SITE DOMAIN PEER PRIVATE PEER PUBLIC REPEAT INSTANCE TYPE PROTOCOL SYSTEM IP ID ID PRIVATE IP PORT PUBLIC IP PORT REMOTE COLOR STATE LOCAL/REMOTE COUNT DOWNTIME ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0 unknown dtls - 0 0 :: 0 1.1.1.2 12386 default tear_down BIDNTVRFD/NOERR 45 2021-04-17T02:37:17+1000
vBond config:
OVEL-VBOND1# sh run system host-name OVEL-VBOND1 system-ip 1.1.255.12 site-id 255 admin-tech-on-failure no route-consistency-check no vrrp-advt-with-phymac organization-name "OVEL Lab" clock timezone Australia/Brisbane vbond 1.1.0.12 local aaa auth-order local radius tacacs usergroup basic task system read write task interface read write ! usergroup netadmin ! usergroup operator task system read task interface read task policy read task routing read task security read ! usergroup tenantadmin ! user admin password $6$54082b12c893a22c$x.4TxtWCjpCqKZV8TYSbC6.5P/G8pST1LxlBK/6tBGkC9jk0rlyF5StAukHx8OmX4x/zV.b/Ekb32cg5kGezI0 ! user ciscotacro description CiscoTACReadOnly group operator status enabled ! user ciscotacrw description CiscoTACReadWrite group netadmin status enabled ! ! logging disk enable ! ! ntp parent no enable stratum 5 exit server 1.1.0.1 version 4 prefer exit ! support zbfw-tcp-finwait-time 30 zbfw-tcp-idle-time 3600 zbfw-tcp-synwait-time 30 zbfw-udp-idle-time 30 ! ! omp no shutdown graceful-restart advertise connected advertise static ! security ipsec authentication-type ah-sha1-hmac sha1-hmac ! ! vpn 0 dns 8.8.8.8 primary interface ge0/0 ip address 1.1.0.12/24 ipv6 dhcp-client tunnel-interface encapsulation ipsec allow-service all no allow-service bgp allow-service dhcp allow-service dns allow-service icmp no allow-service sshd no allow-service netconf no allow-service ntp no allow-service ospf no allow-service stun allow-service https ! no shutdown ! ip route 0.0.0.0/0 1.1.0.1 ! vpn 512 interface eth0 ip dhcp-client ipv6 dhcp-client shutdown ! !
root cert is installed:
OVEL-VBOND1# show certificate root-ca-cert | i OVEL Issuer: DC=LOCAL, DC=OVEL, CN=OVEL-WINSERVER-CA Subject: DC=LOCAL, DC=OVEL, CN=OVEL-WINSERVER-CA OVEL-VBOND1#
I also tried to play with invalid/staging/valid in vmanage gui. after moving the CSRv to invalid, vmanage is no longer able to get CSRv details (because it cant make the connection to vBond). So this ste didn't help
Thank you for your help! Spent 3 days, tried everything. looks like something wrong with vBond or i'm missing somethign crucial.
Solved! Go to Solution.
04-22-2021 03:57 PM - edited 04-25-2021 03:32 AM
After a week of various attempts I found the solution. Guys, make sure ALL your devices have good NTP synchronisation! This was the key!
I have put eventually 4 NTP servers into all controllers and edges configs and it started working after that.
And yes, I can prove now that it's working ok in EVE environment.
p.s. and also not sure this helps but in your EVE environment try to set your CSRv network interfaces as "virtio-net-pci" instead of vmxnet3.
04-22-2021 10:50 AM
Assuming you are not running incompatible versions of vBond/vManage and CSR1000v, check if you are not hitting https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvp75927
04-22-2021 03:57 PM - edited 04-25-2021 03:32 AM
After a week of various attempts I found the solution. Guys, make sure ALL your devices have good NTP synchronisation! This was the key!
I have put eventually 4 NTP servers into all controllers and edges configs and it started working after that.
And yes, I can prove now that it's working ok in EVE environment.
p.s. and also not sure this helps but in your EVE environment try to set your CSRv network interfaces as "virtio-net-pci" instead of vmxnet3.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide