Cisco SDWAN Lab in EVE NG - vEdges can't ping next hop

[Steven-Williams-83]

Running Images 18.4.5 of SDWAN, on latest version of EVE-NG. Have this issue where after the lab sits idle powered on for overnight, the vEdges cannot ping next hop or anything but themselves. 

I did notice that the static default route was dropping from the table and it took a reboot to fix that until I found this. 

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvp46172 

So after that the static default route is injected again but vEdge cannot ping next hop IP. 

vEdge-01# show run system host-name vEdge-01 system-ip 10.12.0.1 site-id 12 admin-tech-on-failure no route-consistency-check organization-name LAB no track-default-gateway clock timezone America/Chicago vbond 223.1.1.11 aaa auth-order local radius tacacs usergroup basic task system read write task interface read write ! usergroup netadmin ! usergroup operator task system read task interface read task policy read task routing read task security read ! usergroup tenantadmin ! logging disk enable ! ! ntp server 223.1.1.13 version 4 prefer exit ! ! omp no shutdown graceful-restart advertise connected advertise static ! ! ! vpn 0 router bgp 65012 address-family ipv4-unicast network 172.31.11.0/24 ! neighbor 172.31.11.1 no shutdown remote-as 100 address-family ipv4-unicast ! ! ! ! interface ge0/0 ip address 192.1.1.2/24 ipv6 dhcp-client tunnel-interface encapsulation ipsec color public-internet restrict allow-service all no allow-service bgp allow-service dhcp allow-service dns allow-service icmp no allow-service sshd no allow-service netconf no allow-service ntp no allow-service ospf no allow-service stun allow-service https ! no shutdown ! interface ge0/1 ip address 172.31.11.2/24 tunnel-interface encapsulation ipsec color mpls restrict allow-service all no allow-service bgp allow-service dhcp allow-service dns allow-service icmp no allow-service sshd no allow-service netconf no allow-service ntp no allow-service ospf no allow-service stun allow-service https ! no shutdown ! ip route 0.0.0.0/0 192.1.1.1 ! vpn 512 interface eth0 ip dhcp-client no shutdown ! ! vEdge-01# show ip route Codes Proto-sub-type: IA -> ospf-intra-area, IE -> ospf-inter-area, E1 -> ospf-external1, E2 -> ospf-external2, N1 -> ospf-nssa-external1, N2 -> ospf-nssa-external2, e -> bgp-external, i -> bgp-internal Codes Status flags: F -> fib, S -> selected, I -> inactive, B -> blackhole, R -> recursive PROTOCOL NEXTHOP NEXTHOP NEXTHOP VPN PREFIX PROTOCOL SUB TYPE IF NAME ADDR VPN TLOC IP COLOR ENCAP STATUS --------------------------------------------------------------------------------------------------------------------------------------------- 0 0.0.0.0/0 static - ge0/0 192.1.1.1 - - - - F,S 0 10.12.0.1/32 connected - system - - - - - F,S 0 172.31.11.0/24 connected - ge0/1 - - - - - F,S 0 192.1.1.0/24 connected - ge0/0 - - - - - F,S vEdge-01# ping vpn 0 192.1.1.1 Ping in VPN 0 PING 192.1.1.1 (192.1.1.1) 56(84) bytes of data. From 192.1.1.2 icmp_seq=1 Destination Host Unreachable From 192.1.1.2 icmp_seq=2 Destination Host Unreachable From 192.1.1.2 icmp_seq=3 Destination Host Unreachable From 192.1.1.2 icmp_seq=4 Destination Host Unreachable From 192.1.1.2 icmp_seq=5 Destination Host Unreachable ^C --- 192.1.1.1 ping statistics --- 6 packets transmitted, 0 received, +5 errors, 100% packet loss, time 4999ms pipe 4 vEdge-01# show arp vpn 0 IF IDLE VPN NAME IP MAC STATE TIMER UPTIME ----------------------------------------------------------------------- 0 ge0/0 192.1.1.2 50:00:00:05:00:01 static - 1:16:42:49 0 ge0/1 172.31.11.2 50:00:00:05:00:02 static - 1:16:42:49 vEdge-01#

INET#ping 192.1.1.1 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 192.1.1.1, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/4 ms INET#ping 192.1.1.2 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 192.1.1.2, timeout is 2 seconds: ..... Success rate is 0 percent (0/5) INET#show arp Protocol Address Age (min) Hardware Addr Type Interface Internet 101.0.0.1 - 5000.000c.0008 ARPA GigabitEthernet0/8 Internet 172.16.31.1 101 78bc.1af2.099f ARPA GigabitEthernet0/9 Internet 172.16.31.73 - 5000.000c.0009 ARPA GigabitEthernet0/9 Internet 172.31.111.1 - 5000.000c.0006 ARPA GigabitEthernet0/6.172 Internet 192.1.1.1 - 5000.000c.0001 ARPA GigabitEthernet0/1 Internet 192.1.1.2 0 5000.0005.0001 ARPA GigabitEthernet0/1 Internet 192.1.2.1 - 5000.000c.0002 ARPA GigabitEthernet0/2 Internet 192.1.2.2 0 5000.0006.0001 ARPA GigabitEthernet0/2 Internet 192.1.3.1 - 5000.000c.0003 ARPA GigabitEthernet0/3 Internet 192.1.3.2 0 5000.0007.0001 ARPA GigabitEthernet0/3 Internet 192.1.4.1 - 5000.000c.0004 ARPA GigabitEthernet0/4 Internet 192.1.4.2 0 5000.0008.0001 ARPA GigabitEthernet0/4 Internet 192.1.100.1 - 5000.000c.0000 ARPA GigabitEthernet0/0 Internet 192.1.100.2 193 5000.0003.0000 ARPA GigabitEthernet0/0 Internet 192.1.101.1 - 5000.000c.0007 ARPA GigabitEthernet0/7 Internet 192.1.111.1 - 5000.000c.0006 ARPA GigabitEthernet0/6.192 INET# show run int gi0/1 Building configuration... Current configuration : 144 bytes ! interface GigabitEthernet0/1 ip address 192.1.1.1 255.255.255.0 duplex auto speed auto media-type rj45 no mop enabled no mop sysid end INET#

So from the INET router directly connected I cannot ping the vEdge either, but it looks to have an arp entry. The vEdge does not seem to have an arp entry. 

Anyone experience this before?

[Steven-Williams-83 · Accepted solution]

So there must be an underlying issue with this version of SDWAN images. Even with configuring vedges to not track gateway using command "no track-default-gateway" under system, at some point vEdges lose connectivity to vManage within EVE-NG. The fix is I just have to reboot all the vEdges when this happens then it lasts for a day or so.