Re: iWAN not discovering WAN interfaces on Branch Router

Maksim Urmanov · ‎12-15-2017

Hello, Support community, I'm trying to deploy iWAN 2.2 in production for several branches. At this moment I have 2 Hub routers and 3 Branches with single router in each. Each routers have at least 2 connections to HUB site, for overlay networking iBGP protocol was deployed with RRs on both Hub borders. Topology picture attached.

Everything looks fine from CLI output, except SITE2 which not showing any WAN interfaces discovered in:

mia-vpnspoke01#show domain wviwan border status 

Fri Dec 15 03:49:36.112
--------------------------------------------------------------------
 **** Border Status ****

Instance Status: UP
Present status last updated: 01:09:25 ago
Loopback: Configured Loopback255 UP (10.255.2.5)
Master: 10.255.2.5
Master version: 2
Connection Status with Master: UP
MC connection info: CONNECTION SUCCESSFUL
Connected for: 01:09:24
External Collector: 10.1.110.40 port: 2055
Route-Control: Enabled
Asymmetric Routing: Disabled
Minimum Mask Length Internet: 24
Minimum Mask Length Enterprise: 24
Connection Keepalive: 5 seconds
Sampling: off
Channel Unreachable Threshold Timer: 4 seconds
Minimum Packet Loss Calculation Threshold: 15 packets
Minimum Byte Loss Calculation Threshold: 1 bytes
Monitor cache usage: 200000 (20%) Auto allocated
Minimum Requirement: Met
Smart Probe Profile:
 General Monitor:
 Current Provision Level: Master Hub
 Master Hub:
 Packets per burst: 1
 Interval(secs): 1
 Quick Monitor:
 Current Provision Level: Master Hub
 Master Hub:
 Packets per burst: 20
 Interval(secs): 1
External Wan interfaces: 

Auto Tunnel information:

 Name:Tunnel0 if_index: 22
 Virtual Template: Not Configured
 Borders reachable via this tunnel: 
--------------------------------------------------------------------
mia-vpnspoke01#show domain wviwan master status

 *** Domain MC Status ***

 Master VRF: Global

 Instance Type: Branch
 Instance id: 0
 Operational status: Up
 Configured status: Up
 Loopback IP Address: 10.255.2.5
 Load Balancing:
 Operational Status: Up
 Max Calculated Utilization Variance: 0%
 Last load balance attempt: never
 Last Reason: Variance less than 20%
 Total unbalanced bandwidth: 
 External links: 0 Kbps Internet links: 0 Kbps
 External Collector: 10.1.110.40 port: 2055
 Route Control: Enabled
 Transit Site Affinity: Enabled
 Load Sharing: Enabled
 Connection Keepalive: 5 seconds
 Mitigation mode Aggressive: Disabled
 Policy threshold variance: 20
 Minimum Mask Length Internet: 24
 Minimum Mask Length Enterprise: 24
 Syslog TCA suppress timer: 180 seconds
 Traffic-Class Ageout Timer: 5 minutes
 Minimum Packet Loss Calculation Threshold: 15 packets
 Minimum Bytes Loss Calculation Threshold: 1 bytes
 Minimum Requirement: Met

 Borders:
 IP address: 10.255.2.5
 Version: 2
 Connection status: CONNECTED (Last Updated 01:09:31 ago )
 Interfaces configured:

--------------------------------------------------------------------------------

Despite of this all loopbacks are reachable and EIGRP SAFI is formed, policies are discovered:

ow-iwanmc-01#sh eigrp service-family ipv4 neighbors 
EIGRP-SFv4 VR(#AUTOCFG#) Service-Family Neighbors for AS(59501)
H   Address                 Interface              Hold Uptime   SRTT   RTO  Q  Seq
                                                   (sec)         (ms)       Cnt Num
4   10.255.2.5              Lo255                   548 01:14:26  110   660  0  4
3   10.255.6.0              Lo255                   510 21:59:31   68   408  0  85
2   10.255.1.7              Lo255                   556 23:25:41    1   100  0  1
0   10.255.1.6              Lo255                   587 23:27:46    1   100  0  1
1   10.255.17.0             Lo255                   589 1d02h       1   100  0  5938

So what I'm missing?

Vasilii Mikhailovskii · ‎12-15-2017

Hello.

There may be a couple of reasons for the issue.

First of all I would check if CEF is enabled on mia-vpnspoke01 device (and all interfaces).

Then if Hub BR has the a channel for the site and what is the next-hop for the channel (show domain .. border channel ).

If next-hop is correct - check if Hub is sending smart-probes and if they arrive on the branch (EPC or ACL).

PS: "show tech iwan" +"show domain ... border tech" from Hub MC, Hub BR and branch device may help to investigate the issue.

Maksim Urmanov · ‎12-15-2017

Hello, Vasilii, thanks for your quick response, all outputs looks correct and I'm bit confused, now everything working on all three routers. Actually I didn't change anything after posting first message.

Requested outputs:

mia-vpnspoke01#sh ip cef summary 
IPv4 CEF is enabled for distributed and running
VRF Default
 156 prefixes (156/0 fwd/non-fwd)
 Table id 0x0
 Database epoch:        2 (156 entries at this epoch)

mia-vpnspoke01#sh ip int br | i Tun
Tunnel0                10.255.2.5      YES unset  up                    up      
Tunnel1                172.24.32.2     YES manual up                    up      
Tunnel2                172.24.33.2     YES manual up                    up      
Tunnel3                172.24.35.2     YES manual up                    up      
Tunnel4                172.24.34.2     YES manual up                    up      



ow-vpnhub01#show domain wviwan border channel dst-site-id 10.255.2.5
Border Smart Probe Stats:

Channel id: 4595
  Version : 3
  Site id : 10.255.2.5
  DSCP : default[0]
  Service provider : INET1
  Pfr-Label : 0:0 | 0:2 [0x2]
  Channel state : Initiated and open
  Channel next hop : 172.24.33.2
  RX Reachability : Reachable
  TX Reachability : Reachable
  Supports Zero-SLA : Yes
  Muted by Zero-SLA : No
  Muted by Path of Last Resort : No
  Number of Probes sent : 14928
  Number of Probes received : 14630
  Number of SMP Profile Bursts sent: 8448
  Number of Active Channel Probes sent: 845
  Number of Reachability Probes sent: 5635
  Number of Force Unreaches sent: 0
  Last Probe sent : 743 msec Ago
  Last Probe received: 423 msec ago
  Number of Data Packets sent : 0
  Number of Data Packets received : 0
  Smart Probe in Burst: No
  Smart Probe enable Burst: Yes

Channel id: 4596
  Version : 3
  Site id : 10.255.2.5
  DSCP : default[0]
  Service provider : MPLS1
  Pfr-Label : 0:0 | 0:1 [0x1]
  Channel state : Initiated and open
  Channel next hop : 172.24.32.2
  RX Reachability : Initial State
  TX Reachability : Reachable
  Supports Zero-SLA : Yes
  Muted by Zero-SLA : No
  Muted by Path of Last Resort : No
  Number of Probes sent : 6990
  Number of Probes received : 0
  Number of SMP Profile Bursts sent: 0
  Number of Active Channel Probes sent: 845
  Number of Reachability Probes sent: 6341
  Number of Force Unreaches sent: 0
  Last Probe sent : 408 msec Ago
  Last Probe received: N/A
  Number of Data Packets sent : 0
  Number of Data Packets received : 0
  Smart Probe in Burst: No
  Smart Probe enable Burst: Yes


ow-vpnhub02#show domain wviwan border channel dst-site-id 10.255.2.5
Border Smart Probe Stats:

Channel id: 4597
  Version : 3
  Site id : 10.255.2.5
  DSCP : default[0]
  Service provider : MPLS2
  Pfr-Label : 0:0 | 0:3 [0x3]
  Channel state : Initiated and open
  Channel next hop : 172.24.35.2
  RX Reachability : Initial State
  TX Reachability : Reachable
  Supports Zero-SLA : Yes
  Muted by Zero-SLA : No
  Muted by Path of Last Resort : No
  Number of Probes sent : 7003
  Number of Probes received : 0
  Number of SMP Profile Bursts sent: 0
  Number of Active Channel Probes sent: 848
  Number of Reachability Probes sent: 6368
  Number of Force Unreaches sent: 0
  Last Probe sent : 1035 msec Ago
  Last Probe received: N/A
  Number of Data Packets sent : 0
  Number of Data Packets received : 0
  Smart Probe in Burst: No
  Smart Probe enable Burst: Yes

Channel id: 4598
  Version : 3
  Site id : 10.255.2.5
  DSCP : default[0]
  Service provider : INET2
  Pfr-Label : 0:0 | 0:4 [0x4]
  Channel state : Initiated and open
  Channel next hop : 172.24.34.2
  RX Reachability : Reachable
  TX Reachability : Reachable
  Supports Zero-SLA : Yes
  Muted by Zero-SLA : No
  Muted by Path of Last Resort : No
  Number of Probes sent : 14707
  Number of Probes received : 14979
  Number of SMP Profile Bursts sent: 8487
  Number of Active Channel Probes sent: 848
  Number of Reachability Probes sent: 5660
  Number of Force Unreaches sent: 0
  Last Probe sent : 140 msec Ago
  Last Probe received: 737 msec ago
  Number of Data Packets sent : 0
  Number of Data Packets received : 0
  Smart Probe in Burst: No
  Smart Probe enable Burst: Yes

Current state:

mia-vpnspoke01#show domain wviwan master status

  *** Domain MC Status ***

 Master VRF: Global

  Instance Type:    Branch
  Instance id:      0
  Operational status:  Up
  Configured status:  Up
  Loopback IP Address: 10.255.2.5
  Load Balancing:
   Operational Status: Up
   Max Calculated Utilization Variance: 1%
   Last load balance attempt: never
   Last Reason:  Variance less than 20%
   Total unbalanced bandwidth: 
         External links: 0 Kbps  Internet links: 0 Kbps
  External Collector: 10.1.110.40 port: 2055
  Route Control: Enabled
  Transit Site Affinity: Enabled
  Load Sharing: Enabled
  Connection Keepalive: 5 seconds
  Mitigation mode Aggressive: Disabled
  Policy threshold variance: 20
  Minimum Mask Length Internet: 24
  Minimum Mask Length Enterprise: 24
  Syslog TCA suppress timer: 180 seconds
  Traffic-Class Ageout Timer: 5 minutes
  Minimum Packet Loss Calculation Threshold: 15 packets
  Minimum Bytes Loss Calculation Threshold: 1 bytes
  Minimum Requirement: Met

  Borders:
    IP address: 10.255.2.5
    Version: 2
    Connection status: CONNECTED (Last Updated 04:21:10 ago )
    Interfaces configured:
      Name: Tunnel2 | type: external | Service Provider: INET1 | Status: UP | Zero-SLA: NO | Path of Last Resort: Disabled
          Number of default Channels: 0

          Path-id list: 0:2

      Name: Tunnel4 | type: external | Service Provider: INET2 | Status: UP | Zero-SLA: NO | Path of Last Resort: Disabled
          Number of default Channels: 0

          Path-id list: 0:4

      Name: Tunnel1 | type: external | Service Provider: MPLS1 | Status: UP | Zero-SLA: NO | Path of Last Resort: Disabled
          Number of default Channels: 0

          Path-id list: 0:1

      Name: Tunnel3 | type: external | Service Provider: MPLS2 | Status: UP | Zero-SLA: NO | Path of Last Resort: Disabled
          Number of default Channels: 0

          Path-id list: 0:3

    Tunnel if: Tunnel0

--------------------------------------------------------------------------------
mia-vpnspoke01#show domain wviwan master traffic-classes summary 

APP - APPLICATION, TC-ID - TRAFFIC-CLASS-ID, APP-ID - APPLICATION-ID
Current-EXIT - Service-Provider(PFR-label)/Border/Interface(Channel-ID) 
UC - UNCONTROLLED, PE - PICK-EXIT, CN - CONTROLLED, UK - UNKNOWN

Dst-Site-Pfx      Dst-Site-Id       State DSCP        TC-ID APP-ID    APP           Current-Exit

10.1.110.0/24     10.255.1.8        CN    default[0]  10    N/A       N/A           INET2(0:4|0:0)/10.255.2.5/Tu4(Ch:570)
 Total Traffic Classes: 1 Site: 1  Internet: 0

Very weired situation, I didn't hit such problem during my lab testing.

You mentioned to check smart-probes, but is it possible to check how they're sending using some debug command or EPC/ACL is only option?

Thank you in advance

Vasilii Mikhailovskii · ‎12-15-2017

Hello,

It looks working, but not 100%.

On two Hub BRs we see:

Number of Probes received : 0

It means the branch is not sending probes back.

This could be due to routing issue on the branch.

Check "show domain ... border channel"(focus on next-hop) from the branch to investigate further.

PS: smart-probes are the best to capture with EPC, as debugs would be much more intrusive method.

Maksim Urmanov · ‎12-15-2017

Thanks for your explanation, I see what are you talking about, I tried to capture traffic on both tunnels on HUB2 and I see packets are flowing in and out, mia-vpnspoke01.int is receiving probes somehow. Output below:

mia-vpnspoke01#sh domain wviwan border channels  
Border Smart Probe Stats:

Channel id: 567
  Version : 3
  Site id : 10.255.1.8
  DSCP : default[0]
  Service provider : INET1
  Pfr-Label : 0:2 | 0:0 [0x20000]
  Channel state : Initiated and open
  Channel next hop : 172.24.33.1
  RX Reachability : Reachable
  TX Reachability : Reachable
  Supports Zero-SLA : Yes
  Muted by Zero-SLA : No
  Muted by Path of Last Resort : No
  Number of Probes sent : 47992
  Number of Probes received : 48116
  Number of SMP Profile Bursts sent: 27368
  Number of Active Channel Probes sent: 2738
  Number of Reachability Probes sent: 18648
  Number of Force Unreaches sent: 0
  Last Probe sent : 70 msec Ago
  Last Probe received: 502 msec ago
  Number of Data Packets sent : 0
  Number of Data Packets received : 0
  Smart Probe in Burst: No
  Smart Probe enable Burst: Yes

Channel id: 570
  Version : 3
  Site id : 10.255.1.8
  DSCP : default[0]
  Service provider : INET2
  Pfr-Label : 0:4 | 0:0 [0x40000]
  Channel state : Initiated and open
  Channel next hop : 172.24.34.1
  RX Reachability : Reachable
  TX Reachability : Reachable
  Supports Zero-SLA : Yes
  Muted by Zero-SLA : No
  Muted by Path of Last Resort : No
  Number of Probes sent : 48750
  Number of Probes received : 47766
  Number of SMP Profile Bursts sent: 27365
  Number of Active Channel Probes sent: 2737
  Number of Reachability Probes sent: 18648
  Number of Force Unreaches sent: 0
  Last Probe sent : 659 msec Ago
  Last Probe received: 150 msec ago
  Number of Data Packets sent : 0
  Number of Data Packets received : 0
  Smart Probe in Burst: No
  Smart Probe enable Burst: Yes

Channel id: 573
  Version : 3
  Site id : 10.255.1.8
  DSCP : default[0]
  Service provider : MPLS1
  Pfr-Label : 0:1 | 0:0 [0x10000]
  Channel state : Initiated and open
  Channel next hop : 172.24.32.1
  RX Reachability : Reachable
  TX Reachability : Reachable
  Supports Zero-SLA : Yes
  Muted by Zero-SLA : No
  Muted by Path of Last Resort : No
  Number of Probes sent : 47988
  Number of Probes received : 22650
  Number of SMP Profile Bursts sent: 27368
  Number of Active Channel Probes sent: 2737
  Number of Reachability Probes sent: 18644
  Number of Force Unreaches sent: 0
  Last Probe sent : 326 msec Ago
  Last Probe received: 1207 msec ago
  Number of Data Packets sent : 0
  Number of Data Packets received : 0
  Smart Probe in Burst: No
  Smart Probe enable Burst: Yes

Channel id: 578
  Version : 3
  Site id : 10.255.1.8
  DSCP : default[0]
  Service provider : MPLS2
  Pfr-Label : 0:3 | 0:0 [0x30000]
  Channel state : Initiated and open
  Channel next hop : 172.24.35.1
  RX Reachability : Reachable
  TX Reachability : Reachable
  Supports Zero-SLA : Yes
  Muted by Zero-SLA : No
  Muted by Path of Last Resort : No
  Number of Probes sent : 48562
  Number of Probes received : 22596
  Number of SMP Profile Bursts sent: 27358
  Number of Active Channel Probes sent: 2736
  Number of Reachability Probes sent: 18634
  Number of Force Unreaches sent: 0
  Last Probe sent : 400 msec Ago
  Last Probe received: 1102 msec ago
  Number of Data Packets sent : 0
  Number of Data Packets received : 0
  Smart Probe in Burst: No
  Smart Probe enable Burst: Yes

HUB MC loopback is reachable and visible from all 4 tunnels, but best path is toward Tun1 HUB1

mia-vpnspoke01#sh ip bgp 10.255.1.8          
BGP routing table entry for 10.255.1.8/32, version 7761
Paths: (4 available, best #4, table default)
  Not advertised to any peer
  Refresh Epoch 2
  Local, (received & used)
    172.24.34.1 from 172.24.34.1 (10.255.1.7)
      Origin incomplete, metric 2, localpref 100, valid, internal, secondary path
      rx pathid: 0, tx pathid: 0
  Refresh Epoch 2
  Local, (received & used)
    172.24.35.1 from 172.24.35.1 (10.255.1.7)
      Origin incomplete, metric 2, localpref 100, valid, internal, secondary path
      rx pathid: 0, tx pathid: 0
  Refresh Epoch 3
  Local, (received & used)
    172.24.33.1 from 172.24.33.1 (10.255.1.6)
      Origin incomplete, metric 2, localpref 100, valid, internal, secondary path
      rx pathid: 0, tx pathid: 0
  Refresh Epoch 8
  Local, (received & used)
    172.24.32.1 from 172.24.32.1 (10.255.1.6)
      Origin incomplete, metric 2, localpref 100, valid, internal, best
      rx pathid: 0, tx pathid: 0x0


mia-vpnspoke01#traceroute 10.255.1.8 sou lo255
Type escape sequence to abort.
Tracing the route to 10.255.1.8
VRF info: (vrf in name/id, vrf out name/id)
  1 172.24.32.1 112 msec 112 msec 113 msec
  2 10.1.113.130 112 msec 112 msec 113 msec
  3 10.1.111.6 113 msec *  113 msec

HUB BR loopbacks are available from correct tunnels as well:

mia-vpnspoke01#traceroute 10.255.1.6 sou lo255
Type escape sequence to abort.
Tracing the route to 10.255.1.6
VRF info: (vrf in name/id, vrf out name/id)
  1 172.24.32.1 113 msec *  112 msec
mia-vpnspoke01#traceroute 10.255.1.7 sou lo255
Type escape sequence to abort.
Tracing the route to 10.255.1.7
VRF info: (vrf in name/id, vrf out name/id)
  1 172.24.34.1 60 msec *  60 msec

Here is pcaps fomr both tunnels ( couldn't attach them for some reason )

Link for pcaps to Google Drive

Can you suggest me any doc related to iWAN troubleshooting and how smart probes should behave in iWAN?

Maksim Urmanov · ‎12-15-2017

Anyway I'll additionally double check my redistribution rules and route-maps between those routers

Maksim Urmanov · ‎12-22-2017

Finally decided to upgrade routers firmware and looks like it fixed the issue.

Software version used:

#sh ver | i IOS.*Version
Cisco IOS XE Software, Version 16.03.05
Cisco IOS Software [Denali], ASR1000 Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 16.3.5, RELEASE SOFTWARE (fc1)