cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
1316
Views
8
Helpful
27
Replies

Unexpected Traffic Routing Through SD-WAN Router despite AAR Configu

Although we have configured all traffic to route through Private1(SD-WAN Router 1) using AAR, there is still some traffic passing through biz-int. We have built the SD-WAN environment with the configuration shown in the attached image.

network_architecture.JPG

On the vSmart, we have implemented the following settings to ensure that all data traffic is routed through Private1 using AAR.

app-route-policy Site01_AAR

  vpn-list VPN3301

   sequence 9999

    action

     count AAR_Sequence_9999

     sla-class SLA_Class_0 preferred-color private1

policy

 sla-class SLA_Class_0

  loss 100

 !


However, when checking with Zabbix (a network monitoring tool), we observed that there is traffic (about 20Mbps) passing through the WAN(Biz-internet) interface of SD-WAN Router 2.

Zabbix.JPG

Our expectation was that all traffic would be routed through the WAN of SD-WAN Router 1.

 

Do you have any idea why this might be happening? Could the traffic bursts observed in Zabbix be related to this issue?

27 Replies 27

dijix1990_0-1724389024027.png

sh flow monitor sdwan_flow_monitor cache format table
  Cache type:                               Normal (Platform cache)
  Cache size:                                32768
  Current entries:                               3
  High Watermark:                               11

  Flows added:                                  35
  Flows aged:                                   32
    - Active timeout      (    60 secs)          1
    - Inactive timeout    (    10 secs)         31

IPV4 SRC ADDR    IPV4 DST ADDR    TRNS SRC PORT  TRNS DST PORT   IP VPN ID  IP PROT  tcp flags  intf input            intf output                     bytes long             pkts long  time abs first  time abs last  flow end reason       conn initiator     intf overlay session id input  intf overlay session id output  conn conn id long    drop cause id       bytes drop long  sdwan sla not met   sdwan preferred color not met   sdwan queue id         pkts drop long  ip dscp  app name
===============  ===============  =============  =============  ==========  =======  =========  ====================  ====================  ====================  ====================  ==============  =============  ====================  =================  =============================  ==============================  ===================  =============  ====================  ==================  ==============================  ===============  ====================  =======  ================================
172.18.62.16     172.26.32.1                  0           2048          15        1  0x02       Gi0/0/1.936           Null                                   252                     3    07:57:30.456   07:57:32.494  Not determined        Reverse initiator                             19                               0  0x00000000000074CD               0                     0                   0                               0                2                     0  0x00     layer7 ping
172.18.7.22      172.26.32.1                  0           2048          15        1  0x02       Gi0/0/1.938           Null                                  1512                    18    07:57:04.159   07:57:31.317  Not determined        Reverse initiator                              8                               0  0x00000000000074CB               0                     0                   0                               0                2                     0  0x00     layer7 ping

First check counter in path vedge/cedge router 

Second use 

Show sdwan policy service-path vpn 3301 ....

To see if both edges routers use same path or not 

Share show sdwan policy service-path here 

Thanks alot 

MHM


@MHM Cisco World wrote:

First check counter in path vedge/cedge router 

Second use 

Show sdwan policy service-path vpn 3301 ....

To see if both edges routers use same path or not 

Share show sdwan policy service-path here 

Thanks alot 

MHM


It was before, from Web gui, aar seems to be  correct, I suggested to try to check via netflow, and maybe it's bug

Thank you!!
I checked the GUI and confirmed that no Localized policy was created.
It seems that the configuration was done via CLI (as described above).
When I checked several other routers with the same Netflow settings, I found that some display Cache information while others do not.
Additionally, it appears that Netflow data has not been reaching Solarwinds (the traffic monitoring tool) from both SD-WAN-Router1 and 2 for several months, so there might indeed be some kind of bug, as you mentioned.

We can not check netflow data for these router, do you have any idea that we can check?

Thank you for your comment!
I did show sdwan policy service-path.
And, both router use private1.

yutashimamura2920_0-1724429441119.png

 

 Let me check one point 

MHM

Which point?

@dijix1990 

I could see the cache info for SD-WAN Router 1, bur no cache entry for SD-WAN Router 2..

SD-WAN-Roouter1#show flow monitor sdwan_flow_monitor cache format table 
  Cache type:                               Normal (Platform cache)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
  Cache size:                                80000                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
  Current entries:                              55                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
  High Watermark:                              348                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
  Flows added:                            21512868                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
  Flows aged:                             21512813                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
    - Active timeout      (    60 secs)     379891                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
    - Inactive timeout    (    10 secs)   21132922                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
IPV4 SRC ADDR    IPV4 DST ADDR    TRNS SRC PORT  TRNS DST PORT   IP VPN ID  IP PROT  tcp flags  intf input            intf output           flow sampler id            bytes long             pkts long  time abs first  time abs last  flow end reason       intf overlay session id input  intf overlay session id output  conn conn id long    drop cause id       bytes drop long  sdwan sla not met   sdwan preferred color not met   sdwan queue id         pkts drop long  ip dscp  app name                                                                                                                                                                                                                                                                                                                           
===============  ===============  =============  =============  ==========  =======  =========  ====================  ====================  ===============  ====================  ====================  ==============  =============  ====================  =============================  ==============================  ===================  =============  ====================  ==================  ==============================  ===============  ====================  =======  ================================                                                                                                                                                                                                                                                                                                   
192.168.210.33   192.168.228.241           8305          43029        3301        6  0x10       Vl1427                Gi0/0/0                             4                   104                     2    04:04:48.052   04:04:57.734  Not determined                                    0                             257  0xEA207930000A6CE5               0                     0                   0                               0                2                     0  0x00     layer7 ssl                                                                                                                                                                                                                                                                                                                         
10.242.79.210    10.240.8.17              61386             80        3301        6  0x10       Vl1427                Gi0/0/0                             4                    80                     2    04:04:47.779   04:04:57.420  Not determined                                    0                             250  0xEA67FE90000EA6E2               0                     0                   0                               0                2                     0  0x00     layer7 ms-sms                                                                                                                                                                                                                                                                                                                      
192.168.210.105  192.168.199.146            443          49675        3301        6  0x10       Vl1427                Gi0/0/0                             4                 67104                    48    04:04:22.837   04:04:59.893  Not determined                                    0                             257  0xEA9763500018A5E2               0                     0                   0                               0                2                     0  0x00     layer7 ssl                                                                                                                                                                                                                                                                                                                         
192.168.210.2    10.240.58.38                 0              0        3301        1  0x00       Vl1427                Gi0/0/0                             4                   168                     2    04:04:51.554   04:04:56.958  Not determined                                    0                             255  0x00000000002D992E               0                     0                   0                               0                2                     0  0x00     layer7 ping                                                                                                                                                                                                                                                                                                                        
10.242.75.5      10.242.82.8              23711           3777        3301        6  0x18       Vl1427                Gi0/0/0                             4                  2224                     2    04:04:55.652   04:04:58.133  Not determined                                    0                             269  0xEA4299300015F412               0                     0                   0                               0                2                     0  0x00     layer7 statistical-p2p                                                                                                                                                                                                                                                                                                             
192.168.214.253  192.168.210.33           42356           8305        3301        6  0x10       Gi0/0/0               Vl1427                              4                  1398                     1    04:04:56.892   04:04:56.892  Not determined                                  255                               0  0xEA4197F0000CEF52               0                     0                   0                               0                2                     0  0x00     layer7 unknown                                                                                                                                                                                                                                                                                                                     
10.240.8.17      10.242.79.210               80          61386        3301        6  0x10       Gi0/0/0               Vl1427                              4                  2600                     2    04:04:48.272   04:04:55.646  Not determined                                  250                               0  0xEA67FE90000EA6E2               0                     0                   0                               0                2                     0  0x00     layer7 ms-sms                                                                                                                                                                                                                                                                                                                      
10.242.73.3      10.240.164.20               53          50355        3301       17  0x00       Vl1427                Gi0/0/0                             4                   338                     1    04:04:59.037   04:04:59.037  Not determined                                    0                             249  0xEA2185600007678D               0                     0                   0                               0                2                     0  0x00     layer7 ms-live-accounts                                                                                                                                                                                                                                                                                                            
10.240.58.38     192.168.210.2                0           2048        3301        1  0x00       Gi0/0/0               Vl1427                              4                    84                     1    04:05:00.190   04:05:00.190  Not determined                                  255                               0  0x00000000002D9932               0                     0                   0                               0                2                     0  0x00     layer7 ping                                                                                                                                                                                                                                                                                                                        
192.168.210.60   192.168.196.61           49964            161        3301       17  0x00       Vl1427                Gi0/0/0                             4                    82                     1    04:04:51.879   04:04:51.879  Not determined                                    0                             250  0xEA4784B000192517               0                     0                   0                               0                2                     0  0x00     port snmp                                                                                                                                                                                                                                                                                                                          
10.242.76.66     192.168.196.125          59251            445        3301        6  0x10       Vl1427                Gi0/0/0                             4                    41                     1    04:04:56.186   04:04:56.186  Not determined                                    0                             250  0xEFAEF5600013F63F               0                     0                   0                               0                2                     0  0x00     layer7 cifs                                                                                                                                                                                                                                                                                                                        
192.168.210.33   10.241.119.220            8305          49459        3301        6  0x10       Vl1427                Gi0/0/0                             4                    52                     1    04:04:58.495   04:04:58.495  Not determined                                    0                             258  0xEA42DA70000D5F3D               0                     0                   0                               0                2                     0  0x00     layer7 unknown                                                                                                                                                                                                                                                                                                                     
192.168.238.5    192.168.210.60               0              0        3301        1  0x00       Gi0/0/0               Vl1427                              4                    51                     1    04:04:57.933   04:04:57.933  Not determined                                  257                               0  0x00000000002D9931               0                     0                   0                               0                2                     0  0x00     layer7 ping                                                                                                                                                                                                                                                                                                                        
192.168.210.33   192.168.214.253           8305          42356        3301        6  0x10       Vl1427                Gi0/0/0                             4                    52                     1    04:04:52.380   04:04:52.380  Not determined                                    0                             255  0xEA4197F0000CEF52               0                     0                   0                               0                2                     0  0x00     layer7 unknown                                                                                                                                                                                                                                                                                                                     
10.242.82.8      10.242.75.5               3777          23711        3301        6  0x18       Gi0/0/0               Vl1427                              4                   156                     3    04:04:50.659   04:04:58.166  Not determined                                  269                               0  0xEA4299300015F412               0                     0                   0                               0                2                     0  0x00     layer7 statistical-p2p                                                                                                                                                                                                                                                                                                             
10.242.73.1      10.241.96.113             5246           5252        3301       17  0x00       Vl1427                Gi0/0/0                             4                   125                     1    04:04:50.816   04:04:50.816  Not determined                                    0                             257  0xE9EDC400001770D7               0                     0                   0                               0                2                     0  0x30     port capwap-control             
10.242.104.125   10.242.73.1               5256           5247        3301       17  0x00       Gi0/0/1.30            Vl1427                              4                   195                     1    04:04:56.369   04:04:56.369  Not determined                                  135                               0  0xE9F461F00017FD25               0                     0                   0                               0                2                     0  0x30     port capwap-data                                                                                                                                                                                                                                                                                                                   
192.168.210.60   10.240.65.11             53994            161        3301       17  0x00       Vl1427                Gi0/0/0                             4                    83                     1    04:04:59.973   04:04:59.973  Not determined                                    0                             260  0xF38E9F0000064CFC               0                     0                   0                               0                2                     0  0x00     port snmp                                                                                                                                                                                                                                                                                                                          
192.168.210.33   10.128.251.9              8305          55029        3301        6  0x10       Vl1427                Gi0/0/0                             4                    52                     1    04:04:54.199   04:04:54.199  Not determined                                    0                             254  0xF3A524200007D097               0                     0                   0                               0                2                     0  0x00     layer7 ssl                                                                                                                                                                                                                                                                                                                         
192.168.244.253  192.168.210.33           57709           8305        3301        6  0x10       Gi0/0/0               Vl1427                              4                  1398                     1    04:04:54.818   04:04:54.818  Not determined                                  245                               0  0xEA7B97900018FB23               0                     0                   0                               0                2                     0  0x00     layer7 ssl                                                                                                                                                                                                                                                                                                                         
192.168.210.2    10.240.69.43                 0              0        3301        1  0x00       Vl1427                Gi0/0/0                             4                    84                     1    04:04:55.313   04:04:55.313  Not determined                                    0                             260  0x00000000002D992F               0                     0                   0                               0                2                     0  0x00     layer7 ping                                                                                                                                                                                                                                                                                                                        
10.241.96.2      10.242.79.3                 22          44884        3301        6  0x18       Gi0/0/0               Vl1427                              4                   912                     3    04:04:54.418   04:04:57.512  Not determined                                  257                               0  0xEA4EA2A00018CC48               0                     0                   0                               0                2                     0  0x30     port ssh                                                                                                                                                                                                                                                                                                                           
10.242.79.3      192.168.199.80           49109            161        3301       17  0x00       Vl1427                Gi0/0/0                             4                   181                     1    04:04:53.321   04:04:53.321  Not determined                                    0                             257  0xEA8CE2A0001062C5               0                     0                   0                               0                2                     0  0x00     port snmp                                                                                                                                                                                                                                                                                                                          
10.240.58.84     192.168.210.2                0           2048        3301        1  0x00       Gi0/0/0               Vl1427                              4                    84                     1    04:04:57.229   04:04:57.229  Not determined                                  255                               0  0x00000000002D9930               0                     0                   0                               0                2                     0  0x00     layer7 ping                                                                                                                                                                                                                                                                                                                        
10.242.79.3      10.241.96.2              44884             22        3301        6  0x10       Vl1427                Gi0/0/0                             4                    40                     1    04:04:58.452   04:04:58.452  Not determined                                    0                             257  0xEA4EA2A00018CC48               0                     0                   0                               0                2                     0  0x04     port ssh                                                                                                                                                                                                                                                                                                                           
192.168.210.2    10.240.44.6                 80          61228        3301        6  0x11       Vl1427                Gi0/0/0                             4                    40                     1    04:05:00.564   04:05:00.564  Not determined                                    0                             255  0xEA5C1E800001DDD2               0                     0                   0                               0                2                     0  0x00     layer7 oracle-ebsuite-unsecured                                                                                                                                                                                                                                                                                                    
10.241.250.22    192.168.210.105            445          55486        3301        6  0x18       Gi0/0/0               Vl1427                              4                   360                     1    04:04:56.180   04:04:56.180  Not determined                                  270                               0  0xEA6B8D0000094936               0                     0                   0                               0                2                     0  0x00     layer7 cifs                                                                                                                                                                                                                                                                                                                        
10.247.247.73    10.242.73.200            53511            445        3301        6  0x18       Gi0/0/0               Vl1427                              4                   328                     1    04:04:52.671   04:04:52.671  Not determined                                  248                               0  0xEA76AA0000194999               0                     0                   0                               0                2                     0  0x00     layer7 cifs                                                                                                                                                                                                                                                                                                                        
10.241.96.15     10.242.79.3                 22          34172        3301        6  0x10       Gi0/0/0               Vl1427                              4                    40                     1    04:04:51.567   04:04:51.567  Not determined                                  257                               0  0xEA803FB0001B3F9C               0                     0                   0                               0                2                     0  0x30     port ssh                                                                                                                                                                                                                                                                                                                           
192.168.214.141  10.242.73.6                443          42812        3301        6  0x10       Gi0/0/0               Vl1427                              4                   968                     1    04:04:53.705   04:04:53.705  Not determined                                  255                               0  0xEA453A30000A8162               0                     0                   0                               0                2                     0  0x00     layer7 vmware-vsphere                                                                                                                                                                                                                                                                                                              
192.168.210.60   10.240.8.28              45820          55791        3301        6  0x18       Vl1427                Gi0/0/0                             4                   196                     1    04:04:53.082   04:04:53.082  Not determined                                    0                             250  0xF08B7510000698CA               0                     0                   0                               0                2                     0  0x00     layer7 ms-rpc                                                                                                                                                                                                                                                                                                                      
10.240.8.27      10.242.79.204              445          54811        3301        6  0x18       Gi0/0/0               Vl1427                              4                   144                     1    04:04:58.390   04:04:58.390  Not determined                                  250                               0  0xF459340000145CBA               0                     0                   0                               0                2                     0  0x00     layer7 cifs                                                                                                                                                                                                                                                                                                                        
10.240.8.16      10.242.76.79             65172          50485        3301        6  0x10       Gi0/0/0               Vl1427                              4                    40                     1    04:04:59.687   04:04:59.687  Not determined                                  250                               0  0xF39CE69000054D73               0                     0                   0                               0                2                     0  0x00     layer7 ms-netlogon              
10.80.254.10     10.242.76.32               443          52720        3301        6  0x10       Gi0/0/0               Vl1427                              4                    40                     1    04:04:51.056   04:04:51.056  Not determined                                  246                               0  0xEA98A440000FA42E               0                     0                   0                               0                2                     0  0x00     layer7 ssl                                                                                                                                                                                                                                                                                                                         
192.168.199.146  192.168.210.105          49675            443        3301        6  0x10       Gi0/0/0               Vl1427                              4                   520                    13    04:04:47.092   04:04:58.976  Not determined                                  257                               0  0xEA9763500018A5E2               0                     0                   0                               0                2                     0  0x00     layer7 ssl                                                                                                                                                                                                                                                                                                                         
10.242.73.1      10.241.96.134             5246           5248        3301       17  0x00       Vl1427                Gi0/0/0                             4                   109                     1    04:04:55.176   04:04:55.176  Not determined                                    0                             257  0xF470F70000132F23               0                     0                   0                               0                2                     0  0x30     port capwap-control                                                                                                                                                                                                                                                                                                                
192.168.210.33   192.168.196.252           8305          44911        3301        6  0x10       Vl1427                Gi0/0/0                             4                    52                     1    04:05:00.364   04:05:00.364  Not determined                                    0                             250  0xF399E4F00010D3BC               0                     0                   0                               0                2                     0  0x00     layer7 ssl                                                                                                                                                                                                                                                                                                                         
192.168.223.54   10.242.73.3              60675             53        3301       17  0x00       Gi0/0/0               Vl1427                              4                    58                     1    04:04:55.965   04:04:55.965  Not determined                                  251                               0  0xEA8C8DE0000B964F               0                     0                   0                               0                2                     0  0x00     layer7 bing                                                                                                                                                                                                                                                                                                                        
10.242.73.1      10.242.104.145            5246           5248        3301       17  0x00       Vl1427                Gi0/0/1.30                          4                   109                     1    04:04:51.462   04:04:51.462  Not determined                                    0                             135  0xF321FD20001499F4               0                     0                   0                               1                2                     0  0x30     port capwap-control                                                                                                                                                                                                                                                                                                                
192.168.210.2    192.168.202.207             80          59185        3301        6  0x10       Vl1427                Gi0/0/0                             4                  1398                     1    04:04:57.828   04:04:57.828  Not determined                                    0                             255  0xF38FD580000E4E0A               0                     0                   0                               0                2                     0  0x00     layer7 oracle-ebsuite-unsecured                                                                                                                                                                                                                                                                                                    
192.168.194.150  10.242.73.6                443          41714        3301        6  0x10       Gi0/0/0               Vl1427                              4                    52                     1    04:04:59.904   04:04:59.904  Not determined                                  259                               0  0xEA8961F000116323               0                     0                   0                               0                2                     0  0x00     layer7 vmware-vsphere                                                                                                                                                                                                                                                                                                              
192.168.228.241  192.168.210.33           43029           8305        3301        6  0x10       Gi0/0/0               Vl1427                              4                  1398                     1    04:04:59.327   04:04:59.327  Not determined                                  257                               0  0xEA207930000A6CE5               0                     0                   0                               0                2                     0  0x00     layer7 ssl                                                                                                                                                                                                                                                                                                                         
192.168.194.84   192.168.210.60             135          45791        3301        6  0x10       Gi0/0/0               Vl1427                              4                    40                     1    04:04:51.513   04:04:51.513  Not determined                                  259                               0  0xF38F574000122619               0                     0                   0                               0                2                     0  0x00     layer7 ms-rpc                                                                                                                                                                                                                                                                                                                      
                          
SD-WAN-Roouter1


SD-WAN-Roouter2#show flow monitor sdwan_flow_monitor cache format table                                                                                                                                                                   
  Cache type:                               Normal (Platform cache)
  Cache size:                                80000                                                                                                                                                                                                   
  Current entries:                               0                                                                                                                                                                                                   
  High Watermark:                               75                                                                                                                                                                                                   

  Flows added:                              296232                                                                                                                                                                                                   
  Flows aged:                               296232                                                                                                                                                                                                   
    - Active timeout      (    60 secs)         23                                                                                                                                                                                                   
    - Inactive timeout    (    10 secs)     296209                                                                                                                                                                                                   

There are no cache entries to display.                                                                                                                                                                                                               

SD-WAN-Roouter2#

dijix1990
VIP
VIP

Today I bumped into a bug with aar

I had a rule (aar) for some sites and for isr1000/4000 it works, but when I added new site to rule (with c8200) I found that it doesn't work for this model, so I already haven't surprised , sdwan from cisco has extremely a lot of bugs

Thank you for your support!
Regarding NetFlow, there might be a bug, so I will open a case with Cisco to confirm. For the current issue, where traffic is flowing through the biz-internet despite AAR being configured to route all traffic through MPLS, I would appreciate it if you could provide any key points to check for investigating the cause.

Each edge in local site use mpls to forward traffic remote site' 

The tloc extensions is use so one edge router is use mpls to forward traffic.

But without color strict then remote site will use 

Public-private color which is more prefer than private-private

Using strict in remote site will solve issue and traffic both way will pass through mpls 

MHM

Thank you for your comment!
We use group ID for color restrict.
Private 1 is only allowed to Tunnel to the opposite Private1.

Hi,

how is your routing configured? There might be case, for some remote subnets you have preferred (or only) route via biz-int and AAR is not subject for this destination, thus you have some traffic on biz-int.

HTH,
Please rate and mark as an accepted solution if you have found any of the information provided useful.

Review Cisco Networking for a $25 gift card