- Cisco Community
- Technology and Support
- Networking
- SD-WAN and Cloud Networking
- DIA local breakout performance issues
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
DIA local breakout performance issues
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-18-2020 05:33 AM
Hello all,
we are running a two transport (MPLS/public internet) SDWAN network and the site interconnection is working perfectly fine.
We now wanted to shift from using the proxy that is located in the central datacenter, to use the DIA feature from SDWAN with Umbrella DNS Security (SIG is planed later on), we are doing the traffic routing via a centralized policy from vManage/vSmart that is NATin all non RF1918 or DHCP broadcast traffic directly to VPN 0 where we have NAT enabled on the public internet facing interface.
The DIA seems to be working fine and when visiting whatismyip.org it shows the IP from the cEdge Router, but the response times from websites are very slow and it seems like they are loading for ever. The strange thing is, when I am using a VPN (e.g. AnyConnect to central DC, Surfshark, Nord VPN) over the DIA and then browse through that VPN connection every thing is working fine.
I have played with different MTU / TCP Adjust mss values, disabled Umbrella, disabled firewalling at all, tried several different DNS servers, did a downgrade from 17.3 to 16.12, but nothing changed.
When running the CLI command "show sdwan app-fwd dpi flows format table" I see some connections beeing dropped because of "IpFragErr" and found a bug mentioning to enable path mtu discovery, but still no luck: https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvt50136
decgnsdwan01#show sdwan app-fwd dpi flows format table
Generating output, this might take time, please wait ...
PKT PKT PKT PKT
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvt50136 TCP SLA COLOR FEC FEC DUP D DUP D DUP CXP
SRC DEST IP CNTRL ICMP TOTAL TOTAL DROP DROP NOT NOT QUEUE D R PKTS PKTS R D
VPN SRC IP DEST IP PORT PORT DSCP PROTO BITS OPCODE PKTS BYTES START TIME EGRESS INTF NAME INGRESS INTF NAME APPLICATION FAMILY DROP CAUSE OCTETS PACKETS MET MET ID PKTS PKTS ORIG DUP PKTS PKTS
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
10 34.231.93.203 10.79.0.4 443 38756 0 6 26 0 11 13100 Fri Dec 18 12:23:03 2020 GigabitEthernet0/0/2.100 GigabitEthernet0/0/0 web-analytics web IpFragErr 10500 7 0 0 2 0 0 0 0 0 0
10 52.23.144.54 10.79.0.4 443 48142 0 6 26 0 11 13070 Fri Dec 18 12:23:03 2020 GigabitEthernet0/0/2.100 GigabitEthernet0/0/0 ssl encrypted IpFragErr 12000 8 0 0 2 0 0 0 0 0 0
10 23.2.13.227 10.79.0.4 443 45058 0 6 26 0 22 21146 Fri Dec 18 12:22:47 2020 GigabitEthernet0/0/2.100 GigabitEthernet0/0/0 ssl encrypted IpFragErr 10500 7 0 0 2 0 0 0 0 0 0
10 23.2.13.227 10.79.0.4 443 45056 0 6 27 0 20 15476 Fri Dec 18 12:23:08 2020 GigabitEthernet0/0/2.100 GigabitEthernet0/0/0 ssl encrypted IpFragErr 9000 6 0 0 2 0 0 0 0 0 0
10 92.123.224.73 10.79.0.4 443 43770 0 6 26 0 13 13692 Fri Dec 18 12:22:54 2020 GigabitEthernet0/0/2.100 GigabitEthernet0/0/0 akamai web IpFragErr 10484 7 0 0 2 0 0 0 0 0 0
10 3.228.30.103 10.79.0.4 443 40502 0 6 26 0 11 13088 Fri Dec 18 12:23:02 2020 GigabitEthernet0/0/2.100 GigabitEthernet0/0/0 ssl encrypted IpFragErr 12000 8 0 0 2 0 0 0 0 0 0
10 54.174.184.180 10.79.0.4 443 51366 0 6 26 0 10 11630 Fri Dec 18 12:23:02 2020 GigabitEthernet0/0/2.100 GigabitEthernet0/0/0 ssl encrypted IpFragErr 10500 7 0 0 2 0 0 0 0 0 0
10 52.40.171.99 10.79.0.4 443 44800 0 6 26 0 13 12765 Fri Dec 18 12:23:06 2020 GigabitEthernet0/0/2.100 GigabitEthernet0/0/0 ssl encrypted IpFragErr 12000 8 0 0 2 0 0 0 0 0 0
10 34.231.93.203 10.79.0.4 443 38760 0 6 26 0 11 13100 Fri Dec 18 12:23:03 2020 GigabitEthernet0/0/2.100 GigabitEthernet0/0/0 web-analytics web IpFragErr 10500 7 0 0 2 0 0 0 0 0 0
10 108.128.13.248 10.79.0.4 443 49590 0 6 26 0 10 10839 Fri Dec 18 12:23:06 2020 GigabitEthernet0/0/2.100 GigabitEthernet0/0/0 adobe-services file-server IpFragErr 10500 7 0 0 2 0 0 0 0 0 0
10 3.228.30.103 10.79.0.4 443 40514 0 6 26 0 11 13088 Fri Dec 18 12:23:09 2020 GigabitEthernet0/0/2.100 GigabitEthernet0/0/0 ssl encrypted IpFragErr 12000 8 0 0 2 0 0 0 0 0 0
10 34.231.93.203 10.79.0.4 443 38764 0 6 26 0 11 13100 Fri Dec 18 12:22:32 2020 GigabitEthernet0/0/2.100 GigabitEthernet0/0/0 web-analytics web IpFragErr 10500 7 0 0 2 0 0 0 0 0 0
10 34.231.93.203 10.79.0.4 443 38766 0 6 26 0 11 13100 Fri Dec 18 12:23:02 2020 GigabitEthernet0/0/2.100 GigabitEthernet0/0/0 web-analytics web IpFragErr 10500 7 0 0 2 0 0 0 0 0 0
10 104.74.80.172 10.79.0.4 443 36320 0 6 26 0 20 14785 Fri Dec 18 12:23:02 2020 GigabitEthernet0/0/2.100 GigabitEthernet0/0/0 web-analytics web IpFragErr 9000 6 0 0 2 0 0 0 0 0 0
10 104.16.149.64 10.79.0.4 443 37522 0 6 26 0 55 45861 Fri Dec 18 12:23:02 2020 GigabitEthernet0/0/2.100 GigabitEthernet0/0/0 ssl encrypted IpFragErr 7500 5 0 0 2 0 0 0 0 0 0
10 34.231.93.203 10.79.0.4 443 38758 0 6 26 0 11 13100 Fri Dec 18 12:23:02 2020 GigabitEthernet0/0/2.100 GigabitEthernet0/0/0 web-analytics web IpFragErr 10500 7 0 0 2 0 0 0 0 0 0
10 23.47.217.196 10.79.0.4 443 47840 0 6 27 0 28 20953 Fri Dec 18 12:23:02 2020 GigabitEthernet0/0/2.100 GigabitEthernet0/0/0 cnn web IpFragErr 12000 8 0 0 2 0 0 0 0 0 0
10 104.20.185.68 10.79.0.4 443 51594 0 6 26 0 23 15114 Fri Dec 18 12:23:02 2020 GigabitEthernet0/0/2.100 GigabitEthernet0/0/0 ssl encrypted IpFragErr 10500 7 0 0 2 0 0 0 0 0 0
10 3.211.216.81 10.79.0.4 443 40340 0 6 26 0 15 16184 Fri Dec 18 12:23:07 2020 GigabitEthernet0/0/2.100 GigabitEthernet0/0/0 pocket web IpFragErr 15000 10 0 0 2 0 0 0 0 0 0
10 52.23.144.54 10.79.0.4 443 48146 0 6 26 0 11 13070 Fri Dec 18 12:23:03 2020 GigabitEthernet0/0/2.100 GigabitEthernet0/0/0 ssl encrypted IpFragErr 12000 8 0 0 2 0 0 0 0 0 0
10 104.16.149.64 10.79.0.4 443 37564 0 6 27 0 16 10974 Fri Dec 18 12:23:02 2020 GigabitEthernet0/0/2.100 GigabitEthernet0/0/0 ssl encrypted IpFragErr 7500 5 0 0 2 0 0 0 0 0 0
10 54.149.50.128 10.79.0.4 443 38954 0 6 26 0 10 11025 Fri Dec 18 12:23:02 2020 GigabitEthernet0/0/2.100 GigabitEthernet0/0/0 ssl encrypted IpFragErr 10500 7 0 0 2 0 0 0 0 0 0
10 99.83.181.31 10.79.0.4 443 54038 0 6 26 0 7 7071 Fri Dec 18 12:23:02 2020 GigabitEthernet0/0/2.100 GigabitEthernet0/0/0 ssl encrypted IpFragErr 6000 4 0 0 2 0 0 0 0 0 0
10 34.231.93.203 10.79.0.4 443 38762 0 6 26 0 11 13100 Fri Dec 18 12:23:02 2020 GigabitEthernet0/0/2.100 GigabitEthernet0/0/0 web-analytics web IpFragErr 10500 7 0 0 2 0 0 0 0 0 0
10 91.215.103.64 10.79.0.4 443 54782 0 6 26 0 12 12536 Fri Dec 18 12:23:02 2020 GigabitEthernet0/0/2.100 GigabitEthernet0/0/0 ssl encrypted IpFragErr 12000 8 0 0 2 0 0 0 0 0 0
10 185.33.220.145 10.79.0.4 443 43938 0 6 26 0 5 4598 Fri Dec 18 12:23:02 2020 GigabitEthernet0/0/2.100 GigabitEthernet0/0/0 ssl encrypted IpFragErr 4486 3 0 0 2 0 0 0 0 0 0
10 52.23.144.54 10.79.0.4 443 48140 0 6 26 0 11 13070 Fri Dec 18 12:23:02 2020 GigabitEthernet0/0/2.100 GigabitEthernet0/0/0 ssl encrypted IpFragErr 12000 8 0 0 2 0 0 0 0 0 0
10 23.47.217.196 10.79.0.4 443 47796 0 6 26 0 172 202028 Fri Dec 18 12:22:51 2020 GigabitEthernet0/0/2.100 GigabitEthernet0/0/0 cnn web IpFragErr 14937 10 0 0 2 0 0 0 0 0 0
10 52.23.144.54 10.79.0.4 443 48148 0 6 26 0 11 13070 Fri Dec 18 12:23:02 2020 GigabitEthernet0/0/2.100 GigabitEthernet0/0/0 ssl encrypted IpFragErr 12000 8 0 0 2 0 0 0 0 0 0
10 54.149.50.128 10.79.0.4 443 38966 0 6 26 0 10 11025 Fri Dec 18 12:23:04 2020 GigabitEthernet0/0/2.100 GigabitEthernet0/0/0 ssl encrypted IpFragErr 10500 7 0 0 2 0 0 0 0 0 0
Does any one have an idea what could be the issue or is experiencing the same problem ?
vManage, vSmart, vBond: 20.3.2
cEdge: 17.03.02.0.3785
Best regards,
Thomas
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-07-2021 04:54 AM - edited 01-07-2021 04:56 AM
Hello,
just to let you know what the resolution of that error was, I had to change the IP MTU and TCP adjust mss values from the service VPN interfaces to another value, Cisco TAC was not sure why the previous values did not work.
Old Settings
interface GigabitEthernet0/0/2 no ip address negotiation auto end interface GigabitEthernet0/0/2.100 description LAN Traffic encapsulation dot1Q 100 vrf forwarding 10 ip address X.X.X.X 255.255.248.0 ip helper-address X.X.X.X no ip redirects ip mtu 1460
ip nbar protocol-discovery ip tcp adjust-mss 1318
vrrp 100 address-family ipv4 timers advertise 100 vrrpv2 address X.X.X.X primary exit-vrrp end
New Settings
interface GigabitEthernet0/0/2 mtu 1505
no ip address no ip redirects load-interval 30 negotiation auto arp timeout 1200 end interface GigabitEthernet0/0/2.100 description LAN Traffic encapsulation dot1Q 100 vrf forwarding 10 ip address X.X.X.X 255.255.248.0 ip helper-address X.X.X.X no ip redirects ip mtu 1500
ip nbar protocol-discovery ip tcp adjust-mss 1460
vrrp 100 address-family ipv4 timers advertise 100 vrrpv2 address X.X.X.X primary exit-vrrp arp timeout 1200 end
Usefull commands during troubleshooting were, watch out for these drop counters with ID 53 IpFragErr.
show sdwan app-fwd dpi flows format table
show platform hardware qfp active statistics drop detail
Regards,
Thomas
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: