Web Traffic Drop | New OSPF/MPLS link

[TrivialPants]

Hi All,

we turned up a new 40g link on an ASR 907 (which has been in place previously on a 903) 

the OSPF and the interface came up with no issue, however once it was moved to use this path, the connections which were past this link on the OSPF links saw intermittent web traffic issues and/or some websites that were totally unreachable. 

we did see a few CRC errors on the link, cleaned it and re-introduced it. However, it seems to still be having issues. 

one thing we saw was that pings were working just fine, but on the affected sites we couldn’t resolve websites such as google.com on https

we currently have the cost for the affected link set higher to avoid using it while investigating further.

does anyone have any insight? 

[Georg Pauwen]

Hello,

difficult to troubleshoot. CRC errors usually indicate some sort of physical (e.g. cabling) problem, but the fact that you cannot access certain sites can also be related to MTU settings. Can you post the running configi of the new ASR907 ?

Actually the first thing I would do is check with your (I assume new) ISP, the problem can also be caused by a setting on their side.

[TrivialPants]

Thanks for the quick response! 

We didn't change the ISP, and in fact, we did attempt to change between our upstream BGP provider and didn't see any change when moving there. 

Our core is OSPF with several 40G routers (with this 907) being part of the ring. It is a new addition/replacement for a 903 router. It has several other 40G links on it, but as soon as we turn up this 40G link we see issues. For context, the 40G link was previously connected on the 903 with the same configuration, but we moved it to the 907. 

I will run through my config and post it here for review, perhaps you may be able to find any issues.

[TrivialPants]

The int fo0/8/1 is the link we are seeing issues with. 

version 17.3
service timestamps debug datetime msec
service timestamps log datetime msec
service unsupported-transceiver
platform fastboot
platform issu reload interface-module sequence 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
no platform punt-keepalive disable-kernel-core
platform bfd-debug-trace 1
!
hostname 907
!
boot-start-marker
boot system flash bootflash:/asr900rsp3-universalk9_npe.17.03.04.SPA.bin
boot-end-marker
!
!
vrf definition Mgmt-intf
 !
 address-family ipv4
 exit-address-family
 !
 address-family ipv6
 exit-address-family
!
!
aaa new-model
!
!
aaa authentication login local_authen local
aaa authorization console
aaa authorization exec local_author local 
!
!
!
!
!
!
aaa session-id common
!
!
!
!
!
login on-success log
!
!
!
!
!
!
!
mpls label protocol ldp
mpls ldp explicit-null
mpls ldp graceful-restart
mpls ldp session protection
mpls ldp igp sync holddown 600000
mpls ldp label
 allocate global host-routes
mpls ldp discovery targeted-hello accept
multilink bundle-name authenticated
!
!
!
no license feature service-offload enable
license udi pid ASR-907 
license boot level metroaggrservices
no license smart enable
memory free low-watermark processor 33728
!
!
sdm prefer disable_8k_efp 
sdm prefer disable_tdm_to_ip_iw 
sdm prefer disable_l2pt_fwd_all 
sdm prefer disable_multicast_stats 
sdm prefer disable_bdi_mtu 
sdm prefer disable_portchannel_qos_multiple_active 
sdm prefer no_efp_feat_ext 
sdm prefer disable_egr_l3vpn_cm 
sdm prefer disable_l3vpn_cm 
sdm prefer disable_copp 
sdm prefer disable_match_inner_dscp 
sdm prefer sr_pfp_disable 
sdm prefer sr_5_label_push_disable 
sdm prefer vpls_stats_disable 
sdm prefer default 
diagnostic bootup level minimal
!
!
redundancy
 mode sso
!
!
controller dwdm 0/3/0
!
controller dwdm 0/3/1
!
controller dwdm 0/4/0
 shutdown
!
controller dwdm 0/4/1
 shutdown
!
controller dwdm 0/7/0
!
controller dwdm 0/8/0
 shutdown
!
controller dwdm 0/8/1
 shutdown
!
controller wanphy 0/12/0
!
controller dwdm 0/12/0
 shutdown
!
controller wanphy 0/12/1
!
controller dwdm 0/12/1
 shutdown
!
controller wanphy 0/12/2
!
controller dwdm 0/12/2
 shutdown
!
controller wanphy 0/12/3
!
controller dwdm 0/12/3
 shutdown
!
controller wanphy 0/12/4
!
controller dwdm 0/12/4
 shutdown
!
controller wanphy 0/12/5
!
controller dwdm 0/12/5
 shutdown
!
controller wanphy 0/12/6
!
controller dwdm 0/12/6
 shutdown
!
controller wanphy 0/12/7
!
controller dwdm 0/12/7
 shutdown
!
!
transceiver type all
 monitoring
cdp run
!
!
l2 router-id 11.10.9.115
! 
!
!
!
!
!
!
!
!
!
!
!
!
!
!
interface Loopback0
 ip address 11.10.9.115 255.255.255.255
!
interface FortyGigabitEthernet0/3/0
 mtu 9200
 ip address 11.0.0.97 255.255.255.252
 no ip redirects
 ip ospf dead-interval minimal hello-multiplier 5
 ip ospf bfd
 ip ospf cost 100
 cdp enable
 mpls ip
 synchronous mode
 bfd interval 50 min_rx 50 multiplier 3
!
interface FortyGigabitEthernet0/3/1 
 mtu 9200
 ip address 11.0.0.1 255.255.255.248
 no ip redirects
 ip ospf dead-interval minimal hello-multiplier 5
 ip ospf bfd
 cdp enable
 mpls ip
 synchronous mode
 bfd interval 50 min_rx 50 multiplier 3
!
interface FortyGigabitEthernet0/4/0
 mtu 9200
 ip address 11.10.9.198 255.255.255.252
 no ip redirects
 ip ospf dead-interval minimal hello-multiplier 5
 ip ospf bfd
 cdp enable
 mpls ip
 synchronous mode
 bfd interval 50 min_rx 50 multiplier 3
!
interface FortyGigabitEthernet0/4/1
 mtu 9200
 ip address 11.0.0.17 255.255.255.252
 no ip redirects
 ip ospf dead-interval minimal hello-multiplier 5
 ip ospf bfd
 cdp enable
 mpls ip
 bfd interval 50 min_rx 50 multiplier 3
!
interface HundredGigE0/7/0
 mtu 9200
 ip address 11.10.9.242 255.255.255.252
 no ip redirects
 cdp enable
 mpls ip
 bfd interval 50 min_rx 50 multiplier 3
!
interface FortyGigabitEthernet0/8/0
 ip address 11.10.9.17 255.255.255.252
 no ip redirects
 ip ospf dead-interval minimal hello-multiplier 5
 ip ospf bfd
 cdp enable
 mpls ip
 synchronous mode
 bfd interval 50 min_rx 50 multiplier 3
!
interface FortyGigabitEthernet0/8/1
 mtu 9200
 ip address 11.0.0.97 255.255.255.252
 no ip redirects
 ip ospf dead-interval minimal hello-multiplier 5
 ip ospf bfd
 ip ospf cost 100
 cdp enable
 mpls ip
 synchronous mode
 bfd interval 50 min_rx 50 multiplier 3

!
interface TenGigabitEthernet0/12/0
 no ip address
!
interface TenGigabitEthernet0/12/1
 no ip address
!
interface TenGigabitEthernet0/12/2
 no ip address
!
interface TenGigabitEthernet0/12/3
 no ip address
!
interface TenGigabitEthernet0/12/4
 no ip address
!
interface TenGigabitEthernet0/12/5
 no ip address
!
interface TenGigabitEthernet0/12/6
 no ip address
!
interface TenGigabitEthernet0/12/7
 no ip address
!
interface GigabitEthernet0
 vrf forwarding Mgmt-intf
 no ip address
 shutdown
 negotiation auto
!
router ospf 100
 router-id 11.10.9.115
 auto-cost reference-bandwidth 100000
 fast-reroute per-prefix enable area 0 prefix-priority low
 fast-reroute per-prefix remote-lfa tunnel mpls-ldp
 timers throttle spf 50 50 5000
 timers throttle lsa 10 20 5000
 timers lsa arrival 10
 timers pacing flood 5
 redistribute connected
 network 11.0.0.0 0.0.1.255 area 0
 network 11.10.9.0 0.0.0.255 area 0
 maximum-paths 8
 bfd all-interfaces
 mpls ldp sync
!
no ip http server
no ip http ctc authentication
no ip http secure-server
ip forward-protocol nd
ip ssh server algorithm encryption aes128-ctr aes192-ctr aes256-ctr
ip ssh client algorithm encryption aes128-ctr aes192-ctr aes256-ctr
!
ip tftp source-interface Loopback0
ip tftp blocksize 1400
!
logging alarm informational
logging source-interface Loopback0
!
mpls ldp router-id Loopback0 force
!
!
!
!
control-plane
!
environment router altitude 0
!
line con 0
 authorization exec local_author
 login authentication local_authen
 stopbits 1
line vty 0 4
 authorization exec local_author
 login authentication local_authen
 transport input telnet ssh
line vty 5 15
 authorization exec local_author
 login authentication local_authen
 transport input telnet ssh 

[Georg Pauwen]

Hello,

glad to hear that you fixed it. I was looking at the auto cost reference of 100000, with that value, the cost of 100 you configured on the interface would equal a 1 Gigabit link. My suggestion was to change the cost on the interface to 1...but obviously that suggestion is now obsolete.

[Joseph W. Doherty]

Cannot say this is the cause of your issue, but as you mention "new" and "MPLS", I have seen the rare case where network service provider doesn't configure some interface to support MPLS MTU increase.  This will cause issue when packet sizes are within the 4 bytes of MTU.

[TrivialPants · Accepted solution]

We fixed the issue. It was one of two things: 

We upgraded the remote router to a new version 17.3.4 from 16.12.4. 

However, before doing that, we defaulted the port and noticed that we weren’t seeing the same issue after reapplying the config. 

our hope is that the upgrade/reboot cleared out some issue process that was hanging it up.