ASR1002 OSPF trouble - Page 2

TarasKotov · ‎08-23-2011

I have a problem with OSPF on ASR1002 router.

It has 2 neighbors - D-Link L3 switches (DGS-3627G). I attached the scheme to this post.

ASR1002 is used for: 2 BGP FW, NAT, ISG, Netflow.

IOS version is: asr1000rp1-adventerprise.03.03.00.S.151-2.S.bin

Neighbors are:

ASR1002#sh ip ospf neighbor

Neighbor ID Pri State Dead Time Address Interface

1.1.1.87 1 FULL/BDR 00:00:31 10.255.10.1 GigabitEthernet0/1/2

1.1.1.99 1 FULL/BDR 00:00:06 10.255.10.5 GigabitEthernet0/1/0

The problem is that neighbors are periodically going down:

6756: Aug 23 10:50:59.005 OMSST: %OSPF-5-ADJCHG: Process 1, Nbr 1.1.1.99 on GigabitEthernet0/1/0 from FULL to DOWN, Neighbor Down: Dead timer expired

016757: Aug 23 10:51:29.018 OMSST: %OSPF-5-ADJCHG: Process 1, Nbr 1.1.1.99 on GigabitEthernet0/1/0 from LOADING to FULL, Loading Done

016758: Aug 23 11:24:59.145 OMSST: %OSPF-5-ADJCHG: Process 1, Nbr 1.1.1.99 on GigabitEthernet0/1/0 from FULL to DOWN, Neighbor Down: Dead timer expired

016759: Aug 23 11:25:29.156 OMSST: %OSPF-5-ADJCHG: Process 1, Nbr 1.1.1.99 on GigabitEthernet0/1/0 from LOADING to FULL, Loading Done

016761: Aug 23 13:04:29.555 OMSST: %OSPF-5-ADJCHG: Process 1, Nbr 1.1.1.99 on GigabitEthernet0/1/0 from FULL to DOWN, Neighbor Down: Dead timer expired

016762: Aug 23 13:04:59.566 OMSST: %OSPF-5-ADJCHG: Process 1, Nbr 1.1.1.99 on GigabitEthernet0/1/0 from LOADING to FULL, Loading Done

016763: Aug 23 13:34:49.683 OMSST: %OSPF-5-ADJCHG: Process 1, Nbr 1.1.1.99 on GigabitEthernet0/1/0 from FULL to DOWN, Neighbor Down: Dead timer expired

016764: Aug 23 13:35:19.695 OMSST: %OSPF-5-ADJCHG: Process 1, Nbr 1.1.1.99 on GigabitEthernet0/1/0 from LOADING to FULL, Loading Done

And the biggest problem is that they sometimes go down simultaneously.

ASR says that it doesn't see hello packets and drops the adjacency:

018299: Aug 23 17:58:39.073 OMSST: OSPF-1 HELLO Gi0/1/0: Rcv hello from 1.1.1.99 area 0.0.0.2 10.255.10.5

018300: Aug 23 17:58:39.450 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

ASR1002#

018301: Aug 23 17:58:40.430 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

018302: Aug 23 17:58:41.367 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

ASR1002#

018303: Aug 23 17:58:42.286 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

018304: Aug 23 17:58:42.327 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

018305: Aug 23 17:58:43.189 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

ASR1002#

018306: Aug 23 17:58:44.119 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

018307: Aug 23 17:58:45.096 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

ASR1002#

018308: Aug 23 17:58:45.794 OMSST: OSPF-1 HLOIN: input packet - Hello GigabitEthernet0/1/2 (0)

018309: Aug 23 17:58:46.093 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

ASR1002#

018310: Aug 23 17:58:47.046 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

018311: Aug 23 17:58:48.010 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

018312: Aug 23 17:58:48.026 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

018313: Aug 23 17:58:48.026 OMSST: OSPF-1 HELLO Gi0/1/0: Send hello to 224.0.0.5 area 0.0.0.2 from 10.255.10.6

ASR1002#

018314: Aug 23 17:58:48.975 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

018315: Aug 23 17:58:49.908 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

ASR1002#

018316: Aug 23 17:58:50.844 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

018317: Aug 23 17:58:51.785 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

ASR1002#

018318: Aug 23 17:58:52.286 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

018319: Aug 23 17:58:52.747 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

ASR1002#

018320: Aug 23 17:58:53.739 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

018321: Aug 23 17:58:54.718 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

ASR1002#

018322: Aug 23 17:58:55.621 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

018323: Aug 23 17:58:55.795 OMSST: OSPF-1 HLOIN: input packet - Hello GigabitEthernet0/1/2 (0)

018324: Aug 23 17:58:56.601 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

ASR1002#

018325: Aug 23 17:58:57.028 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

018326: Aug 23 17:58:57.028 OMSST: OSPF-1 HELLO Gi0/1/0: Send hello to 224.0.0.5 area 0.0.0.2 from 10.255.10.6

018327: Aug 23 17:58:57.506 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

ASR1002#

018328: Aug 23 17:58:58.489 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

018329: Aug 23 17:58:59.418 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

ASR1002#

018330: Aug 23 17:59:00.355 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

018331: Aug 23 17:59:01.288 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

ASR1002#

018332: Aug 23 17:59:01.675 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

018333: Aug 23 17:59:02.216 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

ASR1002#

018334: Aug 23 17:59:03.123 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

018335: Aug 23 17:59:04.083 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

ASR1002#

018336: Aug 23 17:59:05.064 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

018337: Aug 23 17:59:05.795 OMSST: OSPF-1 HLOIN: input packet - Hello GigabitEthernet0/1/2 (0)

018338: Aug 23 17:59:06.059 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

ASR1002#

018339: Aug 23 17:59:06.263 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

018340: Aug 23 17:59:06.263 OMSST: OSPF-1 HELLO Gi0/1/0: Send hello to 224.0.0.5 area 0.0.0.2 from 10.255.10.6

018341: Aug 23 17:59:06.998 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

ASR1002#

018342: Aug 23 17:59:07.983 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

018343: Aug 23 17:59:08.975 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

ASR1002#

018344: Aug 23 17:59:09.936 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

018345: Aug 23 17:59:10.930 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

ASR1002#

018346: Aug 23 17:59:11.399 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

018347: Aug 23 17:59:11.889 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

ASR1002#

018348: Aug 23 17:59:12.876 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

018349: Aug 23 17:59:13.869 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

ASR1002#

018350: Aug 23 17:59:14.792 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

018351: Aug 23 17:59:15.733 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

ASR1002#

018352: Aug 23 17:59:15.796 OMSST: OSPF-1 HLOIN: input packet - Hello GigabitEthernet0/1/2 (0)

018353: Aug 23 17:59:16.174 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

018354: Aug 23 17:59:16.174 OMSST: OSPF-1 HELLO Gi0/1/0: Send hello to 224.0.0.5 area 0.0.0.2 from 10.255.10.6

018355: Aug 23 17:59:16.714 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

ASR1002#

018356: Aug 23 17:59:17.648 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

018357: Aug 23 17:59:18.616 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

018358: Aug 23 17:59:19.073 OMSST: OSPF-1 HLOTM: timer expiration - NBR_DEAD_TIMER (0)

018359: Aug 23 17:59:19.073 OMSST: OSPF-1 ADJ Gi0/1/0: 1.1.1.99 address 10.255.10.5 is dead

018360: Aug 23 17:59:19.073 OMSST: OSPF-1 ADJ Gi0/1/0: 1.1.1.99 address 10.255.10.5 is dead, state DOWN

018361: Aug 23 17:59:19.073 OMSST: %OSPF-5-ADJCHG: Process 1, Nbr 1.1.1.99 on GigabitEthernet0/1/0 from FULL to DOWN, Neighbor Down: Dead timer expired

ASR1002#

018362: Aug 23 17:59:19.073 OMSST: OSPF-1 ADJ Gi0/1/0: Neighbor change event

018363: Aug 23 17:59:19.073 OMSST: OSPF-1 ADJ Gi0/1/0: DR/BDR election

018364: Aug 23 17:59:19.073 OMSST: OSPF-1 ADJ Gi0/1/0: Elect BDR 0.0.0.0

018365: Aug 23 17:59:19.073 OMSST: OSPF-1 ADJ Gi0/1/0: Elect DR 10.255.0.1

018366: Aug 23 17:59:19.073 OMSST: OSPF-1 ADJ Gi0/1/0: DR: 10.255.0.1 (Id) BDR: none

018367: Aug 23 17:59:19.564 OMSST: OSPF-1 HLOTM: timer expiration - IF_HELLO_TIMER (0)

I placed another switch between ASR1002 and one of its neighbors (neighbor 1.1.1.99) to mirror traffic (Cisco calls it SPAN) and see hello packets to and from D-Link switch. At some moment (there is no exact pattern like "once an hour" or "once 30 minutes") ASR begins to send hello packets without Neighbor field in it. After 4 hello packets it drops an adjacency and begins DR/BDR election and reestablishes adjacency again.

Here is config section about OSPF on ASR:

ASR1002#sh run | s router ospf

router ospf 1

router-id 10.255.0.1

auto-cost reference-bandwidth 10000

area 4 nssa

passive-interface default

no passive-interface GigabitEthernet0/1/0

no passive-interface GigabitEthernet0/1/2

network 10.255.10.2 0.0.0.0 area 0.0.0.1

network 10.255.10.6 0.0.0.0 area 0.0.0.2

neighbor 10.255.10.5

default-information originate always metric-type 1

ASR1002#sh run int gi0/1/0

Building configuration...

Current configuration : 456 bytes

!

interface GigabitEthernet0/1/0

description === ASR1002[Gi0/1/0] --- agsw01-99[23] [Access Net] ===

dampening

ip address 10.255.10.6 255.255.255.252

ip access-group nat-pool in

no ip redirects

no ip unreachables

no ip proxy-arp

ip nat inside

ip flow egress

ip virtual-reassembly

load-interval 30

carrier-delay msec 0

negotiation auto

service-policy type control ISG-IP-POLICY

ip subscriber routed

initiator unclassified ip-address

end

ASR1002#sh run int gi0/1/2

Building configuration...

Current configuration : 456 bytes

!

interface GigabitEthernet0/1/2

description === ASR1002[Gi0/1/2] --- agsw01-87[23] [Access Net] ===

dampening

ip address 10.255.10.2 255.255.255.252

ip access-group nat-pool in

no ip redirects

no ip unreachables

no ip proxy-arp

ip nat inside

ip flow egress

ip virtual-reassembly

load-interval 30

carrier-delay msec 0

negotiation auto

service-policy type control ISG-IP-POLICY

ip subscriber routed

initiator unclassified ip-address

I suspect that the problem can be somehow connected with ESP resources like DRAM, TCAM, etc and ASR drops hello packets before getting them to RP. But can't find any diagnostic command to confirm it.

TarasKotov · ‎08-25-2011

1. Dear douhanm,

If you read RFC2328 or any book about OSPF you'll notice that router will not form an adjacency with its neighbor if it has different timers. In my case there are standard timers on all devices - 10/40.

2. There is no multicast filtering on any devise.

3. D-Link is capable of IGMP snooping. But there is no non-default configuration on ASR or D-Link. Also I turned off IGMP snooping for vlan 699 on 1.1.1.99 and for vlan 687 on 1.1.1.87 (switches have SVI's in those vlans, and this SVI's are for L3 connection to ASR):

config igmp_snooping vlan 699 state disable fast_leave disable

4. Traffic dump is not ready yet. As soon as I recieve it, I'll place it here.

Sincerely,

Taras Kotov