EIGRP/PIM Routing Issues

michaelcaldwell · ‎09-06-2011

Hi,

We have been having an issue with our routers for the past couple of weeks where there are a lot of EIGRP/PIM error messages in the logs.

This flapping is occuring on a handful of our core switches. They are in the following format:

Sep 6 08:15:18.785 CST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 5: Neighbor 192.168.255.4 (GigabitEthernet8/48) is up: new adjacency

Sep 6 08:18:36.569 CST: %PIM-5-NBRCHG: neighbor 192.168.255.82 DOWN on interface GigabitEthernet8/48 non DR

Sep 6 08:18:50.612 CST: %PIM-5-NBRCHG: neighbor 192.168.255.82 UP on interface GigabitEthernet8/48

Sep 6 08:41:46.275 CST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 5: Neighbor 192.168.255.84 (GigabitEthernet8/48) is down: holding time expired

Sep 6 08:41:46.763 CST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 5: Neighbor 192.168.255.84 (GigabitEthernet8/48) is up: new adjacency

Sep 6 09:02:49.908 CST: %PIM-5-NBRCHG: neighbor 192.168.255.56 DOWN on interface GigabitEthernet8/48 non DR

Sep 6 09:03:02.904 CST: %PIM-5-NBRCHG: neighbor 192.168.255.56 UP on interface GigabitEthernet8/48

Sep 6 09:05:16.629 CST: %PIM-5-NBRCHG: neighbor 192.168.255.48 DOWN on interface GigabitEthernet8/48 non DR

Sep 6 09:05:30.704 CST: %PIM-5-NBRCHG: neighbor 192.168.255.48 UP on interface GigabitEthernet8/48

Sep 6 09:38:53.536 CST: %PIM-5-NBRCHG: neighbor 192.168.255.4 DOWN on interface GigabitEthernet8/48 non DR

Sep 6 09:39:07.351 CST: %PIM-5-NBRCHG: neighbor 192.168.255.4 UP on interface GigabitEthernet8/48

Sep 6 09:39:56.190 CST: %PIM-5-NBRCHG: neighbor 192.168.255.100 DOWN on interface GigabitEthernet8/48 non DR

Sep 6 09:40:09.106 CST: %PIM-5-NBRCHG: neighbor 192.168.255.100 UP on interface GigabitEthernet8/48

All the WAN facing interfaces are configured the exact same:

ip hold-time eigrp 5 120

ip pim dense-mode

speed 1000

duplex full

We have about 25 sites and it seems to be happening between only a handful. We had changed the hold time for EIGRP to try and fix the %DUAL-5-NBRCHANGE errors as we saw that it was exceeding the default hold-time. Now it seems that we are seeing the PIM errors more today than the NBRCHANGE errors.

Any help would be much appreciated!

Thanks!

Amit Aneja · ‎09-06-2011

Hi,

Could you please share the following o/p from this router?

show run

sho int gi8/48

show proc cpu hist

show ip eigrp traffic

Since PIM is also flapping, I would assume that there is multicast packets being dropped. To isolate it & rule out if multicast is the what's being dropped, can you try "neighbor" command in EIGRP with one of its neighbor & see if it works well with unicast?

I would also look for drops on gi8/48...

Regards,

Amit

michaelcaldwell · ‎09-06-2011

Amit,

Thank you for replying. Here is the information you requested!

sh run int gi8/48

interface GigabitEthernet8/48

description Fiber Connection

no switchport

ip address 192.168.255.88 255.255.255.0

ip hold-time eigrp 5 120

ip pim dense-mode

speed 1000

duplex full

qos trust dscp

auto qos voip trust

tx-queue 3

bandwidth percent 33

priority high

shape percent 33

service-policy output autoqos-voip-policy

end

michaelcaldwell · ‎09-06-2011

sh proc cpu history

4455555444444444444444444444444555554444444444444444444444

6611111555555555566666666688888222226666677777777774444466

100

90

80

70

60

50 *************************************************** **

40 **********************************************************

30 **********************************************************

20 **********************************************************

10 **********************************************************

0....5....1....1....2....2....3....3....4....4....5....5....

0 5 0 5 0 5 0 5 0 5

CPU% per second (last 60 seconds)

5555555555555555555555555555655566555555555555555555555555

2219121291111921109232492233123100121832449654373322732228

100

90

80

70

60 * * * * * * ** * *** * * *

50 ##########################################################

40 ##########################################################

30 ##########################################################

20 ##########################################################

10 ##########################################################

0....5....1....1....2....2....3....3....4....4....5....5....

0 5 0 5 0 5 0 5 0 5

CPU% per minute (last 60 minutes)

* = maximum CPU% # = average CPU%

6866666976966666666669866666666669655655556676666555666666666666666656

1281424916843000800018133102324138188099991186010889202101301000000081

100 * * * *

90 * * * *

80 * * * ** * *

70 ** **** * ** * **

60 **********************************************************************

50 ######################################################################

40 ######################################################################

30 ######################################################################

20 ######################################################################

10 ######################################################################

0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.

0 5 0 5 0 5 0 5 0 5 0 5 0

CPU% per hour (last 72 hours)

* = maximum CPU% # = average CPU%

(This is a Catalyst 4510R - which seem to run higher than other Catalysts)

michaelcaldwell · ‎09-06-2011

sh ip eigrp traffic

EIGRP-IPv4 Traffic Statistics for AS(5)

Hellos sent/received: 2283784/4731742

Updates sent/received: 2186/3959

Queries sent/received: 826/2020

Replies sent/received: 1823/4543

Acks sent/received: 8797/12634

SIA-Queries sent/received: 0/0

SIA-Replies sent/received: 0/0

Hello Process ID: 256

PDM Process ID: 255

Socket Queue: 0/2000/49/0 (current/max/highest/drops)

Input Queue: 0/2000/49/0 (current/max/highest/drops)

michaelcaldwell · ‎09-06-2011

sh run

Building configuration...

Current configuration : 84101 bytes

!

! Last configuration change at 08:10:00 CST Tue Sep 6 2011 by admin

! NVRAM config last updated at 08:10:01 CST Tue Sep 6 2011 by admin

!

version 15.0

no service slave-log

no service pad

service tcp-keepalives-in

service tcp-keepalives-out

service timestamps debug datetime msec localtime show-timezone

service timestamps log datetime msec localtime show-timezone

service password-encryption

service compress-config

!

hostname kjhs-mdf-cs01

!

boot-start-marker

boot system flash bootflash:/cat4500-entservicesk9-mz.150-2.SG.bin

boot-end-marker

!

logging buffered 131072

enable secret 5 (omitted)

!

username admin privilege 15 secret 5 (omitted)

!

no aaa new-model

clock timezone CST -6

clock summer-time CST recurring

qos

qos dbl

qos map dscp 24 25 26 27 28 29 30 31 to tx-queue 4

qos map dscp 32 33 34 35 36 37 38 39 to tx-queue 4

qos map cos 5 to dscp 46

ip subnet-zero

ip domain-name unit5.org

ip name-server 10.49.1.10

ip name-server 10.49.1.12

ip name-server 10.49.94.2

ip vrf mgmtVrf

!

ip multicast-routing

ip dhcp excluded-address 10.49.95.1

ip dhcp excluded-address 10.49.95.10

!

ip dhcp pool vlan_500

network 10.49.95.0 255.255.255.0

default-router 10.49.95.1

domain-name unit5.org

dns-server 10.49.1.10 10.49.1.12

option 150 ip 10.49.111.10 10.49.3.10

lease 0 8

!

crypto pki trustpoint TP-self-signed-90803

enrollment selfsigned

subject-name cn=IOS-Self-Signed-Certificate-90803

revocation-check none

rsakeypair TP-self-signed-90803

!

crypto pki certificate chain TP-self-signed-90803

certificate self-signed 01

(omitted)

quit

power redundancy-mode combined

!

spanning-tree mode rapid-pvst

spanning-tree extend system-id

spanning-tree vlan 1-4094 priority 24576

!

redundancy

mode sso

main-cpu

auto-sync standard

!

vlan internal allocation policy ascending

!

policy-map autoqos-voip-policy

class class-default

dbl

!

interface FastEthernet1

ip vrf forwarding mgmtVrf

no ip address

speed auto

duplex auto

!

(interfaces omitted)

!

interface Vlan10

description SVI: VLAN 10 - Building LAN

no ip address

!

interface Vlan90

ip address 192.168.88.1 255.255.255.0

!

interface Vlan101

description SVI: VLAN 101 - Workstation VLAN

ip address 10.49.88.1 255.255.255.0

ip helper-address 10.49.94.2

ip pim dense-mode

!

interface Vlan102

description SVI: VLAN 102 - Workstation VLAN

ip address 10.49.89.1 255.255.255.0

ip helper-address 10.49.94.2

ip pim dense-mode

!

interface Vlan103

description SVI: VLAN 103 - Workstation VLAN

ip address 10.49.90.1 255.255.255.0

ip helper-address 10.49.94.2

ip pim dense-mode

!

interface Vlan104

description SVI: VLAN 104 - Workstation VLAN

ip address 10.88.1.1 255.255.0.0 secondary

ip address 10.49.91.1 255.255.255.0

ip helper-address 10.49.94.2

no ip redirects

ip pim dense-mode

!

interface Vlan105

description SVI: VLAN 105 - NOT IN USE - Reserved

no ip address

ip pim dense-mode

shutdown

!

interface Vlan106

description SVI: VLAN 106 - NOT IN USE - Reserved

no ip address

ip pim dense-mode

shutdown

!

interface Vlan107

description SVI: VLAN 107 - NOT IN USE - Reserved

no ip address

ip pim dense-mode

shutdown

!

interface Vlan108

description SVI: VLAN 108 - NOT IN USE - Reserved

no ip address

ip pim dense-mode

shutdown

!

interface Vlan200

description SVI: VLAN 200 - Network Servers & Printers

ip address 10.49.94.1 255.255.255.192

ip pim dense-mode

!

interface Vlan300

description SVI: VLAN 300 - Facilities Network

ip address 10.49.94.65 255.255.255.192

ip pim dense-mode

!

interface Vlan400

description SVI: VLAN 400 - Security Network

ip address 10.49.94.129 255.255.255.128

ip pim dense-mode

!

interface Vlan500

description SVI: VLAN 500 - Voice Network

ip address 10.49.95.1 255.255.255.0

ip pim dense-mode

!

interface Vlan900

description SVI: VLAN 900 - Network Management

ip address 10.49.246.1 255.255.255.0

ip pim dense-mode

!

interface Vlan1000

description SVI: VLAN 1000 - Wireless WAN Management Network

ip address 10.49.226.97 255.255.255.248

ip pim dense-mode

!

router eigrp 5

network 10.49.0.0 0.0.255.255

network 192.168.255.0

eigrp router-id 192.168.255.88

!

ip default-gateway 10.49.246.1

no ip http server

ip http authentication local

ip http secure-server

!

logging trap debugging

logging 10.49.1.121

!

snmp-server engineID local 800000090300000C30ABAD64

snmp-server community (omitted)

snmp-server enable traps tty

!

control-plane

!

line con 0

stopbits 1

line vty 0 4

login local

length 0

transport input telnet ssh

!

ntp clock-period 17179453

ntp server 10.49.1.1

end

michaelcaldwell · ‎09-06-2011

sh ip eigrp neighbors detail

EIGRP-IPv4 Neighbors for AS(5)

H Address Interface Hold Uptime SRTT RTO Q Seq

(sec) (ms) Cnt Num

5 192.168.255.84 Gi8/48 12 07:40:59 12 200 0 62222

Version 12.2/1.2, Retrans: 2, Retries: 0, Prefixes: 141

Topology-ids from peer - 0

0 192.168.255.4 Gi8/48 115 08:07:27 12 200 0 4617

Version 5.1/3.0, Retrans: 6, Retries: 0, Prefixes: 33

Topology-ids from peer - 0

3 192.168.255.8 Gi8/48 116 08:08:48 10 200 0 4817

Version 5.1/3.0, Retrans: 5, Retries: 0, Prefixes: 33

Topology-ids from peer - 0

18 192.168.255.60 Gi8/48 115 08:18:19 29 200 0 5039

Version 5.1/3.0, Retrans: 8, Retries: 0, Prefixes: 7

Topology-ids from peer - 0

6 192.168.255.56 Gi8/48 117 08:18:57 19 200 0 4429

Version 5.1/3.0, Retrans: 9, Retries: 0, Prefixes: 7

Topology-ids from peer - 0

13 192.168.255.52 Gi8/48 117 08:19:43 25 200 0 4387

Version 5.1/3.0, Retrans: 9, Retries: 0, Prefixes: 7

Topology-ids from peer - 0

19 192.168.255.48 Gi8/48 118 08:20:42 12 200 0 4354

Version 5.1/3.0, Retrans: 7, Retries: 0, Prefixes: 7

Topology-ids from peer - 0

1 192.168.255.36 Gi8/48 118 08:25:38 12 200 0 4168

Version 5.1/3.0, Retrans: 10, Retries: 0, Prefixes: 7

Topology-ids from peer - 0

22 192.168.255.20 Gi8/48 117 08:28:00 13 200 0 4593

Version 5.1/3.0, Retrans: 9, Retries: 0, Prefixes: 7

Topology-ids from peer - 0

11 192.168.255.100 Gi8/48 117 4d00h 39 234 0 5715

Version 5.2/3.0, Retrans: 16, Retries: 0, Prefixes: 9

Topology-ids from peer - 0

20 192.168.255.12 Gi8/48 116 4d02h 37 222 0 5413

Version 5.2/3.0, Retrans: 18, Retries: 0, Prefixes: 9

Topology-ids from peer - 0

23 192.168.255.16 Gi8/48 115 4d05h 39 234 0 5237

Version 5.2/3.0, Retrans: 18, Retries: 0, Prefixes: 9

Topology-ids from peer - 0

9 192.168.255.1 Gi8/48 115 4d05h 10 200 0 8222

Version 5.1/3.0, Retrans: 18, Retries: 0, Prefixes: 31

Topology-ids from peer - 0

21 192.168.255.64 Gi8/48 116 4d05h 55 330 0 4124

Version 5.2/3.0, Retrans: 16, Retries: 0, Prefixes: 7

Topology-ids from peer - 0

4 192.168.255.72 Gi8/48 119 4d05h 19 200 0 4337

Version 5.2/3.0, Retrans: 16, Retries: 0, Prefixes: 7

Topology-ids from peer - 0

14 192.168.255.82 Gi8/48 116 4d06h 37 222 0 3471

Version 5.2/3.0, Retrans: 17, Retries: 0, Prefixes: 7

Topology-ids from peer - 0

12 192.168.255.40 Gi8/48 119 4d07h 94 564 0 4757

Version 5.2/3.0, Retrans: 19, Retries: 0, Prefixes: 7

Topology-ids from peer - 0

15 192.168.255.32 Gi8/48 115 4d08h 15 200 0 3999

Version 5.2/3.0, Retrans: 21, Retries: 0, Prefixes: 7

Topology-ids from peer - 0

8 192.168.255.80 Gi8/48 118 5d03h 141 846 0 4119

Version 5.2/3.0, Retrans: 26, Retries: 0, Prefixes: 7

Topology-ids from peer - 0

17 192.168.255.76 Gi8/48 116 5d04h 33 200 0 4209

Version 5.2/3.0, Retrans: 25, Retries: 0, Prefixes: 7

Topology-ids from peer - 0

16 192.168.255.96 Gi8/48 119 5d06h 21 200 0 4508

Version 5.2/3.0, Retrans: 29, Retries: 0, Prefixes: 7

Topology-ids from peer - 0

7 192.168.255.28 Gi8/48 118 6d02h 16 200 0 3158

Version 5.2/3.0, Retrans: 26, Retries: 0, Prefixes: 7

Topology-ids from peer - 0

2 192.168.255.92 Gi8/48 116 6d02h 30 200 0 4472

Version 5.2/3.0, Retrans: 33, Retries: 0, Prefixes: 19

Topology-ids from peer - 0

Amit Aneja · ‎09-06-2011

Please share "show int gi8/48"

michaelcaldwell · ‎09-06-2011

Amit - I tried to share that as well, but it kept saying that "this message can not be displayed due to its content."

This is all that it would take...

L3 in Switched: ucast: 293639320 pkt, 242648536673 bytes - mcast: 82577516 pkt, 17440865645 bytes

L3 out Switched: ucast: 209704398 pkt, 50728868903 bytes - mcast: 93614 pkt, 17561681 bytes

393886374 packets input, 268468312912 bytes, 0 no buffer

Received 100245154 broadcasts (87559213 IP multicasts)

438 input errors, 78 CRC, 0 frame, 11518 overrun, 0 ignored

210857959 packets output, 55039504285 bytes, 0 underruns

Amit Aneja · ‎09-06-2011

438 input errors, 78 CRC, 0 frame, 11518 overrun, 0 ignored

Let's watch out for this counter, I would like to make sure that this counter isn't incrementing.

Also, do you see any drops in the input queue on this interface?

I see high CPU util as well, that could very well be a cause on why adjacencies would flap or it may be happening due to the flap itself. So, CPU could be the culprit or victim.

To find out what it is, I would like to see what processes in the o/p of "show proc cpu sorted" are eating up maximum resources, if its "ip input" then CPU is the victim.

Regards,

Amit

michaelcaldwell · ‎09-06-2011

Input queue: 1/75/0/0 (size/max/drops/flushes); Total output drops: 358

sh proc cpu sorted shows these two as the top two:

58 141164142 63869252 2210 17.82% 19.92% 19.98% 0 Cat4k Mgmt LoPri

57 127462404 187599343 679 14.39% 15.35% 15.41% 0 Cat4k Mgmt HiPri

and IP input is 5 or 6 down in the list currently

117 6711019 58522887 114 0.63% 0.69% 0.67% 0 IP Input

Amit Aneja · ‎09-06-2011

The above two processes normally show this much utilization, so this looks normal.

I will also see if the output drops increase when flapping happens again.

michaelcaldwell · ‎09-06-2011

Okay. I will keep an eye on it tomorrow and see what happens when it is doing it again. It normally only happens during regular work hours.

michaelcaldwell · ‎09-07-2011

This morning, it started back up, and from the int g8/48 there is this:

438 input errors, 78 CRC, 0 frame, 11544 overrun, 0 ignored

Which looks to be more overrun errors, but the same # of input errors.

The processes show IP input running at .6% average.

Input queue: 2/75/0/0 (size/max/drops/flushes); Total output drops: 358