cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
3917
Views
0
Helpful
13
Replies
Highlighted

EIGRP/PIM Routing Issues

Hi,

  We have been having an issue with our routers for the past couple of weeks where there are a lot of EIGRP/PIM error messages in the logs. 

This flapping is occuring on a handful of our core switches.  They are in the following format: 

Sep  6 08:15:18.785 CST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 5: Neighbor 192.168.255.4 (GigabitEthernet8/48) is up: new adjacency

Sep  6 08:18:36.569 CST: %PIM-5-NBRCHG: neighbor 192.168.255.82 DOWN on interface GigabitEthernet8/48 non DR

Sep  6 08:18:50.612 CST: %PIM-5-NBRCHG: neighbor 192.168.255.82 UP on interface GigabitEthernet8/48

Sep  6 08:41:46.275 CST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 5: Neighbor 192.168.255.84 (GigabitEthernet8/48) is down: holding time expired

Sep  6 08:41:46.763 CST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 5: Neighbor 192.168.255.84 (GigabitEthernet8/48) is up: new adjacency

Sep  6 09:02:49.908 CST: %PIM-5-NBRCHG: neighbor 192.168.255.56 DOWN on interface GigabitEthernet8/48 non DR

Sep  6 09:03:02.904 CST: %PIM-5-NBRCHG: neighbor 192.168.255.56 UP on interface GigabitEthernet8/48

Sep  6 09:05:16.629 CST: %PIM-5-NBRCHG: neighbor 192.168.255.48 DOWN on interface GigabitEthernet8/48 non DR

Sep  6 09:05:30.704 CST: %PIM-5-NBRCHG: neighbor 192.168.255.48 UP on interface GigabitEthernet8/48

Sep  6 09:38:53.536 CST: %PIM-5-NBRCHG: neighbor 192.168.255.4 DOWN on interface GigabitEthernet8/48 non DR

Sep  6 09:39:07.351 CST: %PIM-5-NBRCHG: neighbor 192.168.255.4 UP on interface GigabitEthernet8/48

Sep  6 09:39:56.190 CST: %PIM-5-NBRCHG: neighbor 192.168.255.100 DOWN on interface GigabitEthernet8/48 non DR

Sep  6 09:40:09.106 CST: %PIM-5-NBRCHG: neighbor 192.168.255.100 UP on interface GigabitEthernet8/48

All the WAN facing interfaces are configured the exact same:

ip hold-time eigrp 5 120

ip pim dense-mode

speed 1000

duplex full

We have about 25 sites and it seems to be happening between only a handful.  We had changed the hold time for EIGRP to try and fix the %DUAL-5-NBRCHANGE errors as we saw that it was exceeding the default hold-time.  Now it seems that we are seeing the PIM errors more today than the NBRCHANGE errors.

Any help would be much appreciated!

Thanks!

Everyone's tags (2)
13 REPLIES 13
Highlighted
Participant

EIGRP/PIM Routing Issues

Hi,

Could you please share the following o/p from this router?

show run

sho int gi8/48

show  proc cpu hist

show ip eigrp traffic

Since PIM is also flapping, I would assume that there is multicast packets being dropped. To isolate it & rule out if multicast is the what's being dropped, can you try "neighbor" command in EIGRP with one of its neighbor & see if it works well with unicast?

I would also look for drops on gi8/48...

Regards,

Amit

Highlighted

EIGRP/PIM Routing Issues

Amit,

  Thank you for replying.  Here is the information you requested!

sh run int gi8/48

interface GigabitEthernet8/48

description Fiber Connection

no switchport

ip address 192.168.255.88 255.255.255.0

ip hold-time eigrp 5 120

ip pim dense-mode

speed 1000

duplex full

qos trust dscp

auto qos voip trust

tx-queue 3

   bandwidth percent 33

   priority high

   shape percent 33

service-policy output autoqos-voip-policy

end

Highlighted

EIGRP/PIM Routing Issues

sh proc cpu history

    4455555444444444444444444444444555554444444444444444444444

    6611111555555555566666666688888222226666677777777774444466

100                                                          

90                                                          

80                                                          

70                                                          

60                                                          

50 ***************************************************     **

40 **********************************************************

30 **********************************************************

20 **********************************************************

10 **********************************************************

   0....5....1....1....2....2....3....3....4....4....5....5....

             0    5    0    5    0    5    0    5    0    5   

               CPU% per second (last 60 seconds)

    5555555555555555555555555555655566555555555555555555555555

    2219121291111921109232492233123100121832449654373322732228

100                                                          

90                                                          

80                                                          

70                                                          

60    *    *    *    *    *    *   **   *    ***  *    *    *

50 ##########################################################

40 ##########################################################

30 ##########################################################

20 ##########################################################

10 ##########################################################

   0....5....1....1....2....2....3....3....4....4....5....5....

             0    5    0    5    0    5    0    5    0    5   

               CPU% per minute (last 60 minutes)

              * = maximum CPU%   # = average CPU%

    6866666976966666666669866666666669655655556676666555666666666666666656

    1281424916843000800018133102324138188099991186010889202101301000000081

100        *  *          *           *                                   

90        *  *          *           *                                   

80  *     *  *          **          *          *                        

70  **    ****     *    **          *          **                       

60 **********************************************************************

50 ######################################################################

40 ######################################################################

30 ######################################################################

20 ######################################################################

10 ######################################################################

   0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.

             0    5    0    5    0    5    0    5    0    5    0    5    0

                   CPU% per hour (last 72 hours)

                  * = maximum CPU%   # = average CPU%

(This is a Catalyst 4510R - which seem to run higher than other Catalysts)

Highlighted

EIGRP/PIM Routing Issues

sh ip eigrp traffic

EIGRP-IPv4 Traffic Statistics for AS(5)

  Hellos sent/received: 2283784/4731742

  Updates sent/received: 2186/3959

  Queries sent/received: 826/2020

  Replies sent/received: 1823/4543

  Acks sent/received: 8797/12634

  SIA-Queries sent/received: 0/0

  SIA-Replies sent/received: 0/0

  Hello Process ID: 256

  PDM Process ID: 255

  Socket Queue: 0/2000/49/0 (current/max/highest/drops)

  Input Queue: 0/2000/49/0 (current/max/highest/drops)

Highlighted

EIGRP/PIM Routing Issues

sh run

Building configuration...

Current configuration : 84101 bytes

!

! Last configuration change at 08:10:00 CST Tue Sep 6 2011 by admin

! NVRAM config last updated at 08:10:01 CST Tue Sep 6 2011 by admin

!

version 15.0

no service slave-log

no service pad

service tcp-keepalives-in

service tcp-keepalives-out

service timestamps debug datetime msec localtime show-timezone

service timestamps log datetime msec localtime show-timezone

service password-encryption

service compress-config

!

hostname kjhs-mdf-cs01

!

boot-start-marker

boot system flash bootflash:/cat4500-entservicesk9-mz.150-2.SG.bin

boot-end-marker

!

logging buffered 131072

enable secret 5 (omitted)

!

username admin privilege 15 secret 5 (omitted)

!

!

no aaa new-model

clock timezone CST -6

clock summer-time CST recurring

qos

qos dbl

qos map dscp 24 25 26 27 28 29 30 31 to tx-queue 4

qos map dscp 32 33 34 35 36 37 38 39 to tx-queue 4

qos map cos 5 to dscp 46

ip subnet-zero

ip domain-name unit5.org

ip name-server 10.49.1.10

ip name-server 10.49.1.12

ip name-server 10.49.94.2

ip vrf mgmtVrf

!

ip multicast-routing

ip dhcp excluded-address 10.49.95.1

ip dhcp excluded-address 10.49.95.10

!        

ip dhcp pool vlan_500

   network 10.49.95.0 255.255.255.0

   default-router 10.49.95.1

   domain-name unit5.org

   dns-server 10.49.1.10 10.49.1.12

   option 150 ip 10.49.111.10 10.49.3.10

   lease 0 8

!

!

!

!

crypto pki trustpoint TP-self-signed-90803

enrollment selfsigned

subject-name cn=IOS-Self-Signed-Certificate-90803

revocation-check none

rsakeypair TP-self-signed-90803

!

!

crypto pki certificate chain TP-self-signed-90803

certificate self-signed 01

  (omitted)

  quit

power redundancy-mode combined

!

!

!

!

spanning-tree mode rapid-pvst

spanning-tree extend system-id

spanning-tree vlan 1-4094 priority 24576

!

redundancy

mode sso

main-cpu

  auto-sync standard

!

vlan internal allocation policy ascending

!

!

policy-map autoqos-voip-policy

class class-default

    dbl

!

!

!

interface FastEthernet1

ip vrf forwarding mgmtVrf

no ip address

speed auto

duplex auto

!        

(interfaces omitted)

!

!

interface Vlan10

description SVI:  VLAN 10 - Building LAN

no ip address

!

interface Vlan90

ip address 192.168.88.1 255.255.255.0

!

interface Vlan101

description SVI:  VLAN 101 - Workstation VLAN

ip address 10.49.88.1 255.255.255.0

ip helper-address 10.49.94.2

ip pim dense-mode

!

interface Vlan102

description SVI:  VLAN 102 - Workstation VLAN

ip address 10.49.89.1 255.255.255.0

ip helper-address 10.49.94.2

ip pim dense-mode

!

interface Vlan103

description SVI:  VLAN 103 - Workstation VLAN

ip address 10.49.90.1 255.255.255.0

ip helper-address 10.49.94.2

ip pim dense-mode

!

interface Vlan104

description SVI:  VLAN 104 - Workstation VLAN

ip address 10.88.1.1 255.255.0.0 secondary

ip address 10.49.91.1 255.255.255.0

ip helper-address 10.49.94.2

no ip redirects

ip pim dense-mode

!

interface Vlan105

description SVI:  VLAN 105 - NOT IN USE - Reserved

no ip address

ip pim dense-mode

shutdown

!

interface Vlan106

description SVI:  VLAN 106 - NOT IN USE - Reserved

no ip address

ip pim dense-mode

shutdown

!

interface Vlan107

description SVI:  VLAN 107 - NOT IN USE - Reserved

no ip address

ip pim dense-mode

shutdown

!

interface Vlan108

description SVI:  VLAN 108 - NOT IN USE - Reserved

no ip address

ip pim dense-mode

shutdown

!

interface Vlan200

description SVI:  VLAN 200 - Network Servers & Printers

ip address 10.49.94.1 255.255.255.192

ip pim dense-mode

!

interface Vlan300

description SVI:  VLAN 300 - Facilities Network

ip address 10.49.94.65 255.255.255.192

ip pim dense-mode

!

interface Vlan400

description SVI:  VLAN 400 - Security Network

ip address 10.49.94.129 255.255.255.128

ip pim dense-mode

!

interface Vlan500

description SVI:  VLAN 500 - Voice Network

ip address 10.49.95.1 255.255.255.0

ip pim dense-mode

!

interface Vlan900

description SVI:  VLAN 900 - Network Management

ip address 10.49.246.1 255.255.255.0

ip pim dense-mode

!

interface Vlan1000

description SVI:  VLAN 1000 - Wireless WAN Management Network

ip address 10.49.226.97 255.255.255.248

ip pim dense-mode

!

!

router eigrp 5

network 10.49.0.0 0.0.255.255

network 192.168.255.0

eigrp router-id 192.168.255.88

!

ip default-gateway 10.49.246.1

no ip http server

ip http authentication local

ip http secure-server

!

!

!

logging trap debugging

logging 10.49.1.121

!

!

snmp-server engineID local 800000090300000C30ABAD64

snmp-server community (omitted)

snmp-server community (omitted)

snmp-server enable traps tty

!        

control-plane

!

!

line con 0

stopbits 1

line vty 0 4

login local

length 0

transport input telnet ssh

!

ntp clock-period 17179453

ntp server 10.49.1.1

end

Highlighted

EIGRP/PIM Routing Issues

sh ip eigrp neighbors detail

EIGRP-IPv4 Neighbors for AS(5)

H   Address                 Interface       Hold Uptime   SRTT   RTO  Q  Seq

                                            (sec)         (ms)       Cnt Num

5   192.168.255.84          Gi8/48            12 07:40:59   12   200  0  62222

   Version 12.2/1.2, Retrans: 2, Retries: 0, Prefixes: 141

   Topology-ids from peer - 0

0   192.168.255.4           Gi8/48           115 08:07:27   12   200  0  4617

   Version 5.1/3.0, Retrans: 6, Retries: 0, Prefixes: 33

   Topology-ids from peer - 0

3   192.168.255.8           Gi8/48           116 08:08:48   10   200  0  4817

   Version 5.1/3.0, Retrans: 5, Retries: 0, Prefixes: 33

   Topology-ids from peer - 0

18  192.168.255.60          Gi8/48           115 08:18:19   29   200  0  5039

   Version 5.1/3.0, Retrans: 8, Retries: 0, Prefixes: 7

   Topology-ids from peer - 0

6   192.168.255.56          Gi8/48           117 08:18:57   19   200  0  4429

   Version 5.1/3.0, Retrans: 9, Retries: 0, Prefixes: 7

   Topology-ids from peer - 0

13  192.168.255.52          Gi8/48           117 08:19:43   25   200  0  4387

   Version 5.1/3.0, Retrans: 9, Retries: 0, Prefixes: 7

   Topology-ids from peer - 0

19  192.168.255.48          Gi8/48           118 08:20:42   12   200  0  4354

   Version 5.1/3.0, Retrans: 7, Retries: 0, Prefixes: 7

   Topology-ids from peer - 0

1   192.168.255.36          Gi8/48           118 08:25:38   12   200  0  4168

   Version 5.1/3.0, Retrans: 10, Retries: 0, Prefixes: 7

   Topology-ids from peer - 0

22  192.168.255.20          Gi8/48           117 08:28:00   13   200  0  4593

   Version 5.1/3.0, Retrans: 9, Retries: 0, Prefixes: 7

   Topology-ids from peer - 0

11  192.168.255.100         Gi8/48           117 4d00h      39   234  0  5715

   Version 5.2/3.0, Retrans: 16, Retries: 0, Prefixes: 9

   Topology-ids from peer - 0

20  192.168.255.12          Gi8/48           116 4d02h      37   222  0  5413

   Version 5.2/3.0, Retrans: 18, Retries: 0, Prefixes: 9

   Topology-ids from peer - 0

23  192.168.255.16          Gi8/48           115 4d05h      39   234  0  5237

   Version 5.2/3.0, Retrans: 18, Retries: 0, Prefixes: 9

   Topology-ids from peer - 0

9   192.168.255.1           Gi8/48           115 4d05h      10   200  0  8222

   Version 5.1/3.0, Retrans: 18, Retries: 0, Prefixes: 31

   Topology-ids from peer - 0

21  192.168.255.64          Gi8/48           116 4d05h      55   330  0  4124

   Version 5.2/3.0, Retrans: 16, Retries: 0, Prefixes: 7

   Topology-ids from peer - 0

4   192.168.255.72          Gi8/48           119 4d05h      19   200  0  4337

   Version 5.2/3.0, Retrans: 16, Retries: 0, Prefixes: 7

   Topology-ids from peer - 0

14  192.168.255.82          Gi8/48           116 4d06h      37   222  0  3471

   Version 5.2/3.0, Retrans: 17, Retries: 0, Prefixes: 7

   Topology-ids from peer - 0

12  192.168.255.40          Gi8/48           119 4d07h      94   564  0  4757

   Version 5.2/3.0, Retrans: 19, Retries: 0, Prefixes: 7

   Topology-ids from peer - 0

15  192.168.255.32          Gi8/48           115 4d08h      15   200  0  3999

   Version 5.2/3.0, Retrans: 21, Retries: 0, Prefixes: 7

   Topology-ids from peer - 0

8   192.168.255.80          Gi8/48           118 5d03h     141   846  0  4119

   Version 5.2/3.0, Retrans: 26, Retries: 0, Prefixes: 7

   Topology-ids from peer - 0

17  192.168.255.76          Gi8/48           116 5d04h      33   200  0  4209

   Version 5.2/3.0, Retrans: 25, Retries: 0, Prefixes: 7

   Topology-ids from peer - 0

16  192.168.255.96          Gi8/48           119 5d06h      21   200  0  4508

   Version 5.2/3.0, Retrans: 29, Retries: 0, Prefixes: 7

   Topology-ids from peer - 0

7   192.168.255.28          Gi8/48           118 6d02h      16   200  0  3158

   Version 5.2/3.0, Retrans: 26, Retries: 0, Prefixes: 7

   Topology-ids from peer - 0

2   192.168.255.92          Gi8/48           116 6d02h      30   200  0  4472

   Version 5.2/3.0, Retrans: 33, Retries: 0, Prefixes: 19

   Topology-ids from peer - 0

Highlighted
Participant

EIGRP/PIM Routing Issues

Please share "show int gi8/48"

Highlighted

Re: EIGRP/PIM Routing Issues

Amit - I tried to share that as well, but it kept saying that "this message can not be displayed due to its content."

This is all that it would take...

  L3 in Switched: ucast: 293639320 pkt, 242648536673 bytes - mcast: 82577516 pkt, 17440865645 bytes

  L3 out Switched: ucast: 209704398 pkt, 50728868903 bytes - mcast: 93614 pkt, 17561681 bytes

     393886374 packets input, 268468312912 bytes, 0 no buffer

     Received 100245154 broadcasts (87559213 IP multicasts)

     438 input errors, 78 CRC, 0 frame, 11518 overrun, 0 ignored

     210857959 packets output, 55039504285 bytes, 0 underruns

Highlighted
Participant

EIGRP/PIM Routing Issues

438 input errors, 78 CRC, 0 frame, 11518 overrun, 0 ignored

Let's watch out for this counter, I would like to make sure that this counter isn't incrementing.

Also, do you see any drops in the input queue on this interface?

I see high CPU util as well, that could very well be a cause on why adjacencies would flap or it may be happening due to the flap itself. So, CPU could be the culprit or victim.

To find out what it is, I would like to see what processes in the o/p of "show proc cpu sorted" are eating up maximum resources, if its "ip input" then CPU is the victim.

Regards,

Amit

Highlighted

EIGRP/PIM Routing Issues

Input queue: 1/75/0/0 (size/max/drops/flushes); Total output drops: 358

sh proc cpu sorted shows these two as the top two:

  58   141164142  63869252       2210 17.82% 19.92% 19.98%   0 Cat4k Mgmt LoPri

  57   127462404 187599343        679 14.39% 15.35% 15.41%   0 Cat4k Mgmt HiPri

and IP input is 5 or 6 down in the list currently

117     6711019  58522887        114  0.63%  0.69%  0.67%   0 IP Input  

Highlighted
Participant

EIGRP/PIM Routing Issues

The above two processes normally show this much utilization, so this looks normal.

I will also see if the output drops increase when flapping happens again.

Highlighted

EIGRP/PIM Routing Issues

Okay.  I will keep an eye on it tomorrow and see what happens when it is doing it again.  It normally only happens during regular work hours. 

Highlighted

EIGRP/PIM Routing Issues

This morning, it started back up, and from the int g8/48 there is this:

438 input errors, 78 CRC, 0 frame, 11544 overrun, 0 ignored

Which looks to be more overrun errors, but the same # of input errors.

The processes show IP input running at .6% average.

Input queue: 2/75/0/0 (size/max/drops/flushes); Total output drops: 358

CreatePlease to create content
Content for Community-Ad