09-06-2011 07:57 AM - edited 03-07-2019 02:04 AM
Hi,
We have been having an issue with our routers for the past couple of weeks where there are a lot of EIGRP/PIM error messages in the logs.
This flapping is occuring on a handful of our core switches. They are in the following format:
Sep 6 08:15:18.785 CST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 5: Neighbor 192.168.255.4 (GigabitEthernet8/48) is up: new adjacency
Sep 6 08:18:36.569 CST: %PIM-5-NBRCHG: neighbor 192.168.255.82 DOWN on interface GigabitEthernet8/48 non DR
Sep 6 08:18:50.612 CST: %PIM-5-NBRCHG: neighbor 192.168.255.82 UP on interface GigabitEthernet8/48
Sep 6 08:41:46.275 CST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 5: Neighbor 192.168.255.84 (GigabitEthernet8/48) is down: holding time expired
Sep 6 08:41:46.763 CST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 5: Neighbor 192.168.255.84 (GigabitEthernet8/48) is up: new adjacency
Sep 6 09:02:49.908 CST: %PIM-5-NBRCHG: neighbor 192.168.255.56 DOWN on interface GigabitEthernet8/48 non DR
Sep 6 09:03:02.904 CST: %PIM-5-NBRCHG: neighbor 192.168.255.56 UP on interface GigabitEthernet8/48
Sep 6 09:05:16.629 CST: %PIM-5-NBRCHG: neighbor 192.168.255.48 DOWN on interface GigabitEthernet8/48 non DR
Sep 6 09:05:30.704 CST: %PIM-5-NBRCHG: neighbor 192.168.255.48 UP on interface GigabitEthernet8/48
Sep 6 09:38:53.536 CST: %PIM-5-NBRCHG: neighbor 192.168.255.4 DOWN on interface GigabitEthernet8/48 non DR
Sep 6 09:39:07.351 CST: %PIM-5-NBRCHG: neighbor 192.168.255.4 UP on interface GigabitEthernet8/48
Sep 6 09:39:56.190 CST: %PIM-5-NBRCHG: neighbor 192.168.255.100 DOWN on interface GigabitEthernet8/48 non DR
Sep 6 09:40:09.106 CST: %PIM-5-NBRCHG: neighbor 192.168.255.100 UP on interface GigabitEthernet8/48
All the WAN facing interfaces are configured the exact same:
ip hold-time eigrp 5 120
ip pim dense-mode
speed 1000
duplex full
We have about 25 sites and it seems to be happening between only a handful. We had changed the hold time for EIGRP to try and fix the %DUAL-5-NBRCHANGE errors as we saw that it was exceeding the default hold-time. Now it seems that we are seeing the PIM errors more today than the NBRCHANGE errors.
Any help would be much appreciated!
Thanks!
09-06-2011 12:25 PM
Hi,
Could you please share the following o/p from this router?
show run
sho int gi8/48
show proc cpu hist
show ip eigrp traffic
Since PIM is also flapping, I would assume that there is multicast packets being dropped. To isolate it & rule out if multicast is the what's being dropped, can you try "neighbor" command in EIGRP with one of its neighbor & see if it works well with unicast?
I would also look for drops on gi8/48...
Regards,
Amit
09-06-2011 02:24 PM
Amit,
Thank you for replying. Here is the information you requested!
sh run int gi8/48
interface GigabitEthernet8/48
description Fiber Connection
no switchport
ip address 192.168.255.88 255.255.255.0
ip hold-time eigrp 5 120
ip pim dense-mode
speed 1000
duplex full
qos trust dscp
auto qos voip trust
tx-queue 3
bandwidth percent 33
priority high
shape percent 33
service-policy output autoqos-voip-policy
end
09-06-2011 02:25 PM
sh proc cpu history
4455555444444444444444444444444555554444444444444444444444
6611111555555555566666666688888222226666677777777774444466
100
90
80
70
60
50 *************************************************** **
40 **********************************************************
30 **********************************************************
20 **********************************************************
10 **********************************************************
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per second (last 60 seconds)
5555555555555555555555555555655566555555555555555555555555
2219121291111921109232492233123100121832449654373322732228
100
90
80
70
60 * * * * * * ** * *** * * *
50 ##########################################################
40 ##########################################################
30 ##########################################################
20 ##########################################################
10 ##########################################################
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%
6866666976966666666669866666666669655655556676666555666666666666666656
1281424916843000800018133102324138188099991186010889202101301000000081
100 * * * *
90 * * * *
80 * * * ** * *
70 ** **** * ** * **
60 **********************************************************************
50 ######################################################################
40 ######################################################################
30 ######################################################################
20 ######################################################################
10 ######################################################################
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%
(This is a Catalyst 4510R - which seem to run higher than other Catalysts)
09-06-2011 02:26 PM
sh ip eigrp traffic
EIGRP-IPv4 Traffic Statistics for AS(5)
Hellos sent/received: 2283784/4731742
Updates sent/received: 2186/3959
Queries sent/received: 826/2020
Replies sent/received: 1823/4543
Acks sent/received: 8797/12634
SIA-Queries sent/received: 0/0
SIA-Replies sent/received: 0/0
Hello Process ID: 256
PDM Process ID: 255
Socket Queue: 0/2000/49/0 (current/max/highest/drops)
Input Queue: 0/2000/49/0 (current/max/highest/drops)
09-06-2011 02:26 PM
sh run
Building configuration...
Current configuration : 84101 bytes
!
! Last configuration change at 08:10:00 CST Tue Sep 6 2011 by admin
! NVRAM config last updated at 08:10:01 CST Tue Sep 6 2011 by admin
!
version 15.0
no service slave-log
no service pad
service tcp-keepalives-in
service tcp-keepalives-out
service timestamps debug datetime msec localtime show-timezone
service timestamps log datetime msec localtime show-timezone
service password-encryption
service compress-config
!
hostname kjhs-mdf-cs01
!
boot-start-marker
boot system flash bootflash:/cat4500-entservicesk9-mz.150-2.SG.bin
boot-end-marker
!
logging buffered 131072
enable secret 5 (omitted)
!
username admin privilege 15 secret 5 (omitted)
!
!
no aaa new-model
clock timezone CST -6
clock summer-time CST recurring
qos
qos dbl
qos map dscp 24 25 26 27 28 29 30 31 to tx-queue 4
qos map dscp 32 33 34 35 36 37 38 39 to tx-queue 4
qos map cos 5 to dscp 46
ip subnet-zero
ip domain-name unit5.org
ip name-server 10.49.1.10
ip name-server 10.49.1.12
ip name-server 10.49.94.2
ip vrf mgmtVrf
!
ip multicast-routing
ip dhcp excluded-address 10.49.95.1
ip dhcp excluded-address 10.49.95.10
!
ip dhcp pool vlan_500
network 10.49.95.0 255.255.255.0
default-router 10.49.95.1
domain-name unit5.org
dns-server 10.49.1.10 10.49.1.12
option 150 ip 10.49.111.10 10.49.3.10
lease 0 8
!
!
!
!
crypto pki trustpoint TP-self-signed-90803
enrollment selfsigned
subject-name cn=IOS-Self-Signed-Certificate-90803
revocation-check none
rsakeypair TP-self-signed-90803
!
!
crypto pki certificate chain TP-self-signed-90803
certificate self-signed 01
(omitted)
quit
power redundancy-mode combined
!
!
!
!
spanning-tree mode rapid-pvst
spanning-tree extend system-id
spanning-tree vlan 1-4094 priority 24576
!
redundancy
mode sso
main-cpu
auto-sync standard
!
vlan internal allocation policy ascending
!
!
policy-map autoqos-voip-policy
class class-default
dbl
!
!
!
interface FastEthernet1
ip vrf forwarding mgmtVrf
no ip address
speed auto
duplex auto
!
(interfaces omitted)
!
!
interface Vlan10
description SVI: VLAN 10 - Building LAN
no ip address
!
interface Vlan90
ip address 192.168.88.1 255.255.255.0
!
interface Vlan101
description SVI: VLAN 101 - Workstation VLAN
ip address 10.49.88.1 255.255.255.0
ip helper-address 10.49.94.2
ip pim dense-mode
!
interface Vlan102
description SVI: VLAN 102 - Workstation VLAN
ip address 10.49.89.1 255.255.255.0
ip helper-address 10.49.94.2
ip pim dense-mode
!
interface Vlan103
description SVI: VLAN 103 - Workstation VLAN
ip address 10.49.90.1 255.255.255.0
ip helper-address 10.49.94.2
ip pim dense-mode
!
interface Vlan104
description SVI: VLAN 104 - Workstation VLAN
ip address 10.88.1.1 255.255.0.0 secondary
ip address 10.49.91.1 255.255.255.0
ip helper-address 10.49.94.2
no ip redirects
ip pim dense-mode
!
interface Vlan105
description SVI: VLAN 105 - NOT IN USE - Reserved
no ip address
ip pim dense-mode
shutdown
!
interface Vlan106
description SVI: VLAN 106 - NOT IN USE - Reserved
no ip address
ip pim dense-mode
shutdown
!
interface Vlan107
description SVI: VLAN 107 - NOT IN USE - Reserved
no ip address
ip pim dense-mode
shutdown
!
interface Vlan108
description SVI: VLAN 108 - NOT IN USE - Reserved
no ip address
ip pim dense-mode
shutdown
!
interface Vlan200
description SVI: VLAN 200 - Network Servers & Printers
ip address 10.49.94.1 255.255.255.192
ip pim dense-mode
!
interface Vlan300
description SVI: VLAN 300 - Facilities Network
ip address 10.49.94.65 255.255.255.192
ip pim dense-mode
!
interface Vlan400
description SVI: VLAN 400 - Security Network
ip address 10.49.94.129 255.255.255.128
ip pim dense-mode
!
interface Vlan500
description SVI: VLAN 500 - Voice Network
ip address 10.49.95.1 255.255.255.0
ip pim dense-mode
!
interface Vlan900
description SVI: VLAN 900 - Network Management
ip address 10.49.246.1 255.255.255.0
ip pim dense-mode
!
interface Vlan1000
description SVI: VLAN 1000 - Wireless WAN Management Network
ip address 10.49.226.97 255.255.255.248
ip pim dense-mode
!
!
router eigrp 5
network 10.49.0.0 0.0.255.255
network 192.168.255.0
eigrp router-id 192.168.255.88
!
ip default-gateway 10.49.246.1
no ip http server
ip http authentication local
ip http secure-server
!
!
!
logging trap debugging
logging 10.49.1.121
!
!
snmp-server engineID local 800000090300000C30ABAD64
snmp-server community (omitted)
snmp-server community (omitted)
snmp-server enable traps tty
!
control-plane
!
!
line con 0
stopbits 1
line vty 0 4
login local
length 0
transport input telnet ssh
!
ntp clock-period 17179453
ntp server 10.49.1.1
end
09-06-2011 02:29 PM
sh ip eigrp neighbors detail
EIGRP-IPv4 Neighbors for AS(5)
H Address Interface Hold Uptime SRTT RTO Q Seq
(sec) (ms) Cnt Num
5 192.168.255.84 Gi8/48 12 07:40:59 12 200 0 62222
Version 12.2/1.2, Retrans: 2, Retries: 0, Prefixes: 141
Topology-ids from peer - 0
0 192.168.255.4 Gi8/48 115 08:07:27 12 200 0 4617
Version 5.1/3.0, Retrans: 6, Retries: 0, Prefixes: 33
Topology-ids from peer - 0
3 192.168.255.8 Gi8/48 116 08:08:48 10 200 0 4817
Version 5.1/3.0, Retrans: 5, Retries: 0, Prefixes: 33
Topology-ids from peer - 0
18 192.168.255.60 Gi8/48 115 08:18:19 29 200 0 5039
Version 5.1/3.0, Retrans: 8, Retries: 0, Prefixes: 7
Topology-ids from peer - 0
6 192.168.255.56 Gi8/48 117 08:18:57 19 200 0 4429
Version 5.1/3.0, Retrans: 9, Retries: 0, Prefixes: 7
Topology-ids from peer - 0
13 192.168.255.52 Gi8/48 117 08:19:43 25 200 0 4387
Version 5.1/3.0, Retrans: 9, Retries: 0, Prefixes: 7
Topology-ids from peer - 0
19 192.168.255.48 Gi8/48 118 08:20:42 12 200 0 4354
Version 5.1/3.0, Retrans: 7, Retries: 0, Prefixes: 7
Topology-ids from peer - 0
1 192.168.255.36 Gi8/48 118 08:25:38 12 200 0 4168
Version 5.1/3.0, Retrans: 10, Retries: 0, Prefixes: 7
Topology-ids from peer - 0
22 192.168.255.20 Gi8/48 117 08:28:00 13 200 0 4593
Version 5.1/3.0, Retrans: 9, Retries: 0, Prefixes: 7
Topology-ids from peer - 0
11 192.168.255.100 Gi8/48 117 4d00h 39 234 0 5715
Version 5.2/3.0, Retrans: 16, Retries: 0, Prefixes: 9
Topology-ids from peer - 0
20 192.168.255.12 Gi8/48 116 4d02h 37 222 0 5413
Version 5.2/3.0, Retrans: 18, Retries: 0, Prefixes: 9
Topology-ids from peer - 0
23 192.168.255.16 Gi8/48 115 4d05h 39 234 0 5237
Version 5.2/3.0, Retrans: 18, Retries: 0, Prefixes: 9
Topology-ids from peer - 0
9 192.168.255.1 Gi8/48 115 4d05h 10 200 0 8222
Version 5.1/3.0, Retrans: 18, Retries: 0, Prefixes: 31
Topology-ids from peer - 0
21 192.168.255.64 Gi8/48 116 4d05h 55 330 0 4124
Version 5.2/3.0, Retrans: 16, Retries: 0, Prefixes: 7
Topology-ids from peer - 0
4 192.168.255.72 Gi8/48 119 4d05h 19 200 0 4337
Version 5.2/3.0, Retrans: 16, Retries: 0, Prefixes: 7
Topology-ids from peer - 0
14 192.168.255.82 Gi8/48 116 4d06h 37 222 0 3471
Version 5.2/3.0, Retrans: 17, Retries: 0, Prefixes: 7
Topology-ids from peer - 0
12 192.168.255.40 Gi8/48 119 4d07h 94 564 0 4757
Version 5.2/3.0, Retrans: 19, Retries: 0, Prefixes: 7
Topology-ids from peer - 0
15 192.168.255.32 Gi8/48 115 4d08h 15 200 0 3999
Version 5.2/3.0, Retrans: 21, Retries: 0, Prefixes: 7
Topology-ids from peer - 0
8 192.168.255.80 Gi8/48 118 5d03h 141 846 0 4119
Version 5.2/3.0, Retrans: 26, Retries: 0, Prefixes: 7
Topology-ids from peer - 0
17 192.168.255.76 Gi8/48 116 5d04h 33 200 0 4209
Version 5.2/3.0, Retrans: 25, Retries: 0, Prefixes: 7
Topology-ids from peer - 0
16 192.168.255.96 Gi8/48 119 5d06h 21 200 0 4508
Version 5.2/3.0, Retrans: 29, Retries: 0, Prefixes: 7
Topology-ids from peer - 0
7 192.168.255.28 Gi8/48 118 6d02h 16 200 0 3158
Version 5.2/3.0, Retrans: 26, Retries: 0, Prefixes: 7
Topology-ids from peer - 0
2 192.168.255.92 Gi8/48 116 6d02h 30 200 0 4472
Version 5.2/3.0, Retrans: 33, Retries: 0, Prefixes: 19
Topology-ids from peer - 0
09-06-2011 02:45 PM
Please share "show int gi8/48"
09-06-2011 03:08 PM
Amit - I tried to share that as well, but it kept saying that "this message can not be displayed due to its content."
This is all that it would take...
L3 in Switched: ucast: 293639320 pkt, 242648536673 bytes - mcast: 82577516 pkt, 17440865645 bytes
L3 out Switched: ucast: 209704398 pkt, 50728868903 bytes - mcast: 93614 pkt, 17561681 bytes
393886374 packets input, 268468312912 bytes, 0 no buffer
Received 100245154 broadcasts (87559213 IP multicasts)
438 input errors, 78 CRC, 0 frame, 11518 overrun, 0 ignored
210857959 packets output, 55039504285 bytes, 0 underruns
09-06-2011 03:12 PM
438 input errors, 78 CRC, 0 frame, 11518 overrun, 0 ignored
Let's watch out for this counter, I would like to make sure that this counter isn't incrementing.
Also, do you see any drops in the input queue on this interface?
I see high CPU util as well, that could very well be a cause on why adjacencies would flap or it may be happening due to the flap itself. So, CPU could be the culprit or victim.
To find out what it is, I would like to see what processes in the o/p of "show proc cpu sorted" are eating up maximum resources, if its "ip input" then CPU is the victim.
Regards,
Amit
09-06-2011 03:30 PM
Input queue: 1/75/0/0 (size/max/drops/flushes); Total output drops: 358
sh proc cpu sorted shows these two as the top two:
58 141164142 63869252 2210 17.82% 19.92% 19.98% 0 Cat4k Mgmt LoPri
57 127462404 187599343 679 14.39% 15.35% 15.41% 0 Cat4k Mgmt HiPri
and IP input is 5 or 6 down in the list currently
117 6711019 58522887 114 0.63% 0.69% 0.67% 0 IP Input
09-06-2011 03:33 PM
The above two processes normally show this much utilization, so this looks normal.
I will also see if the output drops increase when flapping happens again.
09-06-2011 03:37 PM
Okay. I will keep an eye on it tomorrow and see what happens when it is doing it again. It normally only happens during regular work hours.
09-07-2011 06:37 AM
This morning, it started back up, and from the int g8/48 there is this:
438 input errors, 78 CRC, 0 frame, 11544 overrun, 0 ignored
Which looks to be more overrun errors, but the same # of input errors.
The processes show IP input running at .6% average.
Input queue: 2/75/0/0 (size/max/drops/flushes); Total output drops: 358
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: