02-07-2012 12:50 PM - edited 03-07-2019 04:47 AM
I have a large network experencing an issue.
Here are some details of the problem-
All active interfaces on all switches on the network have a high number of underruns and output buffer failures.
When problem occurs, test computer plugged into network experiences 100% cpu load.
Router freezes (linux based router).
All remote switches are 3550s. (C3550-I9Q3L2-M), Version 12.1(22)EA8a
Core switch is 6500, WS-X6K-S2U-MSFC2. (c6sup2_rp-PSV-M), Version 12.1(27b)E3
Star topology, all uplinks are 1gb.
1 VLAN
Network is pppoe based.
Broadcast storm control is enabled at a level of 10% on all 100mb access ports. (recent addition)
Any thoughts on what the issue could be? I had assumed it was some sort of broadcast storm, but network is still having the issue.
The problem is not consistant and will appear and disapear for short to long periods.
Below is output of an example interface :
-------
29914103 packets input, 3854413165 bytes, 0 no buffer
Received 277951 broadcasts (0 multicast)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 264622 multicast, 0 pause input
0 input packets with dribble condition detected
118378232 packets output, 2088884846 bytes, 281203864 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
281203864 output buffer failures, 0 output buffers swapped out 29914103 packets input, 3854413165 bytes, 0 no buffer
Received 277951 broadcasts (0 multicast)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 264622 multicast, 0 pause input
0 input packets with dribble condition detected
118378232 packets output, 2088884846 bytes, 281203864 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
281203864 output buffer failures, 0 output buffers swapped out
------
Any thoughts are welcome, Thanks.
02-07-2012 05:58 PM
Hi,
Could you please post the output of show run? (Do remove all sensitive information). Do you even see a high CPU utilization on the switch?
There are many reasons which may cause a high drops.
Thanks
Vivek
02-07-2012 06:35 PM
Core switch :
--------------------------------------------------------------------------------
!
version 12.1
service timestamps debug datetime localtime
service timestamps log datetime localtime
service password-encryption
service counters max age 10
!
clock timezone EST -5
clock summer-time EDT recurring
vtp mode transparent
ip subnet-zero
!
!
mls flow ip destination
mls flow ipx destination
!
spanning-tree mode rapid-pvst
spanning-tree vlan 1 priority 4094
!
redundancy
mode rpr-plus
main-cpu
auto-sync running-config
auto-sync standard
!
!
!
!
! ALL Interfaces are as follows:
!
!
!
!
interface GigabitEthernetx/x
description XX
switchport
switchport mode access
storm-control broadcast level 2.00
!
interface Vlan1
description MGMT
ip address x.x.x.x x.x.x.x
no ip redirects
!
no ip http server
!
logging trap debugging
----------------------------------------------------------------------
Remote switches:
----------------------------------------------------------------------
version 12.1
no service pad
service timestamps debug datetime localtime
service timestamps log datetime localtime
service password-encryption
!
ip subnet-zero
!
no ip domain-lookup
vtp mode transparent
!
!
spanning-tree mode rapid-pvst
spanning-tree portfast bpduguard default
spanning-tree extend system-id
!
!
! ALL Interfaces are as follows:
!
!
interface FastEthernet0/x
switchport mode access
storm-control broadcast level 10.00
spanning-tree portfast
spanning-tree guard root
!
!
interface GigabitEthernet0/1
description UPLINK
switchport mode access
!
interface Vlan1
description MGMT
ip address x.x.x.x x.x.x.x
no ip route-cache
!
ip classless
no ip http server
----------------------------------------------------------------
02-07-2012 06:42 PM
I also noticed this:
#show mls statistics
Statistics for Earl in Module 1
L2 Forwarding Engine
Total packets Switched : 90215938952
L3 Forwarding Engine
Total Packets L3 Switched : 90215938952 @ 30560 pps
Total Packets Bridged : 63125899407
Total Packets FIB Switched : 439
Total Packets ACL Routed : 0
Total Packets Netflow Switched : 0
Total Mcast Packets Switched/Routed : 27082771033
02-08-2012 06:45 AM
CPU utilization is very low on all switches. I am not seeing any spikes.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide