Buy or Renew
Buy or Renew
Cisco + Splunk: It’s a new day for your data. Learn more.
 
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
cancel
Start a conversation
1589
Views
0
Helpful
12
Replies

6509-E performance issues with two separate modules

andy.bussiere
Level 1
Level 1

‎02-23-2011 07:44 AM - edited ‎03-06-2019 03:42 PM

I have a 6509-e with the following four modules:

1   16  SFM-capable 16 port 1000mb GBIC        WS-X6516A-GBIC

2   16  16 port 1000mb GBIC ethernet           WS-X6416-GBIC

5    9  Supervisor Engine 32 8GE (Active)      WS-SUP32-GE-3B

6    9  Supervisor Engine 32 8GE (Hot)         WS-SUP32-GE-3B

A few days ago, I started having performance issues with the edge switches connected to the 2nd module (WS-X6416-GBIC) in this 6509-E. I could ping the edge switches connected to the 6416, but they would only reply about half the time, with the other half timing out. I moved all edge switches connected to the 6416 up to the 1st module (WS-X6516A-GBIC). As soon as I moved them, all edge switches started replying more consistant, which is how it's supposed to be.

Three days later, it started again, but on the 6516A. All edge switches connected to the 6516A started timing out during a ping about half the time. As you can expect, my end users are seeing very large performance issues. While physically at the switch, I tried to move a couple edge switches back to the 6416 and the port err-disabled immediately with a UDLD error, UDLD-SP-4-UDLD_PORT_DISABLED.

I then started moving things down to the module 5 supervisor and the edge switches started acting normal again. I did move something to the module 6 supervisor and it err-disabled immedialtey with the same UDLD error.

While in the middle of troubleshooting, I rebooted my 6509-E a couple of times. On one reboot, it booted up to rommom. I then rebooted it again and it came up normal.

Any ideas on what my problem could be?

Labels:
0 Helpful
12 Replies 12

kapathak
Cisco Employee
Cisco Employee

‎02-23-2011 08:35 AM

Can you paste the outputs from following

#sh spann de ac | i exe|Number of top|from
#show int | i drops| up
#show proc cpu history
#show int from any one of the affected ports



0 Helpful

andy.bussiere
Level 1
Level 1
In response to kapathak

‎02-23-2011 08:57 AM

Thanks for the response. See below:

sh spann de ac | i exe|Number of top|from
VLAN0001 is executing the ieee compatible Spanning Tree protocol
  Number of topology changes 77 last change occurred 02:49:03 ago
          from GigabitEthernet5/7
VLAN0010 is executing the ieee compatible Spanning Tree protocol
  Number of topology changes 78 last change occurred 02:49:03 ago
          from GigabitEthernet5/7
VLAN0028 is executing the ieee compatible Spanning Tree protocol
  Number of topology changes 84 last change occurred 02:49:03 ago
          from GigabitEthernet5/7
VLAN0032 is executing the ieee compatible Spanning Tree protocol
  Number of topology changes 79 last change occurred 02:49:03 ago
          from GigabitEthernet5/7
VLAN0033 is executing the ieee compatible Spanning Tree protocol
  Number of topology changes 47 last change occurred 02:49:03 ago
          from GigabitEthernet5/7
VLAN0034 is executing the ieee compatible Spanning Tree protocol
  Number of topology changes 82 last change occurred 02:49:03 ago
          from GigabitEthernet5/7
VLAN0035 is executing the ieee compatible Spanning Tree protocol
  Number of topology changes 82 last change occurred 02:49:03 ago
          from GigabitEthernet5/7
VLAN0036 is executing the ieee compatible Spanning Tree protocol
  Number of topology changes 78 last change occurred 02:49:03 ago
--More--                                     from GigabitEthernet5/7
VLAN0037 is executing the ieee compatible Spanning Tree protocol
  Number of topology changes 38 last change occurred 02:49:05 ago
          from GigabitEthernet5/7
VLAN0040 is executing the ieee compatible Spanning Tree protocol
  Number of topology changes 84 last change occurred 02:49:05 ago
          from GigabitEthernet5/7
VLAN0044 is executing the ieee compatible Spanning Tree protocol
  Number of topology changes 84 last change occurred 02:49:05 ago
          from GigabitEthernet5/7
PBNSTSC-32-13#show int | i drops| up
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Vlan32 is up, line protocol is up
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
GigabitEthernet1/8 is up, line protocol is up (connected)
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
GigabitEthernet1/12 is up, line protocol is up (connected)
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
GigabitEthernet1/14 is up, line protocol is up (connected)
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
GigabitEthernet1/16 is up, line protocol is up (connected)
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
--More--                             Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
GigabitEthernet5/1 is up, line protocol is up (connected)
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
GigabitEthernet5/2 is up, line protocol is up (connected)
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
GigabitEthernet5/3 is up, line protocol is up (connected)
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
GigabitEthernet5/4 is up, line protocol is up (connected)
--More--                             Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
GigabitEthernet5/5 is up, line protocol is up (connected)
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
GigabitEthernet5/6 is up, line protocol is up (connected)
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
GigabitEthernet5/7 is up, line protocol is up (connected)
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
GigabitEthernet5/8 is up, line protocol is up (connected)
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
GigabitEthernet5/9 is up, line protocol is up (connected)
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
PBNSTSC-32-13#    show proc cpu history
                                                             
                                                             
        3333311111333336666622222     222224444477777     1111
100                                                          
90                                                          
80                                                          
70                                                          
60                                                          
50                                                          
40                                                          
30                                                          
20                                                          
10                    *****                    *****        
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5   
               CPU% per second (last 60 seconds)
                                                             
               4444                                          
    7776666463424245589679966665655686688565666556566566659655
100                                                          
90                                                          
80                                                          
70                                                          
--More--                            60                                                          
50                                                          
40            *##*                                          
30            *###                                          
20            *###                                          
10 ******* *  ####*******************************************
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5   
               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%
                                                                         
      4  1       12  2 1   23                                            
    9958808998888078879089885                                            
100                                                                      
90                                                                      
80                                                                      
70                                                                      
60                                                                      
50   *                                                                  
40   *                     *                                            
30   *           *  *     **                                            
20   *           *  *     **                                            
10 *************************                                            
--More--                              0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
             0    5    0    5    0    5    0    5    0    5    0    5    0
                   CPU% per hour (last 72 hours)
                  * = maximum CPU%   # = average CPU%

PBNSTSC-32-13#show int gi1/9
GigabitEthernet1/9 is down, line protocol is down (notconnect)
  Hardware is C6k 1000Mb 802.3, address is 0026.cb57.9580 (bia 0026.cb57.9580)
  Description: PBNSSG-32-34 Gi0/1
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 0/255, rxload 0/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is SX
  input flow-control is off, output flow-control is off
  Clock mode is auto
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 02:51:36, output 02:51:16, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     472390 packets input, 120077663 bytes, 0 no buffer
     Received 50094 broadcasts (31367 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
--More--                                0 input packets with dribble condition detected
     4886095 packets output, 914371715 bytes, 0 underruns
     0 output errors, 0 collisions, 4 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out
PBNSTSC-32-13#show int gi1/9         2.   /16
GigabitEthernet2/16 is down, line protocol is down (err-disabled)
  Hardware is C6k 1000Mb 802.3, address is 0003.32bd.cb78 (bia 0003.32bd.cb78)
  Description: PBNSNSB-32-11 Gi4/2
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s
  input flow-control is off, output flow-control is off
  Clock mode is auto
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 1d01h, output never, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     0 packets input, 0 bytes, 0 no buffer
     Received 0 broadcasts (0 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
--More--                                0 input packets with dribble condition detected
     0 packets output, 0 bytes, 0 underruns
     0 output errors, 0 collisions, 3 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out
PBNSTSC-32-13#

0 Helpful

kapathak
Cisco Employee
Cisco Employee
In response to andy.bussiere

‎02-23-2011 09:10 AM

When was the last occurence of the problem?

What is connected to 5/7 ?

0 Helpful

andy.bussiere
Level 1
Level 1
In response to kapathak

‎02-23-2011 09:19 AM

The last occurance was 5-9 hours ago on Gi1/9. First is was responding about half the time before it finally disabled itself. See syslog below.

2011-02-23 05:40:20   Local7.Notice  172.27.32.13   445: Feb 23 11:38:35.211: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/9, changed state to down

2011-02-23 05:40:20   Local7.Error     172.27.32.13   446: Feb 23 11:38:35.235: %LINK-3-UPDOWN: Interface GigabitEthernet1/9, changed state to down

2011-02-23 05:40:20   Local7.Warning            172.27.32.13   447: Feb 23 11:38:35.211: %UDLD-SP-4-UDLD_PORT_DISABLED: UDLD disabled interface Gi1/9, unidirectional link detected

2011-02-23 05:40:20   Local7.Warning            172.27.32.13   448: Feb 23 11:38:35.211: %PM-SP-4-ERR_DISABLE: udld error detected on Gi1/9, putting Gi1/9 in err-disable state

2011-02-23 05:40:21   Local7.Notice  172.27.32.13   449: Feb 23 11:38:35.243: %LINEPROTO-SP-5-UPDOWN: Line protocol on Interface GigabitEthernet1/9, changed state to down

2011-02-23 05:40:21   Local7.Error     172.27.32.13   450: Feb 23 11:38:35.243: %LINK-SP-3-UPDOWN: Interface GigabitEthernet1/9, changed state to down

2011-02-23 05:40:22   Local7.Warning            172.27.32.13   451: Feb 23 11:38:35.343: %PM-SPSTBY-4-ERR_DISABLE: udld error detected on Gi1/9, putting Gi1/9 in err-disable state

I moved the edge switch after this error from Gi1/9 to Gi5/7. Since then, it's been running fine.

0 Helpful

kapathak
Cisco Employee
Cisco Employee
In response to andy.bussiere

‎02-23-2011 09:47 AM

So, is this problem seen with mutiple edge switches or  any particular edge switch. I would also check the health of fibres and the sfps. UDLD can sometimes erroneously put a port into errdisabled if any one of the switches at either end has High CPU which may utimately result in a failure to send UDLD keepalives.

0 Helpful

andy.bussiere
Level 1
Level 1
In response to kapathak

‎02-23-2011 09:54 AM

This issue is with all edge switches connected through these modules. There are 14 different edge switches connected using 14 different routes and GBICs.

Do you think the IOS could be corrupt or the primary sup32 could be failing?

0 Helpful

niro
Level 1
Level 1
In response to andy.bussiere

‎02-23-2011 10:09 AM

What's the output of "show spanning-tree root"?

0 Helpful

andy.bussiere
Level 1
Level 1
In response to niro

‎02-23-2011 10:14 AM

PBNSTSC-32-13#show spanning-tree root

                                        Root Hello Max Fwd
Vlan                   Root ID          Cost  Time Age Dly  Root Port
---------------- -------------------- ------ ----- --- ---  ----------------
VLAN0001           100 00d0.050c.4000         4    2   20  15  Gi5/8
VLAN0010           100 00d0.050c.4009         4    2   20  15  Gi5/8
VLAN0028           100 00d0.050c.401b         4    2   20  15  Gi5/8
VLAN0032           100 00d0.050c.401f         4    2   20  15  Gi5/8
VLAN0033         32768 0007.0e8c.c484         8    2   20  15  Gi1/16
VLAN0034           100 00d0.050c.4021         4    2   20  15  Gi5/8
VLAN0035           100 00d0.050c.4022         4    2   20  15  Gi5/8
VLAN0036           100 00d0.050c.4023         4    2   20  15  Gi5/8
VLAN0037           100 00d0.050c.4024         8    2   20  15  Gi1/14
VLAN0040           100 00d0.050c.4027         4    2   20  15  Gi5/8
VLAN0044           100 00d0.050c.402b         4    2   20  15  Gi5/8

0 Helpful

niro
Level 1
Level 1
In response to andy.bussiere

‎02-23-2011 10:21 AM

What's attached to port 5/8, 1/14 and 1/16?  I'm not sure of your topology, but I would assume you want your 6509 to be the root for all those VLANs.  Make sure it has a higher priority then the access switches that are attached. 

0 Helpful

andy.bussiere
Level 1
Level 1
In response to niro

‎02-23-2011 10:31 AM

This is not the core switch. The core switch is connected to 5/8 and should be the root for all VLANS with the expection of 33, which is a different 6509 (Gi1/16), due to the isolation of the 33 VLAN. The only question is the 37 VLAN, which isn't used, but it's showing 1/14 as the root when it should be 5/8. As I said though, 37 isn't used.

Any other thoughts?

0 Helpful

niro
Level 1
Level 1
In response to andy.bussiere

‎02-23-2011 06:17 PM

Not sure.  Are any of the links overly saturated?  Has this EVER worked, or did it just suddenly start acting up now?  Any recent iOS upgrades or other changes changes?  I would try opening a TAC case.  I'm curious about the resolution to this, so if you do open up a TAC case please post back what they tell you.

0 Helpful

andy.bussiere
Level 1
Level 1
In response to niro

‎02-25-2011 09:33 AM

Thank you all for your help. The issue ended up being a bad supervisor. As soon as I switched to the back

up supervisor, everything started working correctly.

Thank you for the comments and suggestions.

0 Helpful
Customers Also Viewed These Support Documents
Review Cisco Networking for a $25 gift card