2960POE+SFP Routed port hangs often

[sheraz butt]

Hi All,

I am having a network core4507 and 2960 as access in any case power of core gose down many access switches shows solid amber light in sfp we need to restart the switch or we remove the sfp and connect to the other sfp port that works but solid light remain same on sfp port that was connected previously  till restart the 2960 switch.

any help

Regard

[Leo Laohoo · Accepted solution]

The next time the amber light on the SFP goes on again, console into the switch and issue the commands "sh interface status err" and "sh logs".  Post the two outputs.

[sheraz butt]

FYI

Vip405#sh interfaces status err-disabled 

Port      Name               Status       Reason               Err-disabled Vlans
Gi1/0/25                     err-disabled link-flap
Vip405#sh lo
Vip405#sh lo
Vip405#sh lo
Vip405#sh log
Vip405#sh log
Vip405#sh log
Vip405#sh log
Vip405#sh logs
Vip405#sh logs
Vip405#sh log?
logging  login  

Vip405#sh logg
Vip405#sh logging 
Syslog logging: enabled (0 messages dropped, 0 messages rate-limited, 0 flushes, 0 overruns, xml disabled, filtering disabled)

No Active Message Discriminator.

No Inactive Message Discriminator.

    Console logging: level debugging, 124 messages logged, xml disabled,
                     filtering disabled
    Monitor logging: level debugging, 0 messages logged, xml disabled,
                     filtering disabled
    Buffer logging:  level debugging, 124 messages logged, xml disabled,
                     filtering disabled
    Exception Logging: size (4096 bytes)
    Count and timestamp logging messages: disabled
    File logging: disabled
    Persistent logging: disabled

No active filter modules.

    Trap logging: level informational, 114 message lines logged
          
Log Buffer (4096 bytes):
e Gi1/0/1: Power Device detected: IEEE PD
*Mar  2 06:46:10.818: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/1: Power granted
*Mar  2 06:46:13.963: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to up
*Mar  2 06:46:14.965: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to up
*Mar  2 06:46:31.983: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/1: PD removed
*Mar  2 06:46:32.450: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to down
*Mar  2 06:46:33.451: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to down
*Mar  2 06:46:44.105: %ILPOWER-7-DETECT: Interface Gi1/0/1: Power Device detected: IEEE PD
*Mar  2 06:46:45.122: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/1: Power granted
*Mar  2 06:46:50.496: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to up
*Mar  2 06:46:51.497: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to up
*Mar  2 06:47:08.164: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to down
*Mar  2 06:47:08.280: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/1: PD removed
*Mar  2 06:47:09.166: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to down
*Mar  2 06:47:12.322: %ILPOWER-7-DETECT: Interface Gi1/0/1: Power Device detected: IEEE PD
*Mar  2 06:47:13.308: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/1: Power granted
*Mar  2 06:47:16.338: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to up
*Mar  2 06:47:17.339: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to up
*Mar  2 06:47:39.496: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/1: PD removed
*Mar  2 06:47:39.632: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to down
*Mar  2 06:47:40.639: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to down
*Mar  2 06:47:50.574: %ILPOWER-7-DETECT: Interface Gi1/0/1: Power Device detected: IEEE PD
*Mar  2 06:47:51.602: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/1: PD removed
*Mar  2 06:47:57.626: %ILPOWER-7-DETECT: Interface Gi1/0/1: Power Device detected: IEEE PD
*Mar  2 06:47:58.638: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/1: Power granted
*Mar  2 06:48:01.715: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to up
*Mar  2 06:48:02.717: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to up
*Mar  2 06:48:07.157: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to down
*Mar  2 06:57:22.247: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to down
*Mar  2 06:57:27.259: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to up
*Mar  2 06:57:41.798: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to down
*Mar  2 06:57:43.869: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/1: PD removed
*Mar  2 07:12:06.118: %ILPOWER-7-DETECT: Interface Gi1/0/1: Power Device detected: IEEE PD
*Mar  2 07:12:07.125: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/1: Power granted
*Mar  2 07:12:10.297: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to up
*Mar  2 07:12:11.298: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to up
*Mar  2 07:14:37.249: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/1: PD removed
*Mar  2 07:14:37.527: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to down
*Mar  2 07:14:38.529: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to down
*Mar  2 07:14:47.326: %ILPOWER-7-DETECT: Interface Gi1/0/1: Power Device detected: IEEE PD
*Mar  2 07:14:48.369: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/1: Power granted
*Mar  2 07:14:51.353: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to up
*Mar  2 07:14:52.354: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to up
*Mar  2 08:08:56.570: %USB_CONSOLE-6-MEDIA_USB: Console media-type is USB.

Regards

[Leo Laohoo]

Ok, link flap.  Please post the output to the commands "sh interface Gi1/0/25" & "sh controll e Gi1/0/25".

[sheraz butt]

Thanks for reply...

Vip405#sh interface Gi1/0/25
GigabitEthernet1/0/25 is down, line protocol is down (err-disabled) 
  Hardware is Gigabit Ethernet, address is f41f.c290.9299 (bia f41f.c290.9299)
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive not set
  Auto-duplex, Auto-speed, link type is auto, media type is Not Present
  input flow-control is off, output flow-control is unsupported 
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 1d05h, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     0 packets input, 0 bytes, 0 no buffer
     Received 0 broadcasts (0 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     6 packets output, 384 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out
Vip405# 
Vip405#
Vip405#
Vip405#sh controll e Gi1/0/25

     Transmit GigabitEthernet1/0/25           Receive
          384 Bytes                                0 Bytes                    
            0 Unicast frames                       0 Unicast frames           
            6 Multicast frames                     0 Multicast frames         
            0 Broadcast frames                     0 Broadcast frames         
            0 Too old frames                       0 Unicast bytes            
            0 Deferred frames                      0 Multicast bytes          
            0 MTU exceeded frames                  0 Broadcast bytes          
            0 1 collision frames                   0 Alignment errors         
            0 2 collision frames                   0 FCS errors               
            0 3 collision frames                   0 Oversize frames          
            0 4 collision frames                   0 Undersize frames         
            0 5 collision frames                   0 Collision fragments      
            0 6 collision frames       
            0 7 collision frames                   0 Minimum size frames      
            0 8 collision frames                   0 65 to 127 byte frames    
            0 9 collision frames                   0 128 to 255 byte frames   
            0 10 collision frames                  0 256 to 511 byte frames   
            0 11 collision frames                  0 512 to 1023 byte frames  
            0 12 collision frames                  0 1024 to 1518 byte frames 
            0 13 collision frames                  0 Overrun frames           
            0 14 collision frames                  0 Pause frames             
            0 15 collision frames      
            0 Excessive collisions                 0 Symbol error frames      
            0 Late collisions                      0 Invalid frames, too large
            0 VLAN discard frames                  0 Valid frames, too large  
            0 Excess defer frames                  0 Invalid frames, too small
            6 64 byte frames                       0 Valid frames, too small  
            0 127 byte frames          
            0 255 byte frames                      0 Too old frames           
            0 511 byte frames                      0 Valid oversize frames    
            0 1023 byte frames                     0 System FCS error frames  
            0 1518 byte frames                     0 RxPortFifoFull drop frame
            0 Too large frames         
            0 Good (1 coll) frames     
            0 Good (>1 coll) frames 

[Leo Laohoo · Accepted solution]

Do this: 

config t

interface Gi 1/0/25

shut

no shut

end

Wait for a day and then issue the commands "sh interface Gi1/0/25" & "sh controll e Gi1/0/25".

[sheraz butt]

During restart port hang again and same error link-flap

Vip405#sh interfaces gigabitEthernet 1/0/26
GigabitEthernet1/0/26 is up, line protocol is up (connected) 
  Hardware is Gigabit Ethernet, address is f41f.c290.929a (bia f41f.c290.929a)
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive not set
  Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseLX SFP
  input flow-control is off, output flow-control is unsupported 
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:01, output 00:00:06, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 2243000 bits/sec, 233 packets/sec
  5 minute output rate 2000 bits/sec, 2 packets/sec
     4988653 packets input, 5177543080 bytes, 0 no buffer
     Received 4875107 broadcasts (4351325 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 4351325 multicast, 0 pause input
     0 input packets with dribble condition detected
     48157 packets output, 3923115 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out
Vip405#sh controll e 1/0/26
                     ^
% Invalid input detected at '^' marker.

Vip405#sh controll e 1/0/26
Vip405#sh controll e 1/0/26 ?
% Unrecognized command
Vip405#sh controll e gig 1/0/26 

     Transmit GigabitEthernet1/0/26           Receive
      3930320 Bytes                        882821334 Bytes                    
        44462 Unicast frames                  113670 Unicast frames           
         3672 Multicast frames               4351829 Multicast frames         
          101 Broadcast frames                526175 Broadcast frames         
            0 Too old frames                 9636831 Unicast bytes            
            0 Deferred frames              796664819 Multicast bytes          
            0 MTU exceeded frames           57136884 Broadcast bytes          
            0 1 collision frames                   0 Alignment errors         
            0 2 collision frames                   0 FCS errors               
            0 3 collision frames                   0 Oversize frames          
            0 4 collision frames                   0 Undersize frames         
            0 5 collision frames                   0 Collision fragments      
            0 6 collision frames       
            0 7 collision frames               78157 Minimum size frames      
            0 8 collision frames             1091958 65 to 127 byte frames    
            0 9 collision frames               57286 128 to 255 byte frames   
            0 10 collision frames              46790 256 to 511 byte frames   
            0 11 collision frames              23897 512 to 1023 byte frames  
            0 12 collision frames            3693524 1024 to 1518 byte frames 
            0 13 collision frames                  0 Overrun frames           
            0 14 collision frames                  0 Pause frames             
            0 15 collision frames      
            0 Excessive collisions                 0 Symbol error frames      
            0 Late collisions                      0 Invalid frames, too large
            0 VLAN discard frames                 62 Valid frames, too large  
            0 Excess defer frames                  0 Invalid frames, too small
         2360 64 byte frames                       0 Valid frames, too small  
        43467 127 byte frames          
         1044 255 byte frames                      0 Too old frames           
         1295 511 byte frames                      0 Valid oversize frames    
           36 1023 byte frames                     0 System FCS error frames  
           33 1518 byte frames                     0 RxPortFifoFull drop frame
            0 Too large frames         
            0 Good (1 coll) frames     
            0 Good (>1 coll) frames    

Vip405#sh interfaces gig1/0/26 transceiver detail
ITU Channel not available (Wavelength not available),
Transceiver is externally calibrated.
mA: milliamperes, dBm: decibels (milliwatts), NA or N/A: not applicable.
++ : high alarm, +  : high warning, -  : low warning, -- : low alarm.
A2D readouts (if they differ), are reported in parentheses.
The threshold values are calibrated.

                              High Alarm  High Warn  Low Warn   Low Alarm
          Temperature         Threshold   Threshold  Threshold  Threshold
Port       (Celsius)          (Celsius)   (Celsius)  (Celsius)  (Celsius)
--------- ------------------  ----------  ---------  ---------  ---------
Gi1/0/26    42.2                80.0        70.0         3.0        0.0

                              High Alarm  High Warn  Low Warn   Low Alarm
           Voltage            Threshold   Threshold  Threshold  Threshold
Port       (Volts)            (Volts)     (Volts)    (Volts)    (Volts)
---------  ---------------    ----------  ---------  ---------  ---------
Gi1/0/26   3.28                  3.50        3.45        2.90       2.80

           Optical            High Alarm  High Warn  Low Warn   Low Alarm
           Transmit Power     Threshold   Threshold  Threshold  Threshold
Port       (dBm)              (dBm)       (dBm)      (dBm)      (dBm)
---------  -----------------  ----------  ---------  ---------  ---------
Gi1/0/26    -4.5                 3.0         2.0        -8.0       -9.0

           Optical            High Alarm  High Warn  Low Warn   Low Alarm
           Receive Power      Threshold   Threshold  Threshold  Threshold
Port       (dBm)              (dBm)       (dBm)      (dBm)      (dBm)
-------    -----------------  ----------  ---------  ---------  ---------
Gi1/0/26    -8.9                -2.0        -3.0       -23.0      -25.1

Vip405#

[Leo Laohoo]

Ok, no line errors.  If the port is still down, then issue the command "sh interface status error".

[sheraz butt]

Hi ,

Same error sir

Port      Name               Status       Reason               Err-disabled Vlans
Gi1/0/26                     err-disabled link-flap
Vip405#

logging=

Mar  3 03:13:15.905: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/25, changed state to down
*Mar  3 03:18:12.112: %PM-4-ERR_DISABLE: link-flap error detected on Gi1/0/25, putting Gi1/0/25 in err-disable state

Regards

[Leo Laohoo]

Link flap means it's a line fault or layer 1 issue. 

Replace THREE (3) times the SFP, the patch cable and observe.

[sheraz butt]

Hi,

Thanks you very much for the kind help can i do with some link temp and till then i can recover other area.

OPT-1

errdisable recovery cause link-flap
errdisable recovery interval 30

Or

OPT-2

errdisable recovery cause all

Please advice  

[sheraz butt]

HI ,

  can you please give me any sloution because it is a production network.

Regards