Solved: 2960X random port flapping

rudimocnik · ‎06-03-2019

Hi

I have 2x WS-C2960X-48LPS-L running 15.2(2)E7 IOS. I've noticed random interface flapping in the logs and I am not sure what else to check. I followed articles in here with similar issues with no luck.

This is just a short snippet of the recent logs:

105688: Jun 2 22:30:26.782: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/13, changed state to up
105689: Jun 2 22:30:27.782: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/13, changed state to up
105690: Jun 2 22:44:31.001: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/40, changed state to down
105691: Jun 2 22:44:32.001: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/40, changed state to down
105692: Jun 2 22:44:35.517: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/40, changed state to up
105693: Jun 2 22:44:36.520: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/40, changed state to up
105694: Jun 2 22:44:49.435: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/22, changed state to down
105695: Jun 2 22:44:50.435: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/22, changed state to down
105696: Jun 2 22:44:53.916: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/22, changed state to up
105697: Jun 2 22:44:54.919: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/22, changed state to up
105698: Jun 2 22:58:47.282: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/13, changed state to down
105699: Jun 2 22:58:48.285: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/13, changed state to down
105700: Jun 2 22:58:49.446: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/12, changed state to down

2960X-3# sh interfaces gi1/0/43 controller 
GigabitEthernet1/0/43 is up, line protocol is up (connected) 
  Hardware is Gigabit Ethernet, address is cc70.edfb.44ab (bia cc70.edfb.44ab)
  Description: AP tajnistvo Butina
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
  input flow-control is off, output flow-control is unsupported 
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:01, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 3000 bits/sec, 3 packets/sec
  5 minute output rate 2000 bits/sec, 3 packets/sec
     14034288 packets input, 3295145189 bytes, 0 no buffer
     Received 648360 broadcasts (569772 multicasts)
     0 runts, 0 giants, 0 throttles 
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 569772 multicast, 0 pause input
     0 input packets with dribble condition detected
     29375439 packets output, 12031813200 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

     Transmit GigabitEthernet1/0/43           Receive
   3441879032 Bytes                       3295145189 Bytes                    
     12912032 Unicast frames                13385928 Unicast frames           
     12835928 Multicast frames                569772 Multicast frames         
      3627485 Broadcast frames                 78588 Broadcast frames         
            0 Too old frames              3099142246 Unicast bytes            
            0 Deferred frames              150910311 Multicast bytes          
            0 MTU exceeded frames           20619779 Broadcast bytes          
            0 1 collision frames                   0 Alignment errors         
            0 2 collision frames                   0 FCS errors               
            0 3 collision frames                   0 Oversize frames          
            0 4 collision frames                   0 Undersize frames         
            0 5 collision frames                   0 Collision fragments      
            0 6 collision frames       
            0 7 collision frames             1131137 Minimum size frames      
            0 8 collision frames             9311153 65 to 127 byte frames    
            0 9 collision frames             1048235 128 to 255 byte frames   
            0 10 collision frames             893648 256 to 511 byte frames   
            0 11 collision frames             726403 512 to 1023 byte frames  
            0 12 collision frames             920030 1024 to 1518 byte frames 
            0 13 collision frames                  0 Overrun frames           
            0 14 collision frames                  0 Pause frames             
            0 15 collision frames      
            0 Excessive collisions                 0 Symbol error frames      
            0 Late collisions                      0 Invalid frames, too large
            0 VLAN discard frames               3682 Valid frames, too large  
            0 Excess defer frames                  0 Invalid frames, too small
      6555596 64 byte frames                       0 Valid frames, too small  
     12924200 127 byte frames          
      1360740 255 byte frames                      0 Too old frames           
       780107 511 byte frames                      0 Valid oversize frames    
       242266 1023 byte frames                     0 System FCS error frames  
      7508258 1518 byte frames                     0 RxPortFifoFull drop frame
         4278 Too large frames         
            0 Good (1 coll) frames     
            0 Good (>1 coll) frames

2960X-3#show cable-diagnostics tdr int gi1/0/43
TDR test last run on: June 03 09:53:56

Interface Speed Local pair Pair length        Remote pair Pair status
--------- ----- ---------- ------------------ ----------- --------------------
Gi1/0/43  1000M Pair A     80   +/- 10 meters Pair B      Normal              
                Pair B     80   +/- 10 meters Pair A      Normal              
                Pair C     80   +/- 10 meters Pair D      Normal              
                Pair D     80   +/- 10 meters Pair C      Normal

These 2 switches are used to connects end user devices, polycom phones, APs etc. I see ports flapping on both switches. Flapping ports connect to APs, trunk port to other 2960X, as well as other devices. So no pattern there. I first noticed this on APs that would just go down after 40 days or so. Not all of them tho. Last time 1/2 went down on switch1 and 2/2 went down on switch2. Looking at the logs I noticed flaping on other ports as well.

Any ideas?

Thanks

Mark Malone · ‎06-05-2019

So that would suggest you had a layer 2 issue at the same time , you seen all the flaps , either someone was working on the switches pulled something caused re-convergence or a link went down and caused it, STP is automatic so it trys to fix it and prevent loops when this occurs as a protection method

Tracking it now may be difficult after 3 days , the whole lot went into spin though and converged at this time ,nothing was probably usable as L2 was trying to recalculate the convergence but it would be quick on a small network, while this is happening network can become temporarily unavailable

View solution in original post

Leo Laohoo · ‎06-03-2019

When a computer log off, log on or goes to sleep, it will generate two down/up log entries.

paul driver · ‎06-03-2019

Hello

Is this just the case of the end host(s) coming and going and the port link status is being reported?

try appending the following to the access ports.

int x/x
no logging event link-status
spanning-tree portfast

Please rate and mark as an accepted solution if you have found any of the information provided useful.
This then could assist others on these forums to find a valuable answer and broadens the community’s global network.

Kind Regards
Paul

rudimocnik · ‎06-03-2019

Oh I see. Great tip both of you. I configured the "no logging event link-status" only the access ports. That should help keep useful logs buffered.

However, the thing is I also see the trunk between the switches flapped once (I only have 50000 buffer locally) and what bothers me also is the access points dropping. Note APs are powered via this switch but the power doesn't seem to be an issue since I cannot find any logs related to power issues and looking at the output below there is plenty of power left on both switches.

2960X-3#sh power inline 

Module   Available     Used     Remaining
          (Watts)     (Watts)    (Watts) 
------   ---------   --------   ---------
1           370.0      200.4       169.6


2960X-1#sh power inline 

Module   Available     Used     Remaining
          (Watts)     (Watts)    (Watts) 
------   ---------   --------   ---------
1           370.0      168.6       201.4

omz · ‎06-03-2019

Hi

Have you done bug search?

Regards

rudimocnik · ‎06-03-2019

Yeah there is nothing under the version I am using. In general there is one similar thing that affect the port when connected to 2rd party hardware CSCvd05873, there is also uplink and GLC-T issue but I am not using those. Nothing I could find.

Mark Malone · ‎06-03-2019

Hi
another thing check there are no underlying layer 2 stp issues constantly occurring could also explain ints dropping in and out, this should show you how may changes and when
show spanning-tree detail | inc ieee|occurr|from|is exec

rudimocnik · ‎06-04-2019

I've checked the logs again and I see no more interface flapping. I configured "no logging event link-status" on all access ports yesterday. However I am still wondering what caused the interfaces that APs are connected to and the trunk between the switches go down. This was the second time the APs went down after being up for 30+ days.

I will keep monitoring and see what happens. At least I will be able to buffer more logs now since I don't have redundant link status messages from the access ports.

rudimocnik · ‎06-04-2019

I've checked the logs again and I see no more interafce flapping. I configured "no logging event link-status" on all access ports yesterday. However I am still wondering what caused the interfaces that APs are connected to and the trunmk between the switches go down. This was the second time the APs went down after being up for 30+ days.

I will keep monitoring and see what happens. At least I will be able to buffer more logs now since I don't have redundant link status messages from the access ports.

Mark Malone · ‎06-04-2019

did you run the STP command to see if there was layer 2 issues , tied in the logs times ?

rudimocnik · ‎06-04-2019

2960X-1#show spanning-tree detail | inc ieee|occurr|from|is exec
 VLAN0005 is executing the rstp compatible Spanning Tree protocol
  Number of topology changes 14 last change occurred 3d01h ago
          from GigabitEthernet1/0/47
 VLAN0006 is executing the rstp compatible Spanning Tree protocol
  Number of topology changes 8 last change occurred 3d01h ago
          from GigabitEthernet1/0/46
 VLAN0007 is executing the rstp compatible Spanning Tree protocol
  Number of topology changes 8 last change occurred 3d01h ago
          from GigabitEthernet1/0/46
 VLAN0100 is executing the rstp compatible Spanning Tree protocol
  Number of topology changes 8 last change occurred 3d01h ago
          from GigabitEthernet1/0/48

2960X-3#show spanning-tree detail | inc ieee|occurr|from|is exec
 VLAN0005 is executing the rstp compatible Spanning Tree protocol
  Number of topology changes 5 last change occurred 3d01h ago
          from GigabitEthernet1/0/48
 VLAN0006 is executing the rstp compatible Spanning Tree protocol
  Number of topology changes 38 last change occurred 3d01h ago
          from GigabitEthernet1/0/43
 VLAN0007 is executing the rstp compatible Spanning Tree protocol
  Number of topology changes 40 last change occurred 3d01h ago
          from GigabitEthernet1/0/43
 VLAN0100 is executing the rstp compatible Spanning Tree protocol
  Number of topology changes 4 last change occurred 3d01h ago
          from GigabitEthernet1/0/48

This is the output of the command you gave me.

On 2960X-3: Gi1/0/48 is the trunk connected to the 2960X-1 Gi1/0/48. Gi1/0/43 is a trunk connected to AP.

On 2960X-1: Gi1/0/47 connects straight to the firewall (gateway). Gi1/0/46 is a trunk to AP.

The flapping occured 3 days ago. But now everything is stable and I haven't done any configuration since then. It just happened and fixed itself. I am curious to know what really caused this. The switches did't not reboot and the cables tests via cli show status Normal on all pairs.

Mark Malone · ‎06-05-2019

So that would suggest you had a layer 2 issue at the same time , you seen all the flaps , either someone was working on the switches pulled something caused re-convergence or a link went down and caused it, STP is automatic so it trys to fix it and prevent loops when this occurs as a protection method

Tracking it now may be difficult after 3 days , the whole lot went into spin though and converged at this time ,nothing was probably usable as L2 was trying to recalculate the convergence but it would be quick on a small network, while this is happening network can become temporarily unavailable

Georg Pauwen · ‎06-05-2019

Hello,

in addition to Mark's remarks regarding STP, you might want to check which of the switches is the root for your VLANs, e.g.:

Switch#sh spanning-tree vlan 1

VLAN0001
Spanning tree enabled protocol rstp
Root ID Priority 32769
Address 5209.314b.8100
This bridge is the root

You might want to set the switches to be the primary and secondary root manually:

spanning-tree vlan 1 root primary/secondary