I upgraded the firmware on a few switches this past weekend:
On my 3750X core stack I went from 15.0(1) SE2 to 15.0(2) SE9
On my 2960S access switch I went from 12.2(55) SE5 to 15.0(2a) SE9
Since the upgrade I've had 2 devices that are continually flapping up and down. These devices are controllers for our doors. One device is connected to the core switch and another is connected to the access switch. Both are continually flapping like this:
Jun 30 15:54:06.538: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/23, changed state to up
Jun 30 15:54:38.610: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/23, changed state to down
Jun 30 15:54:40.619: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/23, changed state to up
Jun 30 15:54:43.335: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/23, changed state to down
Jun 30 15:54:45.343: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/23, changed state to up
Jun 30 15:54:58.293: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/23, changed state to down
Jun 30 15:55:00.349: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/23, changed state to up
Jun 30 15:55:03.018: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/23, changed state to down
Jun 30 15:55:05.026: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/23, changed state to up
I've tried swapping cables, setting speed resetting the interface, etc.
On one of the switches (3750X) we are also see some CPU warnings like this:
011979: Jun 30 15:36:18.505: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 86%/7%, Top 3 processes(Pid/Util): 40/32%, 180/20%, 90/6%
011980: Jun 30 15:36:28.496: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 55%/7%.
011981: Jun 30 15:41:08.490: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 70%/3%, Top 3 processes(Pid/Util): 180/21%, 5/13%, 90/7%
011982: Jun 30 15:41:13.490: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 42%/0%.
011983: Jun 30 15:56:38.454: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 70%/3%, Top 3 processes(Pid/Util): 5/20%, 180/19%, 90/6%
011984: Jun 30 15:56:43.453: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 48%/3%.
011985: Jun 30 15:58:38.449: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 74%/8%, Top 3 processes(Pid/Util): 180/21%, 5/16%, 90/8%
011986: Jun 30 15:58:43.449: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 46%/4%.
I think the CPU increase is related to the flapping and not causing the flapping because it is just this one particular device having this issue. To make things more confusing, we have these particular devices in other buildings throughout our network and they only seem to be affected in this particular VLAN.
We had this same issue last week with another access switch on the same VLAN with the same type of devices. Upgrading the firmware somehow fixed this issue but based on the fact that it is only these particular devices having any issues, I'm inclined to think the firmware on the switch isn't the problem and we just got lucky last time.
I'm really stumped at this point.
BC_RATROOM_SW#sh run int gig 1/0/23
Building configuration...
Current configuration : 372 bytes
!
interface GigabitEthernet1/0/23
description Amano Controller
switchport access vlan 10
switchport mode access
switchport voice vlan 40
speed 100
duplex full
srr-queue bandwidth share 1 30 35 5
priority-queue out
snmp trap mac-notification change added
snmp trap mac-notification change removed
mls qos trust cos
auto qos trust
spanning-tree portfast
end
BC_RATROOM_SW#sh int gig 1/0/23
GigabitEthernet1/0/23 is down, line protocol is down (notconnect)
Hardware is Gigabit Ethernet, address is 5897.1e2a.5c17 (bia 5897.1e2a.5c17)
Description: Amano Controller
MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:01, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 21428392
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 14000 bits/sec, 20 packets/sec
6582403 packets input, 504033981 bytes, 0 no buffer
Received 7645 broadcasts (0 multicasts)
0 runts, 0 giants, 0 throttles
22 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
113619565 packets output, 68050892791 bytes, 0 underruns
0 output errors, 0 collisions, 2 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
BC_RATROOM_SW#sh interfaces gig 1/0/23 counters errors
Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize OutDiscards
Gi1/0/23 0 0 0 22 0 21428392
Port Single-Col Multi-Col Late-Col Excess-Col Carri-Sen Runts Giants
Gi1/0/23 0 0 0 0 0 0 0
BC_RATROOM_SW#
