01-15-2013 02:32 AM - edited 03-07-2019 11:05 AM
Working on this fault on behalf of a customer and looking for help. (E-mail from customer below)
We have a 2960 access switch that has two uplinks to the core; one uplink goes to core switch 1 (6509), and the second uplink goes to core switch 2 (6509). This was working fine, for a few years, until December 24, when one of the two uplinks failed. I have replaced all physical layer components, but this has not resolved. I have noticed on a few occassions that when the failed uplink does come up the remaining working one goes down. There are no errdisable msgs in the logs but there are some STP msgs (below).
I am sure this is a config issue, but cannot see what is causing one uplink port to constantly shutdown. Also confused as to why port moves to disabled from forwarding and the interface changes to updown. Port 23 is the faulty port.
Jan 3 15:14:43 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/23, changed state to up
Jan 3 15:14:43 GMT: %SPANTREE-6-PORT_STATE: Port Gi0/23 instance 2 moving from blocking to forwarding
Jan 3 15:14:44 GMT: %SPANTREE-6-PORT_STATE: Port Gi0/23 instance 301 moving from blocking to forwarding
Jan 3 15:20:08 GMT: %SPANTREE-6-PORT_STATE: Port Gi0/23 instance 2 moving from forwarding to disabled
Jan 3 15:20:08 GMT: %SPANTREE-6-PORT_STATE: Port Gi0/23 instance 301 moving from forwarding to disabled
Jan 3 15:20:09 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/23, changed state to down
01-15-2013 09:47 AM
It looks like you have spanning-tree logging enabled on this switch. The logs seem normal, considering that the line protocol is flapping.
Once the port comes up, the port is moving to a forwarding state. As it is going down, it moves back to a disabled state. The bigger question is why the line protocol is flapping. Also, just to clarify, is only the line protocol flapping, or does the port go to a down/down state during the problem?
Additionally, can you post the running-configuration and the show interface for both those ports?
- Aninda
01-15-2013 10:33 AM
Hi Aninda,
thanks for the reply. Please see show run for the devices attached.
Johnny
01-15-2013 10:48 AM
Hi,
For me it seems like a loop on your network, there is a command that might help you to see if this issue is related to a loop:
show spanning-tree detail | inc is executing|from|topology changes
This command will show you if there are topology changes on your network and where they are coming from; topology changes are firm indication of a loop, please post the result of that command.
Regards,
Rolando Ramirez
01-15-2013 11:25 AM
Hi Rolando,
I have included another text doc with details the customer had sent previously of show spanning-tree.
I agree this looks like a spanning tree issue on the surface, but the customer is adamant that the 2 links were always up at the same time in the past and no changes have been made to the configurations. The log files are also not fully clear on the exact issue. You would imagine the port to go into a blocking state rather than a disabled state and showing:
%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/23, changed state to down
Thanks for your help so far.
01-15-2013 04:01 PM
Thanks for the response, please let your customer know that even though the links had been up during a long period of time it doesn't mean that cannot fail all of a sudden so we need to be open to that option, now can you please send me the result of the above command mentioned:
show spanning-tree detail | inc is executing|from|topology changes
Also, you said that you replaced the physical part but have you tried using a different port on both sides?
Have you checked the CPU when this issue happened?
Regards,
Rolando Ramirez
01-16-2013 12:30 AM
Will try and get the output from that command today Rolando.
I belive the customer has changed the ports both side and all cabling as well as using a different fibre pair in the internal connection. They have also tried a different switch at the access side.
01-16-2013 03:33 AM
You would imagine the port to go into a blocking state rather than a disabled state and showing:%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/23, changed state to down
That is correct. I would not imagine a port going into a disabled state unless you have layer 1/2 problems across that link. The port going into disabled is clearly because your line protocol is flapping and/or the port itself is flapping on the physical layer as well.
I would suggest that you confirm if this connection was moved to a different ports on both sides or not as part of the troubleshooting.
I haven't had a chance to go through the configurations attached, but if you haven't already, could you enable logging event link-status on the core switch ports that go to the 2960?
01-16-2013 03:29 AM
topology changes are firm indication of a loop
I do not agree with this. Topology changes can be quite common in networks and they are not necessarily indicative of a problem. However, they should naturally be tracked down and controlled.
If we are looking for a loop, the best way would be to check if mac addresses are flapping between ports. This can be done by enabling the following command in global configuration mode:
mac address-table notification mac-move
The syntax might be slightly different on a 2960, I have not verified this in a lab setup.
01-16-2013 04:04 AM
I have sent the above commands on to the custome to run and will post the results once they are supplied.
Aninda, the customer has said that they have tried different ports at both sides.
01-18-2013 08:06 AM
Still awaiting cutomer reply. Thanks for help so far and I would be grateful if you could continue to check this post for updates.
many thanks
01-21-2013 07:19 AM
01-22-2013 04:13 AM
Hi Johnny,
Does the customer have a patch panel between access switch gig0/24 and gig2/23 core 2? There seems to be physical layer issues between this link:
Jan 21 14:14:05 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/24, changed state to down
Jan 21 14:14:06 GMT: %LINK-3-UPDOWN: Interface GigabitEthernet0/24, changed state to down
The link now stays down for over two minutes and then comes up:
Jan 21 14:16:13 GMT: %LINK-3-UPDOWN: Interface GigabitEthernet0/24, changed state to up
Jan 21 14:16:14 GMT: %SPANTREE-6-PORT_STATE: Port Gi0/24 instance 2 moving from disabled to blocking
Jan 21 14:16:14 GMT: %SPANTREE-6-PORT_STATE: Port Gi0/24 instance 301 moving from disabled to blocking
Jan 21 14:16:15 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/24, changed state to up
The TCNs also will probably tell us the same thing. If you look at the spanning-tree output you attached, you can see TCNs received recently only for VLAN 2 and VLAN 301 which are the VLANs allowed across gig0/24.
From Core 1:
VLAN0002 is executing the rstp compatible Spanning Tree protocol
Number of topology changes 22 last change occurred 00:08:18 ago
from Port-channel255
VLAN0301 is executing the rstp compatible Spanning Tree protocol
Number of topology changes 56 last change occurred 00:08:18 ago
from Port-channel255
TCNs were received on po255. This port-channel takes you to Core 2. With the same commands issued on Core 2, you see the following:
VLAN0002 is executing the rstp compatible Spanning Tree protocol
Number of topology changes 27 last change occurred 00:10:04 ago
from GigabitEthernet2/23
VLAN0301 is executing the rstp compatible Spanning Tree protocol
Number of topology changes 54 last change occurred 00:10:04 ago
from GigabitEthernet2/23
Core 2 says it received TCNs for these VLANs on gig2/23. This port leads you to the access switch, which in turn shows the following:
VLAN0002 is executing the rstp compatible Spanning Tree protocol
Number of topology changes 3 last change occurred 00:11:14 ago
from GigabitEthernet0/24
VLAN0301 is executing the rstp compatible Spanning Tree protocol
Number of topology changes 2 last change occurred 00:11:14 ago
from GigabitEthernet0/24
The root cause of these TCNs is because gig0/24 is bouncing. This still looks to be a physical layer problem.
Can I get the output of show interface gig0/24 from the access switch and show interface gig2/23 from Core 2 please?
Regards,
Aninda
01-22-2013 04:17 AM
Also, I see that initially the concern was with gig0/23 on the access switch. But now the problem has moved over to gig0/24. Any changes that were made? Were the two ports, gig0/23 and gig0/24 interchanged? Or is gig0/23 still having the same problem as well?
01-22-2013 04:35 AM
Aninda,
the reason for the above is because I had told the customer to do a shut command on port 24 and a no shut on port 23 to see if port 23 would come back up. I told him to leave for a minute or two and if port 23 did not come up then to re enable port 24. The customer confirmed that this was carried out and port 23 did not come up.
There switch has been replaced, the sfp module and port have been swapped at the core and the fibre pair linking the switch and core have been changed also.
In earier logs you can see port 24 goes to disabled, then port 23 comes up, then about 5 secs later port 23 goes to back to disabled and port 24 comes up.
It has me baffled.
Jan 3 15:12:59 GMT: %SPANTREE-6-PORT_STATE: Port Gi0/24 instance 2 moving from forwarding to disabled
Jan 3 15:12:59 GMT: %SPANTREE-6-PORT_STATE: Port Gi0/24 instance 301 moving from forwarding to disabled
Jan 3 15:13:00 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/24, changed state to down
Jan 3 15:13:01 GMT: %LINK-3-UPDOWN: Interface GigabitEthernet0/24, changed state to down
Jan 3 15:14:41 GMT: %LINK-3-UPDOWN: Interface GigabitEthernet0/23, changed state to up
Jan 3 15:14:42 GMT: %SPANTREE-6-PORT_STATE: Port Gi0/23 instance 2 moving from disabled to blocking
Jan 3 15:14:42 GMT: %SPANTREE-6-PORT_STATE: Port Gi0/23 instance 301 moving from disabled to blocking
Jan 3 15:14:43 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/23, changed state to up
Jan 3 15:14:43 GMT: %SPANTREE-6-PORT_STATE: Port Gi0/23 instance 2 moving from blocking to forwarding
Jan 3 15:14:44 GMT: %SPANTREE-6-PORT_STATE: Port Gi0/23 instance 301 moving from blocking to forwarding
Jan 3 15:20:08 GMT: %SPANTREE-6-PORT_STATE: Port Gi0/23 instance 2 moving from forwarding to disabled
Jan 3 15:20:08 GMT: %SPANTREE-6-PORT_STATE: Port Gi0/23 instance 301 moving from forwarding to disabled
Jan 3 15:20:09 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/23, changed state to down
Jan 3 15:20:10 GMT: %LINK-3-UPDOWN: Interface GigabitEthernet0/23, changed state to down
Jan 3 15:24:57 GMT: %LINK-3-UPDOWN: Interface GigabitEthernet0/24, changed state to up
Jan 3 15:24:58 GMT: %SPANTREE-6-PORT_STATE: Port Gi0/24 instance 2 moving from disabled to blocking
Jan 3 15:24:58 GMT: %SPANTREE-6-PORT_STATE: Port Gi0/24 instance 301 moving from disabled to blocking
Jan 3 15:24:59 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/24, changed state to up
Jan 3 15:25:00 GMT: %SPANTREE-6-PORT_STATE: Port Gi0/24 instance 2 moving from blocking to forwarding
Jan 3 15:25:00 GMT: %SPANTREE-6-PORT_STATE: Port Gi0/24 instance 301 moving from blocking to forwarding
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide