Re: %SPANTREE-6-PORT_STATE: Port Gi0/23 instance 2 moving from f

johnnynextiraone · ‎01-15-2013

Working on this fault on behalf of a customer and looking for help. (E-mail from customer below)

We have a 2960 access switch that has two uplinks to the core; one uplink goes to core switch 1 (6509), and the second uplink goes to core switch 2 (6509). This was working fine, for a few years, until December 24, when one of the two uplinks failed. I have replaced all physical layer components, but this has not resolved. I have noticed on a few occassions that when the failed uplink does come up the remaining working one goes down. There are no errdisable msgs in the logs but there are some STP msgs (below).

I am sure this is a config issue, but cannot see what is causing one uplink port to constantly shutdown. Also confused as to why port moves to disabled from forwarding and the interface changes to updown. Port 23 is the faulty port.

Jan 3 15:14:43 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/23, changed state to up

Jan 3 15:14:43 GMT: %SPANTREE-6-PORT_STATE: Port Gi0/23 instance 2 moving from blocking to forwarding

Jan 3 15:14:44 GMT: %SPANTREE-6-PORT_STATE: Port Gi0/23 instance 301 moving from blocking to forwarding

Jan 3 15:20:08 GMT: %SPANTREE-6-PORT_STATE: Port Gi0/23 instance 2 moving from forwarding to disabled

Jan 3 15:20:08 GMT: %SPANTREE-6-PORT_STATE: Port Gi0/23 instance 301 moving from forwarding to disabled

Jan 3 15:20:09 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/23, changed state to down

Aninda Chatterjee · ‎01-15-2013

It looks like you have spanning-tree logging enabled on this switch. The logs seem normal, considering that the line protocol is flapping.

Once the port comes up, the port is moving to a forwarding state. As it is going down, it moves back to a disabled state. The bigger question is why the line protocol is flapping. Also, just to clarify, is only the line protocol flapping, or does the port go to a down/down state during the problem?

Additionally, can you post the running-configuration and the show interface for both those ports?

- Aninda

johnnynextiraone · ‎01-15-2013

Hi Aninda,

thanks for the reply. Please see show run for the devices attached.

Johnny

rolramir · ‎01-15-2013

Hi,

For me it seems like a loop on your network, there is a command that might help you to see if this issue is related to a loop:

show spanning-tree detail | inc is executing|from|topology changes

This command will show you if there are topology changes on your network and where they are coming from; topology changes are firm indication of a loop, please post the result of that command.

Regards,

Rolando Ramirez

johnnynextiraone · ‎01-15-2013

Hi Rolando,

I have included another text doc with details the customer had sent previously of show spanning-tree.

I agree this looks like a spanning tree issue on the surface, but the customer is adamant that the 2 links were always up at the same time in the past and no changes have been made to the configurations. The log files are also not fully clear on the exact issue. You would imagine the port to go into a blocking state rather than a disabled state and showing:

%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/23, changed state to down

Thanks for your help so far.

rolramir · ‎01-15-2013

Thanks for the response, please let your customer know that even though the links had been up during a long period of time it doesn't mean that cannot fail all of a sudden so we need to be open to that option, now can you please send me the result of the above command mentioned:

show spanning-tree detail | inc is executing|from|topology changes

Also, you said that you replaced the physical part but have you tried using a different port on both sides?

Have you checked the CPU when this issue happened?

Regards,

Rolando Ramirez

johnnynextiraone · ‎01-16-2013

Will try and get the output from that command today Rolando.

I belive the customer has changed the ports both side and all cabling as well as using a different fibre pair in the internal connection. They have also tried a different switch at the access side.

Aninda Chatterjee · ‎01-16-2013

You would imagine the port to go into a blocking state rather than a disabled state and showing:

%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/23, changed state to down

That is correct. I would not imagine a port going into a disabled state unless you have layer 1/2 problems across that link. The port going into disabled is clearly because your line protocol is flapping and/or the port itself is flapping on the physical layer as well.

I would suggest that you confirm if this connection was moved to a different ports on both sides or not as part of the troubleshooting.

I haven't had a chance to go through the configurations attached, but if you haven't already, could you enable logging event link-status on the core switch ports that go to the 2960?

Aninda Chatterjee · ‎01-16-2013

topology changes are firm indication of a loop

I do not agree with this. Topology changes can be quite common in networks and they are not necessarily indicative of a problem. However, they should naturally be tracked down and controlled.

If we are looking for a loop, the best way would be to check if mac addresses are flapping between ports. This can be done by enabling the following command in global configuration mode:

mac address-table notification mac-move

The syntax might be slightly different on a 2960, I have not verified this in a lab setup.

johnnynextiraone · ‎01-16-2013

I have sent the above commands on to the custome to run and will post the results once they are supplied.

Aninda, the customer has said that they have tried different ports at both sides.

johnnynextiraone · ‎01-18-2013

Still awaiting cutomer reply. Thanks for help so far and I would be grateful if you could continue to check this post for updates.

many thanks

johnnynextiraone · ‎01-21-2013

Please see latest update from customer detailing output of specified commands

Thanks

Aninda Chatterjee · ‎01-22-2013

Hi Johnny,

Does the customer have a patch panel between access switch gig0/24 and gig2/23 core 2? There seems to be physical layer issues between this link:

Jan 21 14:14:05 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/24, changed state to down

Jan 21 14:14:06 GMT: %LINK-3-UPDOWN: Interface GigabitEthernet0/24, changed state to down

The link now stays down for over two minutes and then comes up:

Jan 21 14:16:13 GMT: %LINK-3-UPDOWN: Interface GigabitEthernet0/24, changed state to up

Jan 21 14:16:14 GMT: %SPANTREE-6-PORT_STATE: Port Gi0/24 instance 2 moving from disabled to blocking

Jan 21 14:16:14 GMT: %SPANTREE-6-PORT_STATE: Port Gi0/24 instance 301 moving from disabled to blocking

Jan 21 14:16:15 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/24, changed state to up

The TCNs also will probably tell us the same thing. If you look at the spanning-tree output you attached, you can see TCNs received recently only for VLAN 2 and VLAN 301 which are the VLANs allowed across gig0/24.

From Core 1:

VLAN0002 is executing the rstp compatible Spanning Tree protocol

Number of topology changes 22 last change occurred 00:08:18 ago

from Port-channel255

VLAN0301 is executing the rstp compatible Spanning Tree protocol

Number of topology changes 56 last change occurred 00:08:18 ago

from Port-channel255

TCNs were received on po255. This port-channel takes you to Core 2. With the same commands issued on Core 2, you see the following:

VLAN0002 is executing the rstp compatible Spanning Tree protocol

Number of topology changes 27 last change occurred 00:10:04 ago

from GigabitEthernet2/23

VLAN0301 is executing the rstp compatible Spanning Tree protocol

Number of topology changes 54 last change occurred 00:10:04 ago

from GigabitEthernet2/23

Core 2 says it received TCNs for these VLANs on gig2/23. This port leads you to the access switch, which in turn shows the following:

VLAN0002 is executing the rstp compatible Spanning Tree protocol

Number of topology changes 3 last change occurred 00:11:14 ago

from GigabitEthernet0/24

VLAN0301 is executing the rstp compatible Spanning Tree protocol

Number of topology changes 2 last change occurred 00:11:14 ago

from GigabitEthernet0/24

The root cause of these TCNs is because gig0/24 is bouncing. This still looks to be a physical layer problem.

Can I get the output of show interface gig0/24 from the access switch and show interface gig2/23 from Core 2 please?

Regards,

Aninda

Aninda Chatterjee · ‎01-22-2013

Also, I see that initially the concern was with gig0/23 on the access switch. But now the problem has moved over to gig0/24. Any changes that were made? Were the two ports, gig0/23 and gig0/24 interchanged? Or is gig0/23 still having the same problem as well?

johnnynextiraone · ‎01-22-2013

Aninda,

the reason for the above is because I had told the customer to do a shut command on port 24 and a no shut on port 23 to see if port 23 would come back up. I told him to leave for a minute or two and if port 23 did not come up then to re enable port 24. The customer confirmed that this was carried out and port 23 did not come up.

There switch has been replaced, the sfp module and port have been swapped at the core and the fibre pair linking the switch and core have been changed also.

In earier logs you can see port 24 goes to disabled, then port 23 comes up, then about 5 secs later port 23 goes to back to disabled and port 24 comes up.

It has me baffled.

Jan 3 15:12:59 GMT: %SPANTREE-6-PORT_STATE: Port Gi0/24 instance 2 moving from forwarding to disabled

Jan 3 15:12:59 GMT: %SPANTREE-6-PORT_STATE: Port Gi0/24 instance 301 moving from forwarding to disabled

Jan 3 15:13:00 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/24, changed state to down

Jan 3 15:13:01 GMT: %LINK-3-UPDOWN: Interface GigabitEthernet0/24, changed state to down

Jan 3 15:14:41 GMT: %LINK-3-UPDOWN: Interface GigabitEthernet0/23, changed state to up

Jan 3 15:14:42 GMT: %SPANTREE-6-PORT_STATE: Port Gi0/23 instance 2 moving from disabled to blocking

Jan 3 15:14:42 GMT: %SPANTREE-6-PORT_STATE: Port Gi0/23 instance 301 moving from disabled to blocking

Jan 3 15:14:43 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/23, changed state to up

Jan 3 15:14:43 GMT: %SPANTREE-6-PORT_STATE: Port Gi0/23 instance 2 moving from blocking to forwarding

Jan 3 15:14:44 GMT: %SPANTREE-6-PORT_STATE: Port Gi0/23 instance 301 moving from blocking to forwarding

Jan 3 15:20:08 GMT: %SPANTREE-6-PORT_STATE: Port Gi0/23 instance 2 moving from forwarding to disabled

Jan 3 15:20:08 GMT: %SPANTREE-6-PORT_STATE: Port Gi0/23 instance 301 moving from forwarding to disabled

Jan 3 15:20:09 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/23, changed state to down

Jan 3 15:20:10 GMT: %LINK-3-UPDOWN: Interface GigabitEthernet0/23, changed state to down

Jan 3 15:24:57 GMT: %LINK-3-UPDOWN: Interface GigabitEthernet0/24, changed state to up

Jan 3 15:24:58 GMT: %SPANTREE-6-PORT_STATE: Port Gi0/24 instance 2 moving from disabled to blocking

Jan 3 15:24:58 GMT: %SPANTREE-6-PORT_STATE: Port Gi0/24 instance 301 moving from disabled to blocking

Jan 3 15:24:59 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/24, changed state to up

Jan 3 15:25:00 GMT: %SPANTREE-6-PORT_STATE: Port Gi0/24 instance 2 moving from blocking to forwarding

Jan 3 15:25:00 GMT: %SPANTREE-6-PORT_STATE: Port Gi0/24 instance 301 moving from blocking to forwarding