int vlan ,weird behavior

[sarahr202]

Hi everybody

I encountered a very strange behavior.
( I changed the ip address and device name to protect customer privacy)

We have a vlan 182 for our customer ;vlan 182 has only port channel 2.

Customer is conncted to int vlan 182 on router  ( int vlan 182: 10.10.10.1, ) ( CE is 10.10.10.2)

CE---sw---po 2-----Router

Po2 has following ports as memebers:

router#show etherchannel summary | include Po2
2      Po2(SU)         LACP      Gi6/11(P)  Gi6/12(P)

show log | include 182

May  9 00:36:30.724 GMT: %LINK-3-UPDOWN: Interface Vlan182, changed state to down
May  9 00:36:30.740 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan182, changed state to down
May  9 00:37:21.716 GMT: %LINK-3-UPDOWN: Interface Vlan182, changed state to up
May  9 00:37:21.720 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan182, changed state to up

In order to vlan 182 goes down either we have to shutdown using " shutdown" under int vlan 182 , or all the members ports of Po2 are down. As long as Po2 is up ( Po2 will be up as long as atleat one memeber port is up and up), and we do not use any " shutdown under int vlan 182, vlan 182 should be up and up.

I am noticing a weird unexpected behavior, where all the member ports ( Gi6/11(P)  Gi6/12(P)) of Po2 are up and up i.e they never bounce but yet int vlan 182 is keep bouncing.

router#show log | include Gi6/11

router#show log | include Gi6/12

no log messages.

++++++++++++++++++++++++++++++++++++++++++

router#show running-config interface vlan182
Building configuration...

Current configuration : 227 bytes
!
interface Vlan182
 
 ip address 10.10.10.1 255.255.255.252
 ip verify unicast source reachable-via any allow-self-ping
 no ip redirects
 no ip proxy-arp
 mls netflow sampling

router#show running-config interface Po2

interface Port-channel2
 
 switchport
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 182
 
 switchport mode trunk
no ip address
 speed nonegotiate

router#show version

Cisco Internetwork Operating System Software
IOS (tm) s72033_rp Software (s72033_rp-ADVIPSERVICESK9_WAN-M), Version 12.2(18)SXF14, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2008 by cisco Systems, Inc.
Compiled Thu 08-May-08 00:57 by kellythw
Image text-base: 0x40101040, data-base: 0x42DC86D0

ROM: System Bootstrap, Version 12.2(17r)S2, RELEASE SOFTWARE (fc1)
BOOTLDR: s72033_rp Software (s72033_rp-ADVIPSERVICESK9_WAN-M), Version 12.2(18)SXF14, RELEASE SOFTWARE (fc1)

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

I apprecaite your help

Thanks

[nkarpysh]

Hi,

I guess its it typo but anyway need to double-check:

router#show log | include Gi6/11

router#show log | include Gi6/12

Above command wont give you any result as in logs interface is showing with full name  - e.g. GigabitEthernet6/11, thus filtering log for short name wont help. Please filter once again for 6/11 or 6/12 only. Other thing is that SVI can go down when all ports it have are in blocking state - thus please check for any STP change/ TCN in this VLAN.

Niko

[sarahr202]

router#  show log | include GigabitEthernet6/11

router#  show log | include GigabitEthernet6/12

router#show spanning-tree vlan 182 detail

 VLAN0182 is executing the ieee compatible Spanning Tree protocol
  Bridge Identifier has priority 32768, sysid 182, address 0011.bcdb.6300
  Configured hello time 2, max age 20, forward delay 15
  We are the root of the spanning tree
  Topology change flag not set, detected flag not set
  Number of topology changes 73 last change occurred 00:13:38 ago
          from Port-channel2
  Times:  hold 1, topology change 35, notification 2
          hello 2, max age 20, forward delay 15
  Timers: hello 0, topology change 0, notification 0, aging 300

 Port 1665 (Port-channel2) of VLAN0182 is forwarding
   Port path cost 3, Port priority 128, Port Identifier 128.1665.
   Designated root has priority 32950, address 0011.bcdb.6300
   Designated bridge has priority 32950, address 0011.bcdb.6300
   Designated port id is 128.1665, designated path cost 0
   Timers: message age 0, forward delay 0, hold 0
   Number of transitions to forwarding state: 37
   Link type is point-to-point by default
   BPDU: sent 4536739, received 43842

++++++++++++++++++++++++++++++++++++++++

We have multiple customers riding on same port channel 2 , port -channel 2 is carrying different vlans , at router, customers traffic are directed to their corresponding int vlan.  But it is only one customer who is riding on vlan 182 has a issue. So we can rule out any issues with port channel 2 and its member ports ( gi6/11.gi6/12)

If you see my output above, you see router is root switch and therefore it sends BPDU . The second int vlan 182 goes down ( as you mentioned it could be both ports g6/11,g6/12 moves to blocking state for some reason ) , root election is held and we see received bpdu counter increases and once the router becomes root, it starts sending bpdu normaly.

We see alot of TCN BPDU, not sure if that counter also increments when int vlan 182 flaps ( which means  gi6/11,gi6/12 moves to blocing state for vlan 182).

The question is why TCN BPDU could cause gi6/11, gi 6/12 to move blocking state ?

I really apprecaite your help

Thanks

[nkarpysh · Accepted solution]

Hey,

Does the router send the STP to this switch? I also want to confirm the config of Po2 - you said it runs different VLANs but as per your run config above it allows only 182:

interface Port-channel2
 
 switchport
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 182 
 switchport mode trunk
no ip address
 speed nonegotiate

Just for us to be on same page.

TCN itself should not change the port state to blocking - but it is anyway is not the good sign. I see last one came from Po2 thus it would be go to go to the device which is running STP behind Po2 (not sure if that is router on your diag above or some device further) and see if it has any link flapping or if that got also TCN from one of the downstream devices. Tracing this we can find the source of topology changes and fix it.

Niko

[sarahr202]

Hi Niko

I hope you are doing fine. Sorry about the late reply..

CE----Sw---Po-------R

Not shown, there are multiple customers  connected to switch, each riding on different vlans through switch over po and eventually reaching Router to their corresponding  layer 3 int vlans.

All the customers are fine, except for one, he was fine until 30 April last month. The only thing we see a lot of TCN from po on router.    When I do the show mac for customer vlan 182 on the switch , i see two mac addresses coming from CE indicating the Customer has installed a switch  to our circuit and then connected his router to that switch. 

I believe everytime the ports on that switch goes and up down, ir sends TC BPDU towards our router . Still not sure how it could cause to int vlan to bounce.

Anyway,asked the customer if he really installed a switch around 30 April.

If yes, remove it and see if the symptoms go away.

So far no response. I will let you know once I hear from him

Thanks for your valuable help