VLAN stops working after few seconds

[Muzzy011]

Hello Everyone,

I have a problem that VLAN100 created is accessible for a very short period of time, and then simply stop working.

So, if I go to config to VLAN100 interface, do 'shutdown' and 'no shutdown" right after, computer connected on one of the ports of switch, with access to VLAN100 starts being ping-able from switch, but after few seconds, it stops. Doing shutdown - no shutdown will make it live for few seconds and then will stop again. When it is stopped, accessing VLAN through any other access/trunk port will not work. VLAN1 works all the time. 

Topology is: 1921 as bridge to rest of the network - C3750 with enabled routing as main, 3 x C3750 as attached switches to main one, all 3 through fiber ports, all ports trunked for all VLAN's on both sides.

Main switch is VTP master, VTP settings on other switches are in sync.

Extreme Wi-Fi controller, which is on 'slave' switch. have VLAN100 set, and IP set in that VLAN is ping-able across the network. Port on switch for Extreme is trunked for all VLAN's.

Short config of main switch below:

version 15.2
no service pad
service timestamps debug datetime msec localtime show-timezone
service timestamps log datetime msec localtime show-timezone
service password-encryption
no aaa new-model
clock timezone PST -8 0
clock summer-time PDT recurring
switch 1 provision ws-c3750x-48p
system mtu routing 1500
ip routing
no ip domain-lookup
ip device tracking
spanning-tree mode pvst
spanning-tree loopguard default
spanning-tree portfast bpduguard default
spanning-tree extend system-id
vlan internal allocation policy ascending

!

interface GigabitEthernet1/0/2 #test port for test computer
 switchport access vlan 100
 switchport mode access
 spanning-tree guard none
!
interface GigabitEthernet1/1/1 #Slave switch1
 switchport trunk encapsulation dot1q
 switchport mode trunk
!
interface GigabitEthernet1/1/2 #Slave switch2
 switchport trunk encapsulation dot1q
 switchport mode trunk
!
interface GigabitEthernet1/1/3 #Slave switch3
 switchport trunk encapsulation dot1q
 switchport mode trunk
!
interface Vlan1
 ip address 10.1.5.254 255.255.255.0
!
interface Vlan100
 ip address 10.100.1.1 255.255.255.0
!
interface Vlan125
 ip address 10.125.26.1 255.255.255.0
 ip helper-address 10.1.5.2
!
ip default-gateway 10.1.5.1 #(bridge router IP)
ip http server
ip http secure-server
ip route 0.0.0.0 0.0.0.0 10.1.5.1
ip route 10.1.0.0 255.255.254.0 10.1.5.1 #Main network
!
logging trap debugging
logging host 10.32.1.21

[Georg Pauwen]

Hello,

find out what switch is the root for your Vlans (it should be the 3750):

show spanning-tree vlan 100

The output should say 'This switch is the root'

[Muzzy011]

This is what I am getting:

VLAN0100
  Spanning tree enabled protocol ieee
  Root ID    Priority    32868
             Address     3c08.f6c7.a580
             Cost        4
             Port        50 (GigabitEthernet1/1/2)
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32868  (priority 32768 sys-id-ext 100)
             Address     6c41.6a0f.0800
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time  300 sec

Interface           Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi1/0/2             Desg FWD 4         128.2    P2p
Gi1/0/46            Desg FWD 4         128.46   P2p
Gi1/1/1             Desg FWD 4         128.49   P2p
Gi1/1/2             Root FWD 4         128.50   P2p
Gi1/1/3             Desg FWD 4         128.51   P2p

[Georg Pauwen]

Hello, 

the root for your Vlan is the switch connected to interface GigabitEthernet1/1/2, try and make the core (the 3750 where you have the routing enabled, and that is connected to the router) the root:

spanning-tree vlan 100 root primary

Actually, I would make that switch the root for ALL your Vlans...

[Muzzy011]

Georg,

I did what you said, now the main switch is root (showing 'this bridge is the root' when type spanning-tree vlan 100), but still having the same issue - shutdown, no shutdown on vlan100 interface made it work for about two minutes, and then stopped again.

Other 'slave' switch that have Extreme Wi-Fi controller, and vlan 100 set, with interface set to 10.100.1.2 in vlan 100, is ping-able from whole network (switch port is trunk), but computers that are on main switch (port in access state for vlan 100) and another 'slave' switch (port is in access state for vlan 100) are not ping-able, not from network or from switches. 

Antivirus on computers is disabled, I checked with NMAP all the possible ports on those computers, and NMAP is saying that host is down.

[Muzzy011]

Here is some addition..

Port on computer is in 'undefined network' state, although some communication is passing through, grabbed with Wireshark, look below - 10.1.5.2 is dc trying to ping this machine, 10.100.1.55 is another computer on another switch in network, 10.100.1.88 is machine that I am at. I don't know why this computer is not replying to ICMP from 10.1.5.2, as firewall is off, and no any kind of antivirus on computer. 

82 45.894607 Dell_13:74:61 Broadcast ARP 60 Who has 10.100.1.1? Tell 10.100.1.55
83 46.236397 Cisco_a6:c0:0c Spanning-tree-(for-bridges)_00 STP 60 Conf. Root = 24576/100/6c:41:6a:0f:08:00 Cost = 4 Port = 0x807c
84 47.019821 10.1.5.2 10.100.1.88 ICMP 74 Echo (ping) request id=0x000f, seq=18090/43590, ttl=127 (no response found!)
85 47.019919 CableMat_03:32:12 Broadcast ARP 42 Who has 10.100.1.1? Tell 10.100.1.88
86 47.402925 10.100.1.55 239.255.255.250 SSDP 175 M-SEARCH * HTTP/1.1
87 47.736367 CableMat_03:32:12 Broadcast ARP 42 Who has 10.100.1.1? Tell 10.100.1.88
88 48.247068 Cisco_a6:c0:0c Spanning-tree-(for-bridges)_00 STP 60 Conf. Root = 24576/100/6c:41:6a:0f:08:00 Cost = 4 Port = 0x807c
89 48.736086 CableMat_03:32:12 Broadcast ARP 42 Who has 10.100.1.1? Tell 10.100.1.88
90 48.814512 Cisco_a6:c0:0c Cisco_a6:c0:0c LOOP 60 Reply
91 50.252485 Cisco_a6:c0:0c Spanning-tree-(for-bridges)_00 STP 60 Conf. Root = 24576/100/6c:41:6a:0f:08:00 Cost = 4 Port = 0x807c
92 50.430936 10.100.1.55 239.255.255.250 SSDP 175 M-SEARCH * HTTP/1.1
93 51.875565 CableMat_03:32:12 Broadcast ARP 42 Who has 10.100.1.1? Tell 10.100.1.88
94 52.003258 10.100.1.88 239.255.255.250 SSDP 179 M-SEARCH * HTTP/1.1
95 52.019725 10.1.5.2 10.100.1.88 ICMP 74 Echo (ping) request id=0x000f, seq=18091/43846, ttl=127 (no response found!)
96 52.019917 CableMat_03:32:12 Broadcast ARP 42 Who has 10.100.1.1? Tell 10.100.1.88
97 52.255564 Cisco_a6:c0:0c Spanning-tree-(for-bridges)_00 STP 60 Conf. Root = 24576/100/6c:41:6a:0f:08:00 Cost = 4 Port = 0x807c

[paul driver]

Hello

What does the logging report when this L3 interface goes down, what the interface state?.

Are you using non Cisco switches?
Can you post a diagram of you topology?

Also suggest not to use the root primary macro and defined an actual value for the root bridge, as the macro will only set a lower value at that time if another switch is introduce after this the root bridge can be overridden  as it doesn't preempt to a lower value.

sh interface vlan 100 status
sh interface status err-disabled

sh logging
sh cdp neighbors
sh spanning-tree

Suggested config changes:
no ip default-gateway 10.1.5.1 <--- not required as ip routing is enabled
no ip route 10.1.0.0 255.255.254.0 10.1.5.1 <--- not required as default route is the same next hop
no ip device tracking <-- can cause unnecessary false duplicate ip addressing
spanning-tree vlan 1-4094 priority 0  < make the switch stp root of all vlans

[Muzzy011]

Hello Paul,

I did applied all four suggested config changes.

All four switches are Cisco, all four are C3750-48-PoE,

Primary one is lanbase license, one of slave switches is also lanbase, other two are ipbase

Topology:

(Firewall)--(Main Loc Stack C3750, 10.1.0.0/23)--(1921)--(Warehouse main sw 10.1.5.0/24)-(3 switches connected to main, fiber 1/1/1 1/1/2 and 1/1/3, trunk ports, all VLAN's)

When VLAN 100 is shudtdown, logging shows:

%LINK-5-CHANGED: Interface Vlan100, changed state to administratively down
%LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan100, changed state to down

sh int vlan 100

Vlan100 is up, line protocol is up
Hardware is EtherSVI, address is 6c41.6a0f.0841 (bia 6c41.6a0f.0841)
Description: Company Wi-Fi Range
Internet address is 10.100.1.1/24
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not supported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:23, output 00:00:23, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
82490 packets input, 5999261 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
80038 packets output, 5175631 bytes, 0 underruns
0 output errors, 23 interface resets
0 unknown protocol drops
0 output buffer failures, 0 output buffers swapped out

sh int status err-disabled shows nothing

sh logging

Syslog logging: enabled (0 messages dropped, 0 messages rate-limited, 0 flushes, 0 overruns, xml disabled, filtering disabled)

No Active Message Discriminator.

No Inactive Message Discriminator.

Console logging: level debugging, 8352 messages logged, xml disabled,
filtering disabled
Monitor logging: level debugging, 0 messages logged, xml disabled,
filtering disabled
Buffer logging: level debugging, 8353 messages logged, xml disabled,
filtering disabled
Exception Logging: size (4096 bytes)
Count and timestamp logging messages: disabled
File logging: disabled
Persistent logging: disabled

No active filter modules.

Trap logging: level alerts, 7581 message lines logged
Logging to 10.32.1.21 (udp port 514, audit disabled,
link up),
2486 message lines logged,
0 message lines rate-limited,
0 message lines dropped-by-MD,
xml disabled, sequence number disabled
filtering disabled
Logging Source-Interface: VRF Name:

Log Buffer (10000 bytes):
low alarm; Operating value: -20.1 dBm, Threshold value: -20.0 dBm.
Jun 19 06:27:21.080 PDT: %SFF8472-5-THRESHOLD_VIOLATION: Gi1/1/1: Rx power low alarm; Operating value: -20.3 dBm, Threshold value: -20.0 dBm. (shows lot of times, stripped the rest, long fiber line)
Jun 19 13:46:57.567 PDT: %SW_MATM-4-MACFLAP_NOTIF: Host 94fb.2929.a83c in vlan 1 is flapping between port Gi1/0/34 and port Gi1/0/36 (Wi-Fi access points, same users switching in between them)
Jun 19 16:05:01.922 PDT: %SW_MATM-4-MACFLAP_NOTIF: Host 94fb.2929.b00e in vlan 1 is flapping between port Gi1/0/36 and port Gi1/0/34
Jun 19 16:06:52.454 PDT: %SW_MATM-4-MACFLAP_NOTIF: Host 94fb.2929.b00e in vlan 1 is flapping between port Gi1/0/32 and port Gi1/1/2
Jun 19 16:11:20.399 PDT: %SW_MATM-4-MACFLAP_NOTIF: Host 94fb.2929.b00e in vlan 1 is flapping between port Gi1/0/32 and port Gi1/0/36
Jun 19 16:14:24.303 PDT: %SW_MATM-4-MACFLAP_NOTIF: Host 94fb.2929.a0ce in vlan 1 is flapping between port Gi1/1/2 and port Gi1/0/32
Jun 19 16:23:27.130 PDT: %SYS-5-CONFIG_I: Configured from console by vty0 (10.1.0.72)
Jun 19 16:35:33.927 PDT: %SYS-5-CONFIG_I: Configured from console by vty0 (10.1.0.72)
Jun 19 16:37:37.175 PDT: %LINK-5-CHANGED: Interface Vlan100, changed state to administratively down
Jun 19 16:37:37.184 PDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan100, changed state to down
Jun 19 16:38:13.714 PDT: %SYS-5-CONFIG_I: Configured from console by vty0 (10.1.0.72)
Jun 19 16:41:32.254 PDT: %LINK-3-UPDOWN: Interface Vlan100, changed state to up
Jun 19 16:41:32.263 PDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan100, changed state to up
Jun 19 16:43:46.994 PDT: %SYS-5-CONFIG_I: Configured from console by vty0 (10.1.0.72)

sh cdp neighbors

Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone,
D - Remote, C - CVTA, M - Two-port Mac Relay

Device ID Local Intrfce Holdtme Capability Platform Port ID
JordanWH2 Gig 1/1/2 123 S I WS-C3750X Gig 1/1/4
JordanWH1 Gig 1/1/1 135 S I WS-C3750X Gig 1/1/2
AP2-NATROL-CW-1854
Gig 1/0/30 121 R S AP-7522-6 ge1
AP1-NATROL-CW-0DD8
Gig 1/0/36 179 R S AP-7522-6 ge1
JordanOffice Gig 1/1/3 149 S I WS-C3750X Gig 5/1/4
AP10-NATROL-CW-1224
Gig 1/0/28 120 R S AP-7522-6 ge1
APf866.f2ab.1d38 Gig 1/0/35 166 T AIR-LAP12 Gig 0
Jordan-WLC Gig 1/0/46 128 H AIR-CT250 Gig 0/0/1
APf866.f2ab.1dcd Gig 1/0/31 133 T AIR-LAP12 Gig 0
APf866.f2ab.1d61 Gig 1/0/14 121 T AIR-LAP12 Gig 0
AP4-NATROL-CW-0E98
Gig 1/0/32 124 R S AP-7522-6 ge1
JordanRoute Gig 1/0/1 159 R B S I CISCO1921 Gig 0/1
AP3-NATROL-CW-0BEC
Gig 1/0/34 126 R S AP-7522-6 ge1

sh spanning-tree

VLAN0001
Spanning tree enabled protocol ieee
Root ID Priority 1
Address 6c41.6a0f.0800
This bridge is the root
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 1 (priority 0 sys-id-ext 1)
Address 6c41.6a0f.0800
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Aging Time 300 sec

Interface Role Sts Cost Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi1/0/1 Desg FWD 4 128.1 P2p
Gi1/0/4 Desg FWD 4 128.4 P2p Edge
Gi1/0/6 Desg FWD 4 128.6 P2p Edge
Gi1/0/8 Desg FWD 4 128.8 P2p Edge
Gi1/0/13 Desg FWD 4 128.13 P2p Edge
Gi1/0/14 Desg FWD 4 128.14 P2p Edge
Gi1/0/24 Desg FWD 19 128.24 P2p Edge
Gi1/0/25 Desg FWD 4 128.25 P2p Edge
Gi1/0/27 Desg FWD 19 128.27 P2p Edge
Gi1/0/28 Desg FWD 4 128.28 P2p Edge
Gi1/0/30 Desg FWD 4 128.30 P2p Edge
Gi1/0/31 Desg FWD 19 128.31 P2p Edge
Gi1/0/32 Desg FWD 4 128.32 P2p Edge
Gi1/0/34 Desg FWD 4 128.34 P2p Edge
Gi1/0/35 Desg FWD 4 128.35 P2p Edge
Gi1/0/36 Desg FWD 4 128.36 P2p Edge
Gi1/0/44 Desg FWD 19 128.44 P2p Edge
Gi1/0/45 Desg FWD 19 128.45 P2p Edge
Gi1/0/46 Desg FWD 4 128.46 P2p
Gi1/1/1 Desg FWD 4 128.49 P2p
Gi1/1/2 Desg FWD 4 128.50 P2p
Gi1/1/3 Desg FWD 4 128.51 P2p

VLAN0100
Spanning tree enabled protocol ieee
Root ID Priority 100
Address 6c41.6a0f.0800
This bridge is the root
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 100 (priority 0 sys-id-ext 100)
Address 6c41.6a0f.0800
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Aging Time 300 sec

Interface Role Sts Cost Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi1/0/2 Desg FWD 4 128.2 P2p
Gi1/0/46 Desg FWD 4 128.46 P2p
Gi1/1/1 Desg FWD 4 128.49 P2p
Gi1/1/2 Desg FWD 4 128.50 P2p
Gi1/1/3 Desg FWD 4 128.51 P2p

[Joseph W. Doherty]

Per chance, you don't have auto-prune enabled (via VTP)?

[Muzzy011]

Hello Joseph, 

No, it is disabled.

VTP Version capable : 1 to 3
VTP version running : 1
VTP Domain Name : Jordan
VTP Pruning Mode : Disabled
VTP Traps Generation : Disabled
Device ID : 6c41.6a0f.0800
Configuration last modified by 10.1.5.254 at 5-12-11 00:00:49
Local updater ID is 10.1.5.254 on interface Vl1 (lowest numbered VLAN interface found)

Feature VLAN:
--------------
VTP Operating Mode : Server
Maximum VLANs supported locally : 255
Number of existing VLANs : 9
Configuration Revision : 9
MD5 digest : 0x7E 0xE7 0xFA 0x02 0x56 0x22 0xCB 0x1C
0xB8 0x66 0x97 0x36 0x79 0xF1 0xE2 0x97

[Muzzy011]

I guess that problem is here (grabbed from Wireshark):

4 1.771557 10.1.5.2 10.100.1.88 ICMP 74 Echo (ping) request id=0x000f, seq=37918/7828, ttl=127 (no response found!)
5 1.771704 CableMat_03:32:12 Broadcast ARP 42 Who has 10.100.1.1? Tell 10.100.1.88

Broadcast to ask who have gateway address, initiated by ICMP, returns nothing.

So why that work on Vlan1, and not work on any other VLAN?

[Muzzy011]

I am now very close to think that is some firmware flaw, or similar.

Version of IOS that is installed only on main swithc and one of slave switches is (C3750E-UNIVERSALK9-M), Version 15.2(1)E, RELEASE SOFTWARE (fc3), all others have  (C3750E-UNIVERSALK9-M), Version 15.0(2)SE9, RELEASE SOFTWARE (fc1)

Two computers, one on main switch, other on slave swithc, both on VLAN 100, when they are pinged, they send broadcast asking who has 10.100.1.1 gateway for VLAN 100, I can see these broadcasts on each other, but switch is not responding to them?

Last night I downed all switches but main, downed all Wireless access points and controllers. left up just uplink router and one computer connected to VLAN 100, tried few times shutdown/no shutdown on VLAN100 interface, computer got accessible any time in between very few seconds and up to 2 minutes, and then not.

doing no vtp/vtp on VLAN100 made it accesible for short time once, also interface came back out of blue for few minutes while I was testing different things, and get down shortly after. 

Very frustrating.

[Muzzy011 · Accepted solution]

I resolved the problem.

Turned out that problem was firmware - C3750E-UNIVERSALK9-M), Version 15.2(1)E, RELEASE SOFTWARE (fc3)

When I flashed C3750E-UNIVERSALK9-M), Version 15.0(2)SE9, RELEASE SOFTWARE (fc1), everything started working as it should.

Thank you everybody for help and support!

Ivan.