Weird VLAN problem (maybe) on Cisco 3850 switch + 5508 WLC.

Chris751 · ‎09-16-2023

Hello everyone!
This is actually my first post here, and I'm seeking help on this forum. I'm really puzzled by the behavior of my switch. Here's the scenario:
I have a 3850 switch with two interfaces, g1/1/1 and g1/1/2, forming a LAG ( interface po1) that is connected to a WLC (5508).
Interface po1 is configured as a trunk allowing VLANs 100, 888, and 999. I have VLANs 100, 888, and 999 with SVIs:
VLAN 100 with SVI IP: 192.168.100.100
VLAN 888 with SVI IP: 192.168.88.100
VLAN 999 with SVI IP: 192.168.99.100
On the WLC, I have a management interface in VLAN 100 with the address 192.168.100.11.
The problem is that I can't ping the address 192.168.100.11 on the WLC from the switch when using VLAN 100 (which is what I want) nor VLAN 888 (888 is not surprising). However, I CAN ping it from VLAN 999! (which I think suggest that the LAG is probably working)
I don't understand why.
Below you can find configuration obtained by "sh run" from the switch and "show interface detailed management" from the WLC.

As far as I can tell, I don't have any routing enabled on the switch.
The only physical connections between the switch and the WLC are two fiber cables on ports g1/1/1 and g1/1/2, creating LAG - po1 - as mentioned above.

SWITCH:
Sat Sep 16 2023 23:31:09 GMT-0400 (Eastern Daylight Time)
===================================================================================
#sh run
Building configuration...
Current configuration : 9541 bytes
!
! Last configuration change at 03:16:49 UTC Sun Sep 17 2023 by admin
!
version 16.12
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
service call-home
platform punt-keepalive disable-kernel-core
!
hostname Switch
!
!
vrf definition Mgmt-vrf
!
address-family ipv4
exit-address-family
!
address-family ipv6
exit-address-family
!
!
no aaa new-model
switch 1 provision ws-c3850-24p
!
!
!
!
call-home
! If contact email address in call-home is configured as sch-smart-licensing@cisco.com
! the email address configured in Cisco Smart License Portal will be used as contact email address to send SCH notifications.
contact-email-addr sch-smart-licensing@cisco.com
profile "CiscoTAC-1"
active
destination transport-method http
no destination transport-method email
!
!
!
!
!
ip dhcp excluded-address 192.168.100.0 192.168.100.199
ip dhcp excluded-address 192.168.100.231 192.168.100.255
!
ip dhcp pool VLAN100-management
network 192.168.100.0 255.255.255.0
default-router 192.168.100.100
option 43 ip 192.168.100.11
lease infinite
!
!
!
login on-success log
!
!
!
!
!
!
!
no device-tracking logging theft
!
crypto pki trustpoint SLA-TrustPoint
enrollment pkcs12
revocation-check crl
!
crypto pki trustpoint TP-self-signed-4198860323
enrollment selfsigned
subject-name cn=IOS-Self-Signed-Certificate-4198860323
revocation-check none
rsakeypair TP-self-signed-4198860323
!
!
---------------------------------------------------------
removed by me
---------------------------------------------------------
!
license boot level ipservicesk9
!
!
diagnostic bootup level minimal
!
spanning-tree mode rapid-pvst
spanning-tree extend system-id
memory free low-watermark processor 79468
!
username admin privilege 15 secret 9 $9$gFGJBiHHVaei3k$fopPH3A0YWJZ0dE/EQU4UZuF.ibx.3eB5BT0YhlW9VE
!
redundancy
mode sso
!
!
!
!
!
transceiver type all
monitoring
!
!
class-map match-any system-cpp-police-topology-control
description Topology control
class-map match-any system-cpp-police-sw-forward
description Sw forwarding, L2 LVX data, LOGGING
class-map match-any system-cpp-default
description EWLC control, EWLC data, Inter FED
class-map match-any system-cpp-police-sys-data
description Learning cache ovfl, High Rate App, Exception, EGR Exception, NFL SAMPLED DATA, RPF Failed
class-map match-any system-cpp-police-punt-webauth
description Punt Webauth
class-map match-any system-cpp-police-l2lvx-control
description L2 LVX control packets
class-map match-any system-cpp-police-forus
description Forus Address resolution and Forus traffic
class-map match-any system-cpp-police-multicast-end-station
description MCAST END STATION
class-map match-any system-cpp-police-multicast
description Transit Traffic and MCAST Data
class-map match-any system-cpp-police-l2-control
description L2 control
class-map match-any system-cpp-police-dot1x-auth
description DOT1X Auth
class-map match-any system-cpp-police-data
description ICMP redirect, ICMP_GEN and BROADCAST
class-map match-any system-cpp-police-stackwise-virt-control
description Stackwise Virtual
class-map match-any non-client-nrt-class
class-map match-any system-cpp-police-routing-control
description Routing control and Low Latency
class-map match-any system-cpp-police-protocol-snooping
description Protocol snooping
class-map match-any system-cpp-police-dhcp-snooping
description DHCP snooping
class-map match-any system-cpp-police-system-critical
description System Critical and Gold Pkt
!
policy-map system-cpp-policy
!
!
!
!
!
!
!
!
!
!
!
interface Port-channel1
switchport trunk allowed vlan 100,888,999
switchport mode trunk
!
interface GigabitEthernet0/0
vrf forwarding Mgmt-vrf
ip address 192.168.27.101 255.255.255.0
negotiation auto
!
interface GigabitEthernet1/0/1
!
interface GigabitEthernet1/0/2
!
interface GigabitEthernet1/0/3
!
interface GigabitEthernet1/0/4
!
interface GigabitEthernet1/0/5
!
interface GigabitEthernet1/0/6
!
interface GigabitEthernet1/0/7
!
interface GigabitEthernet1/0/8
!
interface GigabitEthernet1/0/9
!
interface GigabitEthernet1/0/10
!
interface GigabitEthernet1/0/11
!
interface GigabitEthernet1/0/12
!
interface GigabitEthernet1/0/13
!
interface GigabitEthernet1/0/14
!
interface GigabitEthernet1/0/15
switchport trunk allowed vlan 100
switchport mode trunk
!
interface GigabitEthernet1/0/16
!
interface GigabitEthernet1/0/17
!
interface GigabitEthernet1/0/18
!
interface GigabitEthernet1/0/19
!
interface GigabitEthernet1/0/20
!
interface GigabitEthernet1/0/21
!
interface GigabitEthernet1/0/22
!
interface GigabitEthernet1/0/23
switchport access vlan 100
switchport mode access
spanning-tree portfast disable
!
interface GigabitEthernet1/0/24
!
interface GigabitEthernet1/1/1
switchport trunk allowed vlan 100,888,999
switchport mode trunk
channel-group 1 mode on
!
interface GigabitEthernet1/1/2
switchport trunk allowed vlan 100,888,999
switchport mode trunk
channel-group 1 mode on
!
interface GigabitEthernet1/1/3
!
interface GigabitEthernet1/1/4
!
interface TenGigabitEthernet1/1/1
!
interface TenGigabitEthernet1/1/2
!
interface TenGigabitEthernet1/1/3
!
interface TenGigabitEthernet1/1/4
!
interface Vlan1
no ip address
!
interface Vlan100
ip address 192.168.100.100 255.255.255.0
!
interface Vlan888
ip address 192.168.88.100 255.255.255.0
!
interface Vlan999
ip address 192.168.99.100 255.255.255.0
!
ip forward-protocol nd
ip http server
ip http authentication local
ip http secure-server
!
!
!
!
!
!
control-plane
service-policy input system-cpp-policy
!
!
line con 0
stopbits 1
line aux 0
stopbits 1
line vty 0 4
login
length 0
line vty 5 15
login
!
!
!
!
!
!
!
end
============================================================================
============================================================================
============================================================================

WLC:
(Cisco Controller) >show interface detailed management

Interface Name................................... management
MAC Address...................................... 54:75:d0:de:4b:8f
IP Address....................................... 192.168.100.11
IP Netmask....................................... 255.255.255.0
IP Gateway....................................... 192.168.100.100
External NAT IP State............................ Disabled
External NAT IP Address.......................... 0.0.0.0
Link Local IPv6 Address.......................... fe80::5675:d0ff:fede:4b8f/64
STATE ........................................... REACHABLE
Primary IPv6 Address............................. ::/128
STATE ........................................... NONE
Primary IPv6 Gateway............................. ::
Primary IPv6 Gateway Mac Address................. 00:00:00:00:00:00
STATE ........................................... INCOMPLETE
VLAN............................................. 100
Quarantine-vlan.................................. 0
Active Physical Port............................. LAG (13)
Primary Physical Port............................ LAG (13)
Backup Physical Port............................. Unconfigured
DHCP Proxy Mode.................................. Global
Primary DHCP Server.............................. 192.168.100.100

--More-- or (q)uit
Secondary DHCP Server............................ Unconfigured
DHCP Option 82................................... Disabled
DHCP Option 82 bridge mode insertion............. Disabled
IPv4 ACL......................................... Unconfigured
IPv6 ACL......................................... Unconfigured
mDNS Profile Name................................ Unconfigured
AP Manager....................................... Yes
Guest Interface.................................. No
L2 Multicast..................................... Enabled

Leo Laohoo · ‎09-16-2023

Where is the VLAN database?

Chris751 · ‎09-17-2023

It looks like it's in flash:/vlan.dat

Sun Sep 17 2023 13:55:05 GMT-0400 (Eastern Daylight Time)
===================================================================================
#sh vlan br
VLAN Name Status Ports
---- -------------------------------- --------- -------------------------------
1 default active Gi1/0/1, Gi1/0/2, Gi1/0/3
Gi1/0/4, Gi1/0/5, Gi1/0/6
Gi1/0/7, Gi1/0/8, Gi1/0/9
Gi1/0/10, Gi1/0/11, Gi1/0/12
Gi1/0/13, Gi1/0/14, Gi1/0/16
Gi1/0/17, Gi1/0/18, Gi1/0/19
Gi1/0/20, Gi1/0/21, Gi1/0/22
Gi1/0/24, Gi1/1/3, Gi1/1/4
100 management active Gi1/0/23
111 voice active
888 gns3lab active
999 test active
1002 fddi-default act/unsup
1003 trcrf-default act/unsup
1004 fddinet-default act/unsup
1005 trbrf-default act/unsup

Leo Laohoo · ‎09-17-2023

I do not see the VLANs in the config.

Chris751 · ‎09-17-2023

Right!
I changed VTP version from 1 to 2 and set it in transparent mode.
For good measure I issued vlan 100, vlan 888, and vlan 999 commands.
Now it shows up in the config:
!
vlan 100
name management
!
vlan 111
name voice
!
vlan 888
name gns3lab
!
vlan 999
name test
!
But I still can't ping from SVI in VLAN 100, address 100 192.168.100.100 to WLC 192.168.100.11 in the same VLAN 100.

At the same time I can keep pinging it from SVI in VLAN 999, and only from this SVI
Switch#ping
Protocol [ip]:
Target IP address: 192.168.100.11
Repeat count [5]:
Datagram size [100]:
Timeout in seconds [2]:
Extended commands [n]: y
Ingress ping [n]:
Source address or interface: 192.168.99.100
DSCP Value [0]:
Type of service [0]:
Set DF bit in IP header? [no]:
Validate reply data? [no]:
Data pattern [0x0000ABCD]:
Loose, Strict, Record, Timestamp, Verbose[none]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.100.11, timeout is 2 seconds:
Packet sent with a source address of 192.168.99.100
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/3 ms
========================================================================
========================================================================

When pinging from the WLC to the switch I have similar results, I can ping only SVI on the switch in VLAN 999, which isn't even configured on any interface of that WLC:
(Cisco Controller) >show interface summ
Number of Interfaces.......................... 6
Interface Name Port Vlan Id IP Address Type Ap Mgr Guest
-------------------------------- ---- -------- --------------- ------- ------ -----
gns3lab LAG 888 192.168.88.11 Dynamic No No
management LAG 100 192.168.100.11 Static Yes No
redundancy-management LAG 100 0.0.0.0 Static No No
redundancy-port - untagged 0.0.0.0 Static No No
service-port N/A N/A 192.168.27.11 Static No No
virtual N/A N/A 192.0.2.1 Static No No
(Cisco Controller) >ping 192.168.100.100
Send count=3, Receive count=0 from 192.168.100.100
(Cisco Controller) >ping 192.168.88.100
Send count=3, Receive count=0 from 192.168.88.100
(Cisco Controller) >ping 192.168.99.100
Send count=3, Receive count=3 from 192.168.99.100

Leo Laohoo · ‎09-17-2023

Where is the Native VLAN?

Chris751 · ‎09-17-2023

I can find it only in output of "sh int trunk"

Switch#sh int trunk

Port Mode Encapsulation Status Native vlan
Gi1/0/15 on 802.1q trunking 1
Po1 on 802.1q trunking 1

Port Vlans allowed on trunk
Gi1/0/15 100
Po1 100,888,999

Port Vlans allowed and active in management domain
Gi1/0/15 100
Po1 100,888,999

Port Vlans in spanning tree forwarding state and not pruned
Gi1/0/15 100
Po1 100,888,999

Leo Laohoo · ‎09-17-2023

On the controller, post the complete output to the command "sh interface detail management".

Chris751 · ‎09-17-2023

(Cisco Controller) >show interface detailed management

Interface Name................................... management
MAC Address...................................... 54:75:d0:de:4b:8f
IP Address....................................... 192.168.100.11
IP Netmask....................................... 255.255.255.0
IP Gateway....................................... 192.168.100.100
External NAT IP State............................ Disabled
External NAT IP Address.......................... 0.0.0.0
Link Local IPv6 Address.......................... fe80::5675:d0ff:fede:4b8f/64
STATE ........................................... REACHABLE
Primary IPv6 Address............................. ::/128
STATE ........................................... NONE
Primary IPv6 Gateway............................. ::
Primary IPv6 Gateway Mac Address................. 00:00:00:00:00:00
STATE ........................................... INCOMPLETE
VLAN............................................. 100
Quarantine-vlan.................................. 0
Active Physical Port............................. LAG (13)
Primary Physical Port............................ LAG (13)
Backup Physical Port............................. Unconfigured
DHCP Proxy Mode.................................. Global
Primary DHCP Server.............................. 192.168.100.100

--More-- or (q)uit
Secondary DHCP Server............................ Unconfigured
DHCP Option 82................................... Disabled
DHCP Option 82 bridge mode insertion............. Disabled
IPv4 ACL......................................... Unconfigured
IPv6 ACL......................................... Unconfigured
mDNS Profile Name................................ Unconfigured
AP Manager....................................... Yes
Guest Interface.................................. No
L2 Multicast..................................... Enabled

Leo Laohoo · ‎09-17-2023

@Chris751 wrote:
As far as I can tell, I don't have any routing enabled on the switch.

If there is no router doing any routing, the switch will need to have routing enabled.

Chris751 · ‎09-18-2023

So I did the following:
in config terminal I issued ip routing command, then I added OSPF routing for the networks I have configured on the switch.
Unfortunately it didn't help.

Luckily this is just my lab, and maybe there is some old configuration remaining in some file I don't know about. I think I'll go ahead, reset to the default configuration(hopefully) and I will rebuild it again with VLAN 1 only, and then I will add remaining VLANs and networks.
Thank you for your time and your willingness to help me!