Alright I'm at my wits end. I need some help configuring a basic WLC setup but from distribution switch 1 to access switch 1 where the WLC makes two hops to get to the Lightweight access point it is supposed to manage. My issue is that I CAN ping the WLC from the guests subnet but the WLC can't ping the guests subnet, the AP is even getting a DHCP address that is correct for the subnet with option 43 to tell it what the logical interface is for the subnets WLC and the gateway for the subnet being the default router of the subnet. I can also ping the WLC 10.0.10.100 from any 10.0.10.0/24 address as well as the 10.0.3.254/22 network from the 10.0.10.24/24 network on my laptop (not shown) so the only device that can't reach the 10.0.3.254 address is the WLC
Here's a network diagram.
Here's the WLC configuration.
Now for the Switch running configs.
----------------------------------
AS1_3750-X#show run
Building configuration...
Current configuration : 4802 bytes
!
! Last configuration change at 01:48:44 UTC Mon Jan 2 2006 by Steven
!
version 15.2
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname AS1_3750-X
!
boot-start-marker
boot-end-marker
!
!
logging monitor informational
!
no aaa new-model
switch 1 provision ws-c3750x-24p
system mtu routing 1500
!
!
!
!
ip routing
no ip cef optimize neighbor resolution
!
!
!
ip domain-name cyberfjord.com
!
!
!
!
!
!
crypto pki trustpoint TP-self-signed-x
enrollment selfsigned
subject-name cn=IOS-Self-Signed-Certificate-x
revocation-check none
rsakeypair TP-self-signed-x
!
!
crypto pki certificate chain TP-self-signed-x
quit
!
!
!
!
spanning-tree mode pvst
spanning-tree extend system-id
!
!
!
!
vlan internal allocation policy ascending
!
!
!
!
!
!
!
!
!
!
!
!
!
interface Loopback0
ip address 1.1.1.1 255.255.255.255
!
interface Port-channel1
description to DS1
no switchport
ip address 10.0.6.1 255.255.255.252
!
interface FastEthernet0
no ip address
no ip route-cache
!
interface GigabitEthernet1/0/1
!
interface GigabitEthernet1/0/2
description Wireless Access Points
switchport access vlan 100
switchport mode access
switchport nonegotiate
spanning-tree bpduguard enable
!
interface GigabitEthernet1/0/3
!
interface GigabitEthernet1/0/4
description Wireless Access Points
switchport access vlan 100
switchport mode access
switchport nonegotiate
spanning-tree bpduguard enable
!
interface GigabitEthernet1/0/5
!
interface GigabitEthernet1/0/6
!
interface GigabitEthernet1/0/7
!
interface GigabitEthernet1/0/8
!
interface GigabitEthernet1/0/9
!
interface GigabitEthernet1/0/10
!
interface GigabitEthernet1/0/11
!
interface GigabitEthernet1/0/12
!
interface GigabitEthernet1/0/13
!
interface GigabitEthernet1/0/14
!
interface GigabitEthernet1/0/15
!
interface GigabitEthernet1/0/16
!
interface GigabitEthernet1/0/17
!
interface GigabitEthernet1/0/18
!
interface GigabitEthernet1/0/19
!
interface GigabitEthernet1/0/20
!
interface GigabitEthernet1/0/21
!
interface GigabitEthernet1/0/22
!
interface GigabitEthernet1/0/23
no switchport
no ip address
channel-group 1 mode on
!
interface GigabitEthernet1/0/24
no switchport
no ip address
channel-group 1 mode on
!
interface GigabitEthernet1/1/1
!
interface GigabitEthernet1/1/2
!
interface GigabitEthernet1/1/3
!
interface GigabitEthernet1/1/4
!
interface TenGigabitEthernet1/1/1
!
interface TenGigabitEthernet1/1/2
!
interface Vlan1
no ip address
!
interface Vlan2
description Guests
no ip address
ip helper-address 10.0.6.2
!
interface Vlan100
ip address 10.0.3.254 255.255.252.0
ip helper-address 10.0.6.2
!
router ospf 1
router-id 1.1.1.1
auto-cost reference-bandwidth 100000
passive-interface Vlan2
passive-interface Loopback0
network 1.1.1.1 0.0.0.0 area 0
network 10.0.0.0 0.0.3.255 area 0
network 10.0.6.0 0.0.0.3 area 0
!
ip forward-protocol nd
!
!
ip http server
ip http secure-server
!
!
ip sla enable reaction-alerts
access-list 1 permit 10.0.10.22
!
!
!
line con 0
exec-timeout 15 0
logging synchronous
login local
line vty 0 4
access-class 1 in
exec-timeout 15 0
logging synchronous
login local
transport input ssh
line vty 5 15
access-class 1 in
exec-timeout 15 0
logging synchronous
login local
transport input ssh
!
!
end
______________________________________________
DS1_3850#show run
Building configuration...
Current configuration : 9202 bytes
!
! Last configuration change at 14:47:37 EDT Sat Oct 21 2023 by Steven
! NVRAM config last updated at 13:57:11 EDT Sat Oct 21 2023 by Steven
!
version 16.12
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
service call-home
platform punt-keepalive disable-kernel-core
platform management port rate-limt-disabled
!
hostname DS1_3850
!
!
vrf definition Mgmt-vrf
!
address-family ipv4
exit-address-family
!
address-family ipv6
exit-address-family
!
logging monitor informational
!
no aaa new-model
boot system switch all flash:packages.conf
clock timezone EST -5 0
clock summer-time EDT recurring
switch 1 provision ws-c3850-24p
!
!
!
!
call-home
! If contact email address in call-home is configured as sch-smart-licensing@cisco.com
! the email address configured in Cisco Smart License Portal will be used as contact email address to send SCH notifications.
contact-email-addr sch-smart-licensing@cisco.com
profile "CiscoTAC-1"
active
destination transport-method http
no destination transport-method email
ip routing
!
!
!
!
!
ip domain name cyberfjord.com
ip dhcp excluded-address 10.0.0.0 10.0.1.0
ip dhcp excluded-address 10.0.10.1 10.0.10.100
!
ip dhcp pool VLAN2
network 10.0.0.0 255.255.252.0
default-router 10.0.3.254
domain-name wifi.cyberfjord.com
dns-server 10.0.5.1
option 43 ip 10.0.0.100
lease 0 4
!
ip dhcp pool VLAN100
network 10.0.10.0 255.255.255.0
dns-server 10.0.5.1
domain-name Management.cyberfjord.com
default-router 10.0.10.254
option 43 ip 10.0.10.100
!
!
!
login on-success log
!
!
!
!
!
!
!
no device-tracking logging theft
!
crypto pki trustpoint SLA-TrustPoint
enrollment pkcs12
revocation-check crl
!
crypto pki trustpoint TP-self-signed-x
enrollment selfsigned
subject-name cn=IOS-Self-Signed-Certificate-x
revocation-check none
rsakeypair TP-self-signed-x
!
!
crypto pki certificate chain SLA-TrustPoint
quit
crypto pki certificate chain TP-self-signed-x
!
!
license boot level ipservicesk9
!
!
diagnostic bootup level minimal
!
spanning-tree mode rapid-pvst
spanning-tree extend system-id
memory free low-watermark processor 79468
!
!
redundancy
mode sso
!
!
!
!
!
transceiver type all
monitoring
!
!
class-map match-any system-cpp-police-topology-control
description Topology control
class-map match-any system-cpp-police-sw-forward
description Sw forwarding, L2 LVX data, LOGGING
class-map match-any system-cpp-default
description EWLC control, EWLC data, Inter FED
class-map match-any system-cpp-police-sys-data
description Learning cache ovfl, High Rate App, Exception, EGR Exception, NFL SAMPLED DATA, RPF Failed
class-map match-any system-cpp-police-punt-webauth
description Punt Webauth
class-map match-any system-cpp-police-l2lvx-control
description L2 LVX control packets
class-map match-any system-cpp-police-forus
description Forus Address resolution and Forus traffic
class-map match-any system-cpp-police-multicast-end-station
description MCAST END STATION
class-map match-any system-cpp-police-multicast
description Transit Traffic and MCAST Data
class-map match-any system-cpp-police-l2-control
description L2 control
class-map match-any system-cpp-police-dot1x-auth
description DOT1X Auth
class-map match-any system-cpp-police-data
description ICMP redirect, ICMP_GEN and BROADCAST
class-map match-any system-cpp-police-stackwise-virt-control
description Stackwise Virtual
class-map match-any non-client-nrt-class
class-map match-any system-cpp-police-routing-control
description Routing control and Low Latency
class-map match-any system-cpp-police-protocol-snooping
description Protocol snooping
class-map match-any system-cpp-police-dhcp-snooping
description DHCP snooping
class-map match-any system-cpp-police-system-critical
description System Critical and Gold Pkt
!
policy-map system-cpp-policy
!
!
!
!
!
!
!
!
!
!
!
interface Loopback0
ip address 14.14.14.14 255.255.255.255
!
interface Port-channel1
description To AS1
no switchport
ip address 10.0.6.2 255.255.255.252
!
interface Port-channel2
description To SRX
no switchport
ip address 192.168.1.1 255.255.255.252
!
interface Port-channel3
switchport trunk native vlan 1001
switchport trunk allowed vlan 2,100
switchport mode trunk
switchport nonegotiate
!
interface GigabitEthernet0/0
vrf forwarding Mgmt-vrf
no ip address
negotiation auto
!
interface GigabitEthernet1/0/1
!
interface GigabitEthernet1/0/2
!
interface GigabitEthernet1/0/3
!
interface GigabitEthernet1/0/4
!
interface GigabitEthernet1/0/5
!
interface GigabitEthernet1/0/6
!
interface GigabitEthernet1/0/7
!
interface GigabitEthernet1/0/8
!
interface GigabitEthernet1/0/9
!
interface GigabitEthernet1/0/10
!
interface GigabitEthernet1/0/11
!
interface GigabitEthernet1/0/12
!
interface GigabitEthernet1/0/13
no switchport
no ip address
channel-group 2 mode passive
!
interface GigabitEthernet1/0/14
no switchport
no ip address
channel-group 2 mode passive
!
interface GigabitEthernet1/0/15
switchport trunk native vlan 1001
switchport trunk allowed vlan 2,100
switchport mode trunk
switchport nonegotiate
channel-group 3 mode on
!
interface GigabitEthernet1/0/16
switchport trunk native vlan 1001
switchport trunk allowed vlan 2,100
switchport mode trunk
switchport nonegotiate
channel-group 3 mode on
!
interface GigabitEthernet1/0/17
!
interface GigabitEthernet1/0/18
!
interface GigabitEthernet1/0/19
!
interface GigabitEthernet1/0/20
!
interface GigabitEthernet1/0/21
!
interface GigabitEthernet1/0/22
switchport access vlan 100
switchport mode access
switchport nonegotiate
!
interface GigabitEthernet1/0/23
no switchport
no ip address
channel-group 1 mode on
!
interface GigabitEthernet1/0/24
no switchport
no ip address
channel-group 1 mode on
!
interface GigabitEthernet1/1/1
!
interface GigabitEthernet1/1/2
!
interface GigabitEthernet1/1/3
!
interface GigabitEthernet1/1/4
!
interface TenGigabitEthernet1/1/1
!
interface TenGigabitEthernet1/1/2
!
interface TenGigabitEthernet1/1/3
!
interface TenGigabitEthernet1/1/4
!
interface Vlan1
no ip address
!
interface Vlan100
description Management
ip address 10.0.10.254 255.255.255.0
!
router ospf 1
router-id 14.14.14.14
auto-cost reference-bandwidth 100000
passive-interface GigabitEthernet1/0/22
passive-interface Loopback0
network 10.0.6.0 0.0.0.3 area 0
network 10.0.10.0 0.0.0.255 area 0
network 14.14.14.14 0.0.0.0 area 0
network 192.168.1.0 0.0.0.3 area 0
!
ip forward-protocol nd
ip http server
ip http authentication local
ip http secure-server
ip route 0.0.0.0 0.0.0.0 192.168.1.2
!
!
!
ip access-list standard 1
10 permit 10.0.10.22
!
!
!
control-plane
service-policy input system-cpp-policy
!
!
line con 0
exec-timeout 15 0
logging synchronous
login local
stopbits 1
line aux 0
stopbits 1
line vty 0 4
access-class 1 in
exec-timeout 15 0
logging synchronous level 6
login local
transport input ssh
line vty 5 15
access-class 1 in
exec-timeout 15 0
logging synchronous level 6
login local
transport input ssh
!
ntp source Loopback0
ntp master
!
!
!
!
!
!
end
anyone know what could possibly be preventing the WLC from pinging 10.0.3.254?
ps. I have tried using both the vlan 2 interface and vlan 100 interface @ 10.0.3.254/32 on AS1 with option 43 and they both pass DHCP to the ap just fine but the situation is the same, the WLC cannot ping 10.0.3.254/32 or anything on the 10.0.0.0/22 guest network. The only other thing I think I might try from my knowledge is make the option 43 on vlan 2 the management interface 10.0.10.100/24 instead of the logical interface 10.0.0.100/22 but after that I'm all out of ideas. it's late, i'm retiring to work on this tomorrow. thank you in advance for any suggestions.
edit: I have configured vlan2 as a span source port and captured some traffic and it appears the AP is trying to broadcast to an address that is not configured in my diagram although it doesn't appear to be receiving any packets from 10.0.10.100 I will configure span on DS1's 10.0.6.2 interface and use wireshark to see if it's able to give me more information
I did some more packet capture and here is what I found:
I monitored DS1 on vlan 100 and AS1 on 10.0.6.1 po1 they turned up similar results, it appears that when the controller pings 10.0.3.254/32 it simply isn't leaving the controller. The controllers default gateway does not receive any ICMP messages from the controller when it pings 10.0.3.254/32, likewise, it's the same story for po1 10.0.6.1 on AS1 it also doesn't receive any messages from the controller 10.0.10.100/32 when it pings 10.0.3.254/32. So the controller itself isn't capable of pinging 10.0.3.254/32 but it IS capable of pinging 1.1.1.1 on AS1, 10.0.6.1 on AS1, it's default gateway on vlan 100 on DS1. So that would mean something is inherently wrong with the controller correct?
It wouldn't be that I need a vlan 2 interface on DS1 would it? if the default router for vlan 2 is on AS1 vlan 2 interface how would putting a vlan 2 interface on DS1 allow the controller connected to DS1 to reach AS1's vlan 2 interface, that can't be it can it? or am I supposed to set the vlan 2 guests default router to DS1 on the controller because that's what the controller needs and DHCP already handles the guests vlan DHCP default router? I may try this in a bit but i'm honestly exhausted troubleshooting this.
Accepted Solutions
First, thanks for taking the time to reply, I'm greatly appreciative, next for the sake of simplicity I will first go through your questions, line-by-line, then add any relevant information.
Q: I think I can summarise your issue as:
Your 2504 WLC cannot ping IP addresses in remote subnets?
A: Correct, plus some extra details that are relevant, the default-router of the WLC's management interface '10.0.10.100/32', is '10.0.10.254/32' that exists on the logical interface 'vlan 100' in the subnet |10.0.10.0/24| on the switch "DS1" directly connected to the WLC via a trunking LAG which allows vlans 2 & 100. it can ping '10.0.10.254/32', the default-router of the WLC, but not '10.0.10.22/32' which is my Laptop in the same subnet and vlan 100 attached to the same 'DS1' switch where i'm using (The laptop) to connect to the WLC via a serial cable to issue the pings. It can ping '1.1.1.1/32' which is the loopback address of 'AS1' but not '10.0.3.254/32' which is the default gateway for subnet |10.0.0.0/22| located on the logical interface 'vlan2' on 'AS1' which is directly attached to the AP that I'm trying to form a capwap tunnel with back to the WLC. The AP is on an access port connected directly to AS1 that allows vlan2.
Q: On your diagram you show VLAN2 10.0.0.100/22 and VLAN100 10.0.10.100/24
Then at AS1 you show VLAN100 10.0.3.254 which is the same subnet as VLAN2 on the WLC (10.0.0.0/22)?
A: I could've notated it better. Let me reiterate the current status of the network for those subnets and interfaces. The WLC's logical interfaces and their subnets are as follows. '10.0.0.100' on subnet |10.0.0.0/22| & '10.0.10.100' on subnet |10.0.10.0/24| albeit '10.0.10.100' is the management interface which is connected on a trunking LAG allowing vlans 2 & 100 connected to DS1.
Q: Have you got VLAN2 and VLAN100 mixed up?
A1: I tried both interfaces in troubleshooting to see if it was a management interface issue vs a directly connected vlan issue neither worked but the current state is that the logical interface VLAN2 is configured with the ip address '10.0.3.254' on the subnet |10.0.0.0/22| on AS1 in an up-up state connected to the AP.
A2 (possibly relevant): I have configured DHCP option 43 on DS1 for subnet |10.0.0.0/22| to advertise '10.0.10.100' as the WLC's ip address, which is the management interface address of the WLC, because the link between AS1 & DS1 is a layer 3 routed LAG so it is not a layer 2 trunking link.
Q: After you fix that then make sure VLAN100 is allowed on all your trunk ports and configured on all the switches.
A1: I have configured vlans 2 & 100 on both DS1 and AS1. DS1 has logical interface vlan 100 with an ip address of 10.0.10.254 & AS1 has logical interface vlan 2 with an ip address 10.0.3.254. However AS1 does not have an ip address for the logical interface vlan 100 and DS1 does not have an ip address for logical interface vlan2. Should I configure logical interface vlan100 with an ip address on AS1?
A2: I have only one trunking LAG that exists between the WLC and DS1 allowing vlans 2 and 100. The link between DS1 and AS1 is a layer 3 routed LAG where AS1 has an ip address of 10.0.6.1 & DS1 has an ip address of 10.0.6.2 both on subnet 10.0.6.0/30 to advertise OSPF routes which currently route network wide for all devices except the WLC to be able to ping subnets off of AS1 and DS1. Lastly the AP link to AS1 is an access port link allowing vlan 2
____________________________________________________________________________________________________
Here is a simplified & current network diagram showing basic routes and configured ip addresses/vlans, might be easier to read.
Note: The following top section of reading may not be necessary, I think I narrowed it down beneath in this posts' 10/30/2023 EDIT: section. i'm not deleting the top section to keep the notes orderly for future reference in case anyone has similar issues.
__________________________________________________________________________________________
So DS1 can ping the laptop pretty much the only thing that has issues communicating with devices on the network is the WLC.
I did a packet capture, it appears that the packets when pinging 10.0.3.254 on the WLC 'cisco controller > ping 10.0.3.254', showed nothing captured, with span ports set up to monitor vlan 100, the trunk LAG to the WLC and the L3 LAG to AS1. With the same settings, I did 'cisco controller > ping 10.0.10.254' the DS1 default-gateway, and it was not only successful at pinging it, but it also showed up in wireshark. So my only assumption is that for whatever reason it's not leaving the WLC when I ping those problematic addresses. however I will perform some more tests tomorrow with wireshark just to be sure, maybe I filtered out the pings to 10.0.3.254.
I'll take a look at the laptop firewall as well but that wouldn't answer the switch AS1's problem because the WLC can ping its loopback address so a firewall shouldn't be preventing it there at first glance. however I will meticulously check each interfaces settings to see if there's something that may be preventing it.
Also, I went ahead and connected a secondary AP to DS1 on vlan 100 and it immediately associated with the WLC but that's not the intended network design but it does state that the WLC can associate APs just fine if it can ping them. I feel like it's going to be something simple and a little digging is all I need to do.
__________________________________________________________________________________________
10/30/2023 EDIT: I have found some new information. I added 10.0.5.0/24 as a WLAN titled "Servers" and the interface as well in VLAN 20 and option 43 to DS1 in pool VLAN_20 to locate the WLC for the APs in VLAN 20 on AS1.
long story short, the moment I added the interface on the WLC it could no longer ping 10.0.5.254 the default gateway on logical interface VLAN 20 on AS1. But before that, it could, and it was recorded in wireshark.
Demonstration:
after adding the WLC logical interface "Servers" w\ vlan id 20 network 10.0.5.0/24 gateway 10.0.5.254/32
After removing the WLC logical interface "Servers" w\ vlan id 20 network 10.0.5.0/24 gateway 10.0.5.254/32
so it's safe to say the issue is with the logical interfaces on the WLC, at least, it would appear so.
