Cannot ping device on same VLAN on different switch after DACL Update

[blake2324]

Hello,

We have a set of IoT devices that ping a controller on the local LAN every ~30 seconds to ensure the link is active. If the link is broken, the associated desktop software stops working. The devices and controller share a DACL, and the current DACL applied is permit ip any any. For security purposes, we are restricting access for these devices, and the new DACL has the following syntax in regards to icmp.

permit icmp any 10.0.0.0 0.255.255.255
permit icmp any 172.16.0.0 0.15.255.255
permit icmp any 192.168.0.0 0.0.255.255

The device and the controller are placed in the same VLAN as part of ISE profiling and authorization, but they are physically connected on two different switches in the building that are connected via a trunk port.

When the test DACL is applied, the devices are unable to ping. I have tried using permit icmp any any as the syntax with the same result. In other facilities where the device and controller are connected to the same switch, the test DACL works without issue.

Switch 1 access interface config

description tcr
switchport access vlan 103
switchport mode access
switchport nonegotiate
switchport voice vlan 104
ip device tracking maximum 65535
ip access-group ACL_DEFAULT_ISE_Lock in
srr-queue bandwidth share 1 30 35 5
priority-queue out
no cdp enable
ipv6 traffic-filter Block_ALLv6 in
authentication control-direction in
authentication event fail action next-method
authentication event server dead action reinitialize vlan 103
authentication event server dead action authorize voice
authentication event server alive action reinitialize
authentication host-mode multi-auth
authentication order dot1x mab
authentication priority dot1x mab
authentication port-control auto
authentication periodic
authentication timer reauthenticate server
authentication timer inactivity 3600
authentication timer unauthorized 3600
authentication violation restrict
mab
mls qos trust cos
dot1x pae authenticator
dot1x timeout tx-period 10
auto qos trust
spanning-tree portfast edge
end

Switch 1 trunk interface config

switchport mode trunk
ip dhcp relay information trusted
ip device tracking maximum 0
ipv6 traffic-filter Block_ALLv6 in
mls qos trust dscp
ip dhcp snooping trust
end

Switch 2 access interface config

description TCR
switchport access vlan 101
switchport mode access
switchport nonegotiate
switchport voice vlan 104
ip device tracking maximum 65535
ip access-group ACL_DEFAULT_ISE_Lock in
srr-queue bandwidth share 1 30 35 5
priority-queue out
no cdp enable
ipv6 traffic-filter Block_ALLv6 in
authentication control-direction in
authentication event fail action next-method
authentication event server dead action reinitialize vlan 101
authentication event server dead action authorize voice
authentication event server alive action reinitialize
authentication host-mode multi-auth
authentication order dot1x mab
authentication priority dot1x mab
authentication port-control auto
authentication periodic
authentication timer reauthenticate server
authentication timer inactivity 3600
authentication timer unauthorized 3600
authentication violation restrict
mab
mls qos trust cos
dot1x pae authenticator
dot1x timeout tx-period 10
auto qos trust
spanning-tree portfast edge
end

Switch 2 trunk interface config

interface GigabitEthernet1/0/24
switchport mode trunk
ip dhcp relay information trusted
ip device tracking maximum 0
ipv6 traffic-filter Block_ALLv6 in
mls qos trust dscp
ip dhcp snooping trust
end

Any ideas of what I could be missing in my DACL/things to try? Thanks in advance!

[Arne Bier]

Why does switch 1 access have a data VLAN 103 and switch 1 access switch has a data VLAN 101 ? If there is a trunk between switch 1 and switch 2 then that means these two access switches are not on the same VLAN.

When you say, dACL is applied, where is this dACL definition?  Is that the stuff below?

permit icmp any 10.0.0.0 0.255.255.255
permit icmp any 172.16.0.0 0.15.255.255
permit icmp any 192.168.0.0 0.0.255.255

Because this command below is a pre-auth ACL. It's always active, unless you override it with a dACL from RADIUS server

ip access-group ACL_DEFAULT_ISE_Lock in

What is the config for ACL_DEFAULT_ISE_Lock  ?

Are you 100% sure that the dACL is applied when the endpoints are authorized by ISE?

show access-session int X/Y/Z detail

And then look for the ACL name and check its contents with this command (the ACL has a dynamic name)

show ip access-list xxxxxxx

[blake2324]

Hi Arnie,

Thanks for the response. I should have stated in my original post, those access configurations are for the specific ports the devices are connected to. The VLAN is overridden by ISE when the device authorizes.

Yes sir, when I say DACL applied that is the stuff below. The definition comes from ISE, we're overriding the pre-auth ACL with it.

Here's the config for ACL_DEFAULT_ISE_Lock

show ip access-list ACL_DEFAULT_ISE_Lock
Extended IP access list ACL_DEFAULT_ISE_Lock
10 permit udp any eq bootpc any eq bootps
20 permit udp any any eq domain
30 permit tcp any host 10.16.128.163 eq 8443
40 permit tcp any host 10.17.128.163 eq 8443
50 permit tcp any host 10.16.128.163 eq 8905
60 permit tcp any host 10.17.128.163 eq 8905
70 deny ip any any (6 matches)

Here's the details of the auth session on the port on switch 1 currently with the production DACL applied

Interface: GigabitEthernet1/0/2
MAC Address: 00a0.585b.c255
IPv6 Address: Unknown
IPv4 Address: 10.1.50.160
User-Name: 00-A0-58-5B-C2-55
Status: Authorized
Domain: DATA
Oper host mode: multi-auth
Oper control dir: in
Session timeout: N/A
Restart timeout: N/A
Periodic Acct timeout: 172800s (local), Remaining: 77461s
Session Uptime: 94029s
Common Session ID: 0A09320900017A13F078C250
Acct Session ID: 0x000575F2
Handle: 0xF9000285
Current Policy: POLICY_Gi1/0/2

Local Policies:
Idle timeout: 3600 sec
Service Template: DEFAULT_LINKSEC_POLICY_SHOULD_SECURE (priority 150)

Server Policies:
Vlan Group: Vlan: 103
ACS ACL: xACSACLx-IP-PERMIT_CASH_HANDLER-5acfbfad

Method status list:
Method State

dot1x Stopped
mab Authc Success

And then checking the contents of the ACL

show ip access-list xACSACLx-IP-PERMIT_CASH_HANDLER-5acfbfad
Extended IP access list xACSACLx-IP-PERMIT_CASH_HANDLER-5acfbfad (per-user)
1 permit ip any any

Thank you sir again.

[Arne Bier]

Thanks - that's clearer now.  Do the endpoints get their IPv4 addresses via DHCP?  If so, I wonder if the DHCP is happening in the pre-auth mode (on vlan X), before ISE has a chance to switch the VLAN to 103. If that is the case, then the endpoint has no idea the VLAN has been switched.  And with MAB (which is what is happening in your case), dynamic VLAN assignment && DHCP is a deadly/bad combination - it probably will always break if the VLAN is changed from under the endpoint's feet. Dynamic VLAN assignment only works well with 802.1X where the VLAN switch happens BEFORE L3 (DHCP). And Windows has special support built in to handle that for user auth (e.g. user logging in, supports dynamic VLAN assignment - the IP stack re-initialises itself if you check a special box in the supplicant config).

Just for a laugh, can you make the VLAN 103 static on both interfaces, and then try again. If this works, then it proves my theory.

[blake2324]

Hi Arnie,

The endpoints do get their IP from DHCP. We did try making the VLAN 103 static on both interfaces with the same issue. I also moved both devices to the same switch with VLAN 103 static and still had the issue. I am wondering if there is a dependency for this particular model that isn't clearly outlined in the documentation I was provided, going to make some changes to the DACL and see if I can narrow it down, and if necessary, make a trip to the facility to run some packet captures.

Thanks!