Solved: asa 5516-x dropping DHCP packets even after ACL allow

Mike Pennycook · ‎09-21-2018

Hi All

Please help me find out why this FTD ASA 5516-X is dropping DHCP packets even after I've allowed it on the ACL:

Phase: 5
Type: ACCESS-LIST
Subtype:
Result: DROP
Config:
Implicit Rule
Additional Information:
Forward Flow based lookup yields rule:
in id=0x2aaada976990, priority=0, domain=permit, deny=true
hits=725, user_data=0xa, cs_id=0x0, use_real_addr, flags=0x1000, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=Inside-CWG_Wifi_SBP, output_ifc=any

Result:
input-interface: Inside-CWG_Wifi_SBP
input-status: up
input-line-status: up
output-interface: NP Identity Ifc
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule

>

393: 16:08:19.513050 802.1Q vlan#554 P0 192.168.55.254.67 > 192.168.55.250.67: udp 300 Drop-reason: (acl-drop) Flow is denied by configured rule
394: 16:08:20.695459 802.1Q vlan#554 P0 192.168.55.254.67 > 192.168.55.250.67: udp 300 Drop-reason: (acl-drop) Flow is denied by configured rule

192.168.55.250 is the DHCP server on this ASA. As the packets are dropped there are no DHCP packets registered:

> show dhcpd statistics
DHCP UDP Unreachable Errors: 0
DHCP Other UDP Errors: 0

Address pools 3
Automatic bindings 0
Expired bindings 0
Malformed messages 0

Message Received
BOOTREQUEST 0
DHCPDISCOVER 0
DHCPREQUEST 0
DHCPDECLINE 0
DHCPRELEASE 0
DHCPINFORM 0

ACL has been allowed for this:

access-list CSM_FW_ACL_ line 27 remark rule-id 268436481: L7 RULE: DHCP
access-list CSM_FW_ACL_ line 28 advanced permit udp ifc Inside any eq bootps any eq bootps rule-id 268436481 (hitcnt=0) 0xe4664fff
access-list CSM_FW_ACL_ line 29 advanced permit udp ifc Inside any eq bootps any eq bootpc rule-id 268436481 (hitcnt=0) 0xe74f89e5
access-list CSM_FW_ACL_ line 30 advanced permit udp ifc Inside any eq bootpc any eq bootps rule-id 268436481 (hitcnt=0) 0x51f2199b
access-list CSM_FW_ACL_ line 31 advanced permit udp ifc Inside any eq bootpc any eq bootpc rule-id 268436481 (hitcnt=0) 0x38d1b7c1
access-list CSM_FW_ACL_ line 32 advanced permit udp ifc Inside-TP-Desktop any eq bootps any eq bootps rule-id 268436481 (hitcnt=0) 0x61419d5e
access-list CSM_FW_ACL_ line 33 advanced permit udp ifc Inside-TP-Desktop any eq bootps any eq bootpc rule-id 268436481 (hitcnt=0) 0xbbd2c408
access-list CSM_FW_ACL_ line 34 advanced permit udp ifc Inside-TP-Desktop any eq bootpc any eq bootps rule-id 268436481 (hitcnt=0) 0xa610180f
access-list CSM_FW_ACL_ line 35 advanced permit udp ifc Inside-TP-Desktop any eq bootpc any eq bootpc rule-id 268436481 (hitcnt=0) 0x5f42843b
access-list CSM_FW_ACL_ line 36 advanced permit udp ifc Inside-CWG_Wifi_SBP any eq bootps any eq bootps rule-id 268436481 (hitcnt=0) 0x5af3a636
access-list CSM_FW_ACL_ line 37 advanced permit udp ifc Inside-CWG_Wifi_SBP any eq bootps any eq bootpc rule-id 268436481 (hitcnt=0) 0xe6b86f1b
access-list CSM_FW_ACL_ line 38 advanced permit udp ifc Inside-CWG_Wifi_SBP any eq bootpc any eq bootps rule-id 268436481 (hitcnt=0) 0xd7403e07
access-list CSM_FW_ACL_ line 39 advanced permit udp ifc Inside-CWG_Wifi_SBP any eq bootpc any eq bootpc rule-id 268436481 (hitcnt=0) 0x677af08d
access-list CSM_FW_ACL_ line 40 advanced permit udp ifc AP_Management any eq bootps any eq bootps rule-id 268436481 (hitcnt=0) 0x1ed16681
access-list CSM_FW_ACL_ line 41 advanced permit udp ifc AP_Management any eq bootps any eq bootpc rule-id 268436481 (hitcnt=0) 0x5826aa69
access-list CSM_FW_ACL_ line 42 advanced permit udp ifc AP_Management any eq bootpc any eq bootps rule-id 268436481 (hitcnt=0) 0xd6d350d0
access-list CSM_FW_ACL_ line 43 advanced permit udp ifc AP_Management any eq bootpc any eq bootpc rule-id 268436481 (hitcnt=0) 0x93785eef

Ajay Saini · ‎09-23-2018

Unfortunately, this design is not supported because of a limitation on ASA. The client must be in same l2 broadcast domain as the ASA interface where the clients are connected. From the ASA guide:

https://www.cisco.com/c/en/us/td/docs/security/asa/asa97/configuration/general/asa-97-general-config/basic-dhcp-ddns.html

You cannot configure a DHCP client or DHCP relay service on an interface on which the server is enabled. Additionally, DHCP clients must be directly connected to the interface on which the server is enabled.

The way it can work is if the ASA acts as L3 gateway and Nexus 9k acts as l2 .

HTH
AJ

View solution in original post

Marius Gunnerud · ‎09-21-2018

Try running the following debug in the FTD CLI to identify which rule it is hitting:

> system support firewall-engine-debug

Please specify an IP protocol: <press enter>
Please specify a client IP address: <IP of client PC>
Please specify a client port: <press enter>
Please specify a server IP address: <IP of DHCP server>
Please specify a server port: <press enter>

--
Please remember to select a correct answer and rate helpful posts

Marvin Rhoads · ‎09-21-2018

In addition to what @Marius Gunnerud correctly suggested, note that ACLs are for traffic THROUGH the firewall - to TO the firewall.

I'd also check that the firewall is listening for DHCP on udp/67 (show asp table sockets) and that it is receiving the DHCP discover packets (via packet capture).

Mike Pennycook · ‎09-22-2018

Hi Marvin,

Thanks -

> show asp table socket stats

TCP Statistics:
Rcvd:
total 0
checksum errors 0
no port 0
Sent:
total 0

UDP Statistics:
Rcvd:
total 0
checksum errors 0
Sent:
total 0
copied 0
Dropped:
Rcv queue full 0

NP SSL System Stats:
Handshake Started: 0
Handshake Complete: 0
SSL Open: 0
SSL Close: 0
SSL Server: 0
SSL Server Verify: 0
SSL Client: 0

>
>
>
> show asp table socket

Protocol Socket State Local Address Foreign Address
>

Packet capture:

476: 07:42:16.831179 802.1Q vlan#554 P0 192.168.55.254.67 > 192.168.55.250.67: udp 0
477: 07:42:44.529422 802.1Q vlan#554 P0 192.168.55.254.67 > 192.168.55.250.67: udp 0
478: 07:44:10.373744 802.1Q vlan#554 P0 192.168.55.254.67 > 192.168.55.250.67: udp 0
479: 07:44:13.255067 802.1Q vlan#554 P0 192.168.55.254.67 > 192.168.55.250.67: udp 0
480: 07:44:15.266480 802.1Q vlan#554 P0 192.168.55.254.67 > 192.168.55.250.67: udp 0
481: 07:44:17.405984 802.1Q vlan#554 P0 192.168.55.254.67 > 192.168.55.250.67: udp 0

192.168.55.250 is the DHCP server interface

Mike Pennycook · ‎09-22-2018

looking at detail packet capture for one of the DHCP packets received on the expected interface:

307: 16:08:40.828738 00fc.baa8.92d1 28ac.9e3b.c954 0x8100 Length: 346
802.1Q vlan#554 P0 192.168.55.254.67 > 192.168.55.250.67: [udp sum ok] udp 300 (ttl 255, id 51328)

separately in drop packet capture:

398: 16:08:40.798832 00fc.baa8.92d1 28ac.9e3b.c954 0x8100 Length: 346
802.1Q vlan#554 P0 192.168.55.254.67 > 192.168.55.250.67: [udp sum ok] udp 300 (ttl 255, id 51072) Drop-reason: (acl-drop) Flow is denied by configured rule

399: 16:08:40.828799 00fc.baa8.92d1 28ac.9e3b.c954 0x8100 Length: 346
802.1Q vlan#554 P0 192.168.55.254.67 > 192.168.55.250.67: [udp sum ok] udp 300 (ttl 255, id 51328) Drop-reason: (acl-drop) Flow is denied by configured rule

400: 16:08:42.456839 00fc.baa8.92d1 28ac.9e3b.c954 0x8100 Length: 346
802.1Q vlan#554 P0 192.168.55.254.67 > 192.168.55.250.67: [udp sum ok] udp 300 (ttl 255, id 51584) Drop-reason: (acl-drop) Flow is denied by configured rule

Mike Pennycook · ‎09-22-2018

Thanks for the suggestion,

Unfortunately I’m not on site to generate DHCP packets but the packet tracer output says it’s denied by an implicit rule:

Phase: 5
Type: ACCESS-LIST
Subtype:
Result: DROP
Config:
Implicit Rule <<<<<<<

Additionally packet tracer shows:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x2aaada976990, priority=0, domain=permit, deny=true

is that id the ACL id?

Ajay Saini · ‎09-23-2018

Hello,

If I understood it correctly, its a dhcp relay scenario. If yes, then packet-tracer won't be a good idea since the dhcp discover (broadcast packet) will be intercepted by ASA and send to DHCP server as a unicast changing the headers. Can you please provide the dhcp relay configuration that you have done.

Or is it being already handled by another dhcp relay agent and sent as a unicast packet to ASA and getting dropped. If this is a case, please confirm the setup you have.

You can run some debugs and see what is happening to packets.

HTH

AJ

Mike Pennycook · ‎09-23-2018

Hi Ajay,

Good point.

The DHCP relay config is on a Nexus 9K, with one physical interface trunking towards the ASA 5516-X. The ASA has 3 subinterfaces each with a DHCP server configured.

On the 9K SVI (default gateway to clients):

interface Vlan554

ip address 192.168.55.254/23
ip dhcp relay address 192.168.55.250

Physical interface is configured as a trunk and allowing the above VLAN:

On ASA:

GigabitEthernet1/2.554 Inside-CWG_Wifi_SBP 192.168.55.250 255.255.254.0 CONFIG

!
dhcpd address 192.168.54.10-192.168.55.9 Inside-CWG_Wifi_SBP
dhcpd enable Inside-CWG_Wifi_SBP

Earlier on the N9K interface VLAN 554 i tried this:

interface Vlan554

ip address dhcp

And this got an IP! So DHCP is working on the ASA but DHCP relay is not working on the N9K

So when the N9K interface i set to receive an address via DHCP:

> show dhcpd binding

IP address Client Identifier Lease expiration Type

192.168.54.11 0046.444f.3232.3039. 3592 seconds Automatic
3234.5638.566c.616e.
3535.34
> show dhcpd statistics
DHCP UDP Unreachable Errors: 0
DHCP Other UDP Errors: 0

Address pools 3
Automatic bindings 1
Expired bindings 0
Malformed messages 0

Message Received
BOOTREQUEST 0
DHCPDISCOVER 2
DHCPREQUEST 2
DHCPDECLINE 0
DHCPRELEASE 1
DHCPINFORM 0

Message Sent
BOOTREPLY 0
DHCPOFFER 2
DHCPACK 2
DHCPNAK 0

597: 21:08:37.082347 802.1Q vlan#554 P0 0.0.0.0.68 > 255.255.255.255.67: udp 314
598: 21:08:37.082576 802.1Q vlan#554 P0 arp who-has 192.168.54.11 tell 192.168.55.250
599: 21:08:37.180135 802.1Q vlan#554 P0 192.168.55.250.67 > 255.255.255.255.68: udp 290
600: 21:08:38.230029 802.1Q vlan#554 P0 arp who-has 192.168.54.11 tell 192.168.55.250
601: 21:08:39.230044 802.1Q vlan#554 P0 arp who-has 192.168.54.11 tell 192.168.55.250
602: 21:08:43.085780 802.1Q vlan#554 P0 0.0.0.0.68 > 255.255.255.255.67: udp 326
603: 21:08:43.085963 802.1Q vlan#554 P0 192.168.55.250.67 > 255.255.255.255.68: udp 290
604: 21:08:43.230044 802.1Q vlan#554 P0 arp who-has 192.168.54.11 tell 192.168.55.250
605: 21:08:43.471395 802.1Q vlan#554 P6 arp who-has 192.168.54.11 (ff:ff:ff:ff:ff:ff) tell 192.168.54.11
606: 21:08:43.551652 802.1Q vlan#554 P6 arp who-has 192.168.55.250 (ff:ff:ff:ff:ff:ff) tell 192.168.54.11
607: 21:08:43.551775 802.1Q vlan#554 P6 arp reply 192.168.55.250 is-at 28:ac:9e:3b:c9:54
608: 21:08:48.230060 802.1Q vlan#554 P0 arp who-has 192.168.54.11 tell 192.168.55.250
609: 21:08:48.230578 802.1Q vlan#554 P6 arp reply 192.168.54.11 is-at 0:fc:ba:a8:92:d1
610: 21:08:48.230838 802.1Q vlan#554 P0 192.168.55.250 > 192.168.54.11: icmp: echo request
611: 21:08:48.231112 802.1Q vlan#554 P0 192.168.54.11 > 192.168.55.250: icmp: echo reply

The captures in earlier post show when the dhcp relay message gets dropped: