Re: DHCP IP renewal in sd-access

Aleksandr Serov · ‎02-09-2022

Hello!

Please help to troubleshoot the case with dhcp lease renewal in sd-access.
According to RFC 2131 Section 4.3.2 (DHCPREQUEST generated during RENEWING state), when the client wants to extend their lease, it sends unicast DHCPREQUEST with their own IP as a source and DHCP server IP as a destination. The nearest Edge router add Option 82 and relay that request to the sever keeping client IP address as a source.
The server respond to that request with DHCPACK directly to the client IP address. But this DHCPACK not reaching the client. It is looks like the Border router drops these DHCPACK

DHCPACK are reaching the clients only when the DHCPREQUEST was broadcasted and the server respond to relays SVI IP address. In these well-documented cases the border router forwards DHCPACK to the appropriate Edge according to option 82 and ACK reaching the clients

Please advise how to troubleshoot it?

Thank you in advance

Flavio Miranda · ‎02-09-2022

Hi

It is not clear what is your problem. What would you mind with DHCP renew? I care about client getting an IP only.

Aleksandr Serov · ‎02-09-2022

Hi

The problem is that the client can't renew their lease. It do not get any ACK from server.

According to the RFC it should renew their IP before the lease expiration. After expiration it starts getting the IP from the scratch, it broadcasts the request to server and sometimes getting the new address.

My SD-access has no problem with broadcast DHCPREQUEST, only with unicast and it broke the normal client behaviour.

Flavio Miranda · ‎02-09-2022

Hi

That´s weird. What I suggest is try to map where this communication is broken. Which IPAM do you use? Is it Infoblox?

One thing you can do is increase DHCP lease so that clients dont need to ask a new one very often. But, if for some reason, let´s say security you need to keep renewing clients DHCP, try to map this flow. As you mentioned you are not getting any ACK from server, that´s a good start to investigate.

Maybe the server is reply the ACK but it is not getting to the client for some reason.

Aleksandr Serov · ‎02-10-2022

Hi
It is ISC DHCP server. It is working well. It respond on all requests with keeping option 82.
My question was how to find and troubleshoot such cases. There are a lot of documentation how to troubleshoot edge switches and no information how to troubleshoot this on the border switch.

I checked that the server respond with DHCPACK on unicast DHCPREQUEST.
I also checked that these DHCPACK do not reach the edge switch by debugging dhcp snooping.
So the packets may be dropped by border or in ingress of edge.
Is packet capturing only way to distinkt it?
Are there any way to configure the filter for "monitor capture" to intercept only DHCP incapsulated in LISP?

Flavio Miranda · ‎02-10-2022

Sure there is but is not necessary. First, install a wireshark on the client machine and make sure the response is reaching the device. I mean, when the DHCPACK is sent from the server, make sure it is getting to the client. Maybe, this is a client problem and not a network problem.

But, lets say you dont see anything arriving on the client, then, start investigating on the device immediatly after the client, which I assume is the switch. You can use the same machine with wireshark on it, mirror a switch port and look the logs. Make sure the response is arriving in the switch. And so on and so forth.

If you garantee that your DHCP server is worknig fine, but your client is not getting what they need, you need to investigate step by step on the path the packet does.

Aleksandr Serov · ‎02-10-2022

Thank you Flavio. I did exactly that way you are describing. I now for sure that server is responding with DCHPACK and these packets disappears somewhere among the border and the edge on their way to client.

And It is my question, how to find where and why the border or the edge are dropping these DHCPACKs.

Flavio Miranda · ‎02-10-2022

Well, I was trying to make sure all the basic step on troubleshooting this kind of problem was taken place. But, it seems you have some more specific problem.

Take a look on this document. It describes detailed the process of dhcp in SDA environment and might help. Further than that, I´d go with cisco TAC.

Parthiv Shah · ‎03-19-2022

If you are using latest version of iOS-xe you can check using “show platform dhcp <>” command and it will display stats for the client MAC address. You can use this command on both border and edge to get the information where it might be dropping.

Johannes_Grimm1 · ‎04-30-2022

Hello all,

we have a similar problem in our SD-Access fabric. The packet arrives at the edge node, but is discarded before it arrives at the client.

10.10.10.50 (Infoblox)

10.10.10.51 (Infoblox)
192.168.1.20 (Windows Client)

Switch Version 17.6.2; Model Number : C9407R

This is the behavior when i try to renew the IP Adress on the windows client with "ipconfig /renew"

Switch#show platform dhcpsnooping client stats 98fa.bbcc.4601
DHCPSN: DHCP snooping server
DHCPD: DHCP protocol daemen
L2FWD: Transmit Packet to driver in L2 format
FWD: Transmit Packet to driver
<MessageType>(B): Dhcp message's response expected as 'B'roadcast
<MessageType>(U): Dhcp message's response expected as 'U'nicast
Packet Trace for client MAC 98FA.bbcc.4601:
Timestamp                Destination MAC  Destination Ip  VLAN  Message      Handler:Action
------------------------ ---------------- --------------- ----  ------------ ----------------
2022/04/30 13:18:16.427  0000.0C9F.F22E   10.10.10.50     1037 DHCPREQUEST(U)  PUNT:RECEIVED
2022/04/30 13:18:16.427  0000.0C9F.F22E   10.10.10.50     1037 DHCPREQUEST(U)  PUNT:TO_DHCPSN
2022/04/30 13:18:16.427  0000.0C9F.F22E   10.10.10.50     1037 DHCPREQUEST(U)  BRIDGE:RECEIVED
2022/04/30 13:18:16.427  0000.0C9F.F22E   10.10.10.50     1037 DHCPREQUEST(U)  BRIDGE:TO_DHCPD
2022/04/30 13:18:16.427  0000.0C9F.F22E   10.10.10.50     1037 DHCPREQUEST(U)  INJECT:RECEIVED
2022/04/30 13:18:16.427  0000.0C9F.F22E   10.10.10.50     1037 DHCPREQUEST(U)  INJECT:TO_L2FWD
2022/04/30 13:18:16.432  FFFF.FFFF.FFFF   192.168.1.20    0    DHCPACK(U)      PUNT:RECEIVED
2022/04/30 13:18:16.432  FFFF.FFFF.FFFF   192.168.1.20    0    DHCPACK(U)      PUNT:TO_DHCPD
2022/04/30 13:18:19.427  0000.0C9F.F22E   10.10.10.50     1037 DHCPREQUEST(U)  PUNT:RECEIVED
2022/04/30 13:18:19.427  0000.0C9F.F22E   10.10.10.50     1037 DHCPREQUEST(U)  PUNT:TO_DHCPSN
2022/04/30 13:18:19.427  0000.0C9F.F22E   10.10.10.50     1037 DHCPREQUEST(U)  BRIDGE:RECEIVED
2022/04/30 13:18:19.427  0000.0C9F.F22E   10.10.10.50     1037 DHCPREQUEST(U)  BRIDGE:TO_DHCPD
2022/04/30 13:18:19.428  0000.0C9F.F22E   10.10.10.50     1037 DHCPREQUEST(U)  INJECT:RECEIVED
2022/04/30 13:18:19.428  0000.0C9F.F22E   10.10.10.50     1037 DHCPREQUEST(U)  INJECT:TO_L2FWD
2022/04/30 13:18:19.430  FFFF.FFFF.FFFF   192.168.1.20    0    DHCPACK(U)      PUNT:RECEIVED
2022/04/30 13:18:19.430  FFFF.FFFF.FFFF   192.168.1.20    0    DHCPACK(U)      PUNT:TO_DHCPD
2022/04/30 13:18:22.428  0000.0C9F.F22E   10.10.10.50     1037 DHCPREQUEST(U)  PUNT:RECEIVED
2022/04/30 13:18:22.428  0000.0C9F.F22E   10.10.10.50     1037 DHCPREQUEST(U)  PUNT:TO_DHCPSN
2022/04/30 13:18:22.428  0000.0C9F.F22E   10.10.10.50     1037 DHCPREQUEST(U)  BRIDGE:RECEIVED
2022/04/30 13:18:22.428  0000.0C9F.F22E   10.10.10.50     1037 DHCPREQUEST(U)  BRIDGE:TO_DHCPD
2022/04/30 13:18:22.428  0000.0C9F.F22E   10.10.10.50     1037 DHCPREQUEST(U)  INJECT:RECEIVED
2022/04/30 13:18:22.428  0000.0C9F.F22E   10.10.10.50     1037 DHCPREQUEST(U)  INJECT:TO_L2FWD
2022/04/30 13:18:22.431  FFFF.FFFF.FFFF   192.168.1.20    0    DHCPACK(U)      PUNT:RECEIVED
2022/04/30 13:18:22.431  FFFF.FFFF.FFFF   192.168.1.20    0    DHCPACK(U)      PUNT:TO_DHCPD
2022/04/30 13:18:26.428  0000.0C9F.F22E   10.10.10.50     1037 DHCPREQUEST(U)  PUNT:RECEIVED
2022/04/30 13:18:26.428  0000.0C9F.F22E   10.10.10.50     1037 DHCPREQUEST(U)  PUNT:TO_DHCPSN
2022/04/30 13:18:26.429  0000.0C9F.F22E   10.10.10.50     1037 DHCPREQUEST(U)  BRIDGE:RECEIVED
2022/04/30 13:18:26.429  0000.0C9F.F22E   10.10.10.50     1037 DHCPREQUEST(U)  BRIDGE:TO_DHCPD
2022/04/30 13:18:26.429  0000.0C9F.F22E   10.10.10.50     1037 DHCPREQUEST(U)  INJECT:RECEIVED
2022/04/30 13:18:26.429  0000.0C9F.F22E   10.10.10.50     1037 DHCPREQUEST(U)  INJECT:TO_L2FWD
2022/04/30 13:18:26.431  FFFF.FFFF.FFFF   192.168.1.20    0    DHCPACK(U)      PUNT:RECEIVED
2022/04/30 13:18:26.431  FFFF.FFFF.FFFF   192.168.1.20    0    DHCPACK(U)      PUNT:TO_DHCPD

When I release my address with "ipconfig /release" and "ipconfig /renew" the DHCP process works as expected:

Switch#show platform dhcpsnooping client stats 98fa.bbcc.4601
DHCPSN: DHCP snooping server
DHCPD: DHCP protocol daemen
L2FWD: Transmit Packet to driver in L2 format
FWD: Transmit Packet to driver
<MessageType>(B): Dhcp message's response expected as 'B'roadcast
<MessageType>(U): Dhcp message's response expected as 'U'nicast
Packet Trace for client MAC 98fa.bbcc.4601:
Timestamp                Destination MAC  Destination Ip  VLAN  Message      Handler:Action
------------------------ ---------------- --------------- ----  ------------ ----------------
2022/04/30 13:19:48.761  0000.0C9F.F22E   10.10.10.50     1037 DHCPRELEASE     PUNT:RECEIVED
2022/04/30 13:19:48.761  0000.0C9F.F22E   10.10.10.50     1037 DHCPRELEASE     PUNT:TO_DHCPSN
2022/04/30 13:19:48.762  0000.0C9F.F22E   10.10.10.50     1037 DHCPRELEASE     BRIDGE:RECEIVED
2022/04/30 13:19:48.762  0000.0C9F.F22E   10.10.10.50     1037 DHCPRELEASE     BRIDGE:TO_DHCPD
2022/04/30 13:19:48.762  0000.0C9F.F22E   10.10.10.50     1037 DHCPRELEASE     INJECT:RECEIVED
2022/04/30 13:19:48.763  0000.0C9F.F22E   10.10.10.50     1037 DHCPRELEASE     INJECT:TO_L2FWD
2022/04/30 13:19:51.468  FFFF.FFFF.FFFF   255.255.255.255 1037 DHCPDISCOVER(U) PUNT:RECEIVED
2022/04/30 13:19:51.468  FFFF.FFFF.FFFF   255.255.255.255 1037 DHCPDISCOVER(U) PUNT:TO_DHCPSN
2022/04/30 13:19:51.468  FFFF.FFFF.FFFF   255.255.255.255 1037 DHCPDISCOVER(U) BRIDGE:RECEIVED
2022/04/30 13:19:51.468  FFFF.FFFF.FFFF   255.255.255.255 1037 DHCPDISCOVER(U) BRIDGE:TO_DHCPD
2022/04/30 13:19:51.468  FFFF.FFFF.FFFF   255.255.255.255 1037 DHCPDISCOVER(U) BRIDGE:TO_INJECT
2022/04/30 13:19:51.468  FFFF.FFFF.FFFF   255.255.255.255 1037 DHCPDISCOVER(U) L2INJECT:TO_FWD
2022/04/30 13:19:51.469  7C31.0EC7.189A   10.10.10.51     0    DHCPDISCOVER(U) INJECT:RECEIVED
2022/04/30 13:19:51.469  7C31.0EC7.189A   10.10.10.51     0    DHCPDISCOVER(U) INJECT:TO_L2FWD
2022/04/30 13:19:51.469  7C31.0EC7.189A   10.10.10.50     0    DHCPDISCOVER(U) INJECT:RECEIVED
2022/04/30 13:19:51.469  7C31.0EC7.189A   10.10.10.50     0    DHCPDISCOVER(U) INJECT:TO_L2FWD
2022/04/30 13:19:52.474  FFFF.FFFF.FFFF   172.31.32.1     1037 DHCPOFFER(U)    PUNT:RECEIVED
2022/04/30 13:19:52.474  7C31.0EC7.189A   192.168.1.20    0    DHCPOFFER(U)    INJECT:RECEIVED
2022/04/30 13:19:52.474  98fa.bbcc.4601   192.168.1.20    0    DHCPOFFER(U)    INTERCEPT:RECEIVED
2022/04/30 13:19:52.474  98fa.bbcc.4601   192.168.1.20    1037 DHCPOFFER(U)    INTERCEPT:TO_DHCPSN
2022/04/30 13:19:52.476  FFFF.FFFF.FFFF   255.255.255.255 1037 DHCPREQUEST(U)  PUNT:RECEIVED
2022/04/30 13:19:52.476  FFFF.FFFF.FFFF   255.255.255.255 1037 DHCPREQUEST(U)  PUNT:TO_DHCPSN
2022/04/30 13:19:52.476  FFFF.FFFF.FFFF   255.255.255.255 1037 DHCPREQUEST(U)  BRIDGE:RECEIVED
2022/04/30 13:19:52.476  FFFF.FFFF.FFFF   255.255.255.255 1037 DHCPREQUEST(U)  BRIDGE:TO_DHCPD
2022/04/30 13:19:52.476  FFFF.FFFF.FFFF   255.255.255.255 1037 DHCPREQUEST(U)  BRIDGE:TO_INJECT
2022/04/30 13:19:52.476  FFFF.FFFF.FFFF   255.255.255.255 1037 DHCPREQUEST(U)  L2INJECT:TO_FWD
2022/04/30 13:19:52.476  7C31.0EC7.189A   10.10.10.51     0    DHCPREQUEST(U)  INJECT:RECEIVED
2022/04/30 13:19:52.476  7C31.0EC7.189A   10.10.10.51     0    DHCPREQUEST(U)  INJECT:TO_L2FWD
2022/04/30 13:19:52.476  7C31.0EC7.189A   10.10.10.50     0    DHCPREQUEST(U)  INJECT:RECEIVED
2022/04/30 13:19:52.476  7C31.0EC7.189A   10.10.10.50     0    DHCPREQUEST(U)  INJECT:TO_L2FWD
2022/04/30 13:19:52.480  FFFF.FFFF.FFFF   172.31.32.1     1037 DHCPACK(U)      PUNT:RECEIVED
2022/04/30 13:19:52.480  7C31.0EC7.189A   192.168.1.20    0    DHCPACK(U)      INJECT:RECEIVED
2022/04/30 13:19:52.480  98fa.bbcc.4601   192.168.1.20    0    DHCPACK(U)      INTERCEPT:RECEIVED
2022/04/30 13:19:52.480  98fa.bbcc.4601   192.168.1.20    1037 DHCPACK(U)      INTERCEPT:TO_DHCPSN

Has anyone found a solution for this yet?

jalejand · ‎04-30-2022

Your problem seems that during the origina, 4way DORA transaction, the DHCP Offer and ACKs are sent from the server to the SVI of the relay agent (the FE):

2022/04/30 13:19:52.474  FFFF.FFFF.FFFF   172.31.32.1     1037 DHCPOFFER(U)    PUNT:RECEIVED

2022/04/30 13:19:52.480  FFFF.FFFF.FFFF   172.31.32.1     1037 DHCPACK(U)      PUNT:RECEIVED

When doing the release/renew procedure, your DHCP server sends the ACK directly to the IP of the client:

2022/04/30 13:18:16.432  FFFF.FFFF.FFFF   192.168.1.20    0    DHCPACK(U)      PUNT:RECEIVED

By definition in SDA, DHCP packets are processed by the "For US" queue, in this case, the destination IP of the packet is not the anycast gateway but the Client IP, hence the packet is never processed by the DHCP Snooping CPU queue. If you find a way to configure your DHCP server to send DHCP ACKs for Refresh/Lease to the SVI rather than the client, this will work. In short, DHCP packets coming from VXLAN, sent to anything except the SVI of the FE will be ultimately dropped.

Regards

Johannes_Grimm1 · ‎05-02-2022

Hello Jalejand,

thank you very much for the explanation. We learned today that there is an SMU for this use case in version 17.6.2. The behavior that the unicast DHCP requests are discarded has been removed. I have already successfully tested the SMU in our environment.

Software Download - Cisco Systems