2960S ip helper address weirdness? bug?

[brettp]

Can someone please explain this to me... I have never been so confused. Over the weekend, I replaced a plain old layer 2 Cisco 2960-S switch. I came in today to find many computer not working because they couldn't get an IP address via DHCP. The setup is like this:

DHCP server (192.168.2.1) --> 6509 (gateway 192.168.1.254 w/ helper address 192.168.2.1) --> 2960S (with svi for remote access 192.168.1.250) --> client

As noted, ip helper address is set up in the 6509/router 192.168.1.0/24 vlan interface. DHCP was not working until I added the helper address to the plain old layer 2 2960S switch too. Why? Why would the help address need to be added to the 2960S switch? No other switch on our entire network is set up in that manner. Is this a weird bug or something? Any info is greatly appreciated!

[Peter Paluch]

Hi Brett,

Curious indeed!

I wonder: Is the 2960-S by any chance configured with DHCP Snooping? Ideally, would it be possible to share its configuration here?

If the 2960-S was configured with DHCP Snooping, it would likely insert the DHCP Option-82 into the relayed messages, and if the Cat6509 is not configured specifically to accept DHCP client packets with Option-82, it would have dropped them instead of relaying them. By configuring the DHCP Relay Agent on the 2960-S, you would bypass this since the 2960-S would be relaying the DHCP packets itself.

That's my best shot for the moment.

Best regards,
Peter

[brettp]

Hate to make this more confusing, but no, DHCP Snooping is not enabled. I've pasted a condensed / sanitized config below (no, we're not really using vlan 1, I changed some stuff in the config pasted below to clean it up.) As you can see, there's nothing amazing going on. Unfortunately, I can't run wireshark or anything because this is working now with and I can't go back and break it (right now at least.) Thanks!

!
version 15.0
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname Switch
!
boot-start-marker
boot-end-marker
!
enable secret 5 *****
!
username ***** secret 5 *****
aaa new-model
!
!
aaa authentication login default group radius local
!
aaa session-id common
switch 1 provision ws-c2960s-48ts-l
switch 2 provision ws-c2960s-48ts-l
!
!
ip domain-name *****
vtp mode off
!
mls qos map cos-dscp 0 8 16 24 32 46 48 56
mls qos srr-queue output cos-map queue 1 threshold 3 4 5
mls qos srr-queue output cos-map queue 2 threshold 1 2
mls qos srr-queue output cos-map queue 2 threshold 2 3
mls qos srr-queue output cos-map queue 2 threshold 3 6 7
mls qos srr-queue output cos-map queue 3 threshold 3 0
mls qos srr-queue output cos-map queue 4 threshold 3 1
mls qos srr-queue output dscp-map queue 1 threshold 3 32 33 40 41 42 43 44 45
mls qos srr-queue output dscp-map queue 1 threshold 3 46 47
mls qos srr-queue output dscp-map queue 2 threshold 1 16 17 18 19 20 21 22 23
mls qos srr-queue output dscp-map queue 2 threshold 1 26 27 28 29 30 31 34 35
mls qos srr-queue output dscp-map queue 2 threshold 1 36 37 38 39
mls qos srr-queue output dscp-map queue 2 threshold 2 24
mls qos srr-queue output dscp-map queue 2 threshold 3 48 49 50 51 52 53 54 55
mls qos srr-queue output dscp-map queue 2 threshold 3 56 57 58 59 60 61 62 63
mls qos srr-queue output dscp-map queue 3 threshold 3 0 1 2 3 4 5 6 7
mls qos srr-queue output dscp-map queue 4 threshold 1 8 9 11 13 15
mls qos srr-queue output dscp-map queue 4 threshold 2 10 12 14
mls qos queue-set output 1 threshold 1 100 100 50 200
mls qos queue-set output 1 threshold 2 125 125 100 400
mls qos queue-set output 1 threshold 3 100 100 100 400
mls qos queue-set output 1 threshold 4 60 150 50 200
mls qos queue-set output 1 buffers 15 25 40 20
mls qos
!
crypto pki trustpoint TP-self-signed-12341234123
enrollment selfsigned
subject-name cn=IOS-Self-Signed-Certificate-12341234123
revocation-check none
rsakeypair TP-self-signed-2197549952
!
!
crypto pki certificate chain TP-self-signed-12341234123
certificate self-signed 01
***** ***** ***** *****
quit
spanning-tree mode pvst
spanning-tree extend system-id
!
vlan internal allocation policy ascending
!
vlan 1
!
interface FastEthernet0
no ip address
shutdown
!
interface GigabitEthernet1/0/1
description client access
switchport access vlan 1
switchport mode access
srr-queue bandwidth share 1 30 35 5
priority-queue out
mls qos trust dscp
auto qos trust
no cdp enable
spanning-tree portfast
spanning-tree bpduguard enable
!
interface GigabitEthernet1/0/52
description uplink to 6509
switchport access vlan 1
switchport mode access
srr-queue bandwidth share 1 30 35 5
priority-queue out
mls qos trust dscp
!
interface GigabitEthernet2/0/1
description client access
switchport access vlan 1
switchport mode access
srr-queue bandwidth share 1 30 35 5
priority-queue out
mls qos trust dscp
no cdp enable
spanning-tree portfast
spanning-tree bpduguard enable
!
interface Vlan1
ip address 192.168.1.250 255.255.255.0
ip helper-address 192.168.2.1 <-- NEEDED TO ADD THIS FOR DHCP TO WORK
!
ip default-gateway 192.168.1.254
ip http server
ip http secure-server
!
logging facility local2
logging host *****
!
snmp-server community ***** RO
!
radius-server host ***** key 7 *****
!
!
vstack
!
line con 0
line vty 0 4
length 0
transport input ssh
line vty 5 15
transport input ssh
!
ntp server *****
end

[Peter Paluch]

Hi Brett,

Thank you very much for your response!

Hmmm... Getting more and more convoluted, to be sure.

Absolutely, the 2960-S config is very straightforward, nothing fancy, and certainly nothing that would be relevant to DHCP operations. Besides DHCP Snooping, I was also looking whether perhaps blocking of broadcasts was enabled on the switchports, but there is nothing of the sort.

From the behavior, I can only say this: Either the client's DHCP packets sent out as broadcasts are not received at all by the 6509, or they are received on it but ignored/dropped for whatever reason. Your addition of the "ip helper-address" on the Vlan1 SVI of your 2960-S helped because you have effectively created a new DHCP Relay Agent that can obviously hear the clients' broadcasts, and have them relayed to the DHCP server. However, this is truly a workaround only.

There is one thing you can try doing on the 6509 without breaking the DHCP now: Try running "debug ip dhcp server packet" and "debug ip dhcp server event" on the 6509, then have a DHCP client entirely forget its address ("ipconfig /release") and ask for it again ("ipconfig /renew"); the release operation is needed, otherwise Windows will refresh the lease via unicast to the DHCP server directly. This debug should still show us that the DHCP client broadcasts are hitting the 6509, and should also tell us what's happening over there, even if you have the "ip helper-address" configured on the 2960-S; it won't prevent the broadcasts from being flooded.

Would this be possible?

Best regards,
Peter

[brettp · Accepted solution]

Oh my, I figured it out and it's embarrassing, but still a bit confusing. There is actually a transparent firewall between the 2960 and the 6509. It's been in place for years -- so why would I bother to look at it? I decided to look and I noticed it was blocking the DHCP discover broadcast. This would have to be a huge coincidence, but what I think was happening was... that for all those years it was in place, no computer behind the firewall ever needed to get an address when it didn't have one (computers put in place back there would be set up by the help desk team in the office and then transplanted back to the area after it already had an IP.) So since the computers had a lease registered with the DHCP server already, it would never have to send a broadcast but instead would send a unicast back to the DHCP server (which had an allow rule on the firewall.) Either there by chance many leases expired during the period of my switch maintenance (the computer where people complained... it wasn't all of them, but then all of them weren't being used at the time) or there is some kind of timeout period... like when the DHCP server can't be contacted, maybe the Windows machines kind of remove that lease info or something. Not sure of the specifics with the problem, but ultimately, it was the firewall that I thought would not be the culprit. Thanks for your help anyway!

[Peter Paluch]

Hi Brett,

That's a wonderful finding.

I don't have all the pieces of the puzzle itself, but I know that Windows store their last DHCP lease in registry, including the identity of the DHCP server. After a reboot, they don't start with broadcasting DHCPDISCOVER - instead, they jump onto DHCPREQUEST immediately, though I do not remember if they send it as a unicast to the DHCP server directly, or if they still broadcast it - obviously, at this stage, they still do not have their own IP address, so there would be a problem in getting the response back.

Either way, I strongly suspect that the behavior is somehow tied into this.

Actually, was the transparent firewall dropping only broadcast DISCOVER, or also broadcast REQUEST packets?

Anyway, thank you for sharing your findings!

Best regards,
Peter

[brettp]

Honestly, I don't understand the specifics either... haha. I think it's the discover broadcast because the source is 0.0.0.0 and destination is 255.255.255.255 (I didn't look at it with wireshark or anything, just the logging on the firewall.) It is my belief, that the request packet would be a unicast packet to the DHCP server directly, not a multicast? I would think if a machine was rebooted, if it bypasses the discover process, the request would be sent to the DHCP server listed in the registry. Well, I'm glad the issue is fixed, but I guess I'll have to read up on DHCP! Thanks again for your help.