DHCP Server Not Found

[ling20125]

Hi All,

  I'm having a very weird problem with the DHCP server on the network. I have a bunch of devices from different vendors that connects at a L2 switch whose uplink connects to a router. I'm using router-on-a-stick to carry all the VLAN information between the switch (3560) and router (2960). And I have a Windows 2008 as my DHCP server somewhere in the production network (I'm not in charge of that, but I have access). Each scope has the default gateway, DNS server, and DNS domain name configured.

  It couldn't be anymore simplier, I need to provide address for the devices from the DHCP server. I have doubled checked the DHCP server, and I can ping it from both the router and the switch. And on the switch, I can also ping the router's IP for each of its subinterfaces. The router also has set its 'ip helper-address' command on each of its sub-interfaces (Router-on-a-stick).

  Here is the dilemma: If I plug in a device, it's not going to get an IP address automatically (resulting in APIPA addresses). However, if I manually configure its IP address, subnet mask, and default gateway, I can ping everything.

  Now, here is what I'm thinking:

- The local network has a rogue DHCP server that I don't know about, and it doesn't have any scopes configured so no IP addresses can be assigned.

Result: I practically have checked every single port and I can't find a rogue DHCP server, at least not through its cables, anyway.

- The local network shuts down TCP port 69 (I believe the port for DHCP), therefore, ping works but not DHCP messages.

Result: I can't find a single ACL on the router or switch that does this.

  So, I ran out of options. And I want to ask, if anyone here knows such a situation and if so, what is the cause?

Regards,

Angela

PS: not sure if this is revelant, but I'm also experiencing MAC address appearing and disappearing. The end devices that are connected to the switch sometimes show up on the MAC address table, and sometimes they don't. And during this time, the port remains physically up.

[cadet alain · Accepted solution]

Hi angela,

the ports for DHCP are 67 for server and 68 for client not 69 which is tftp server.

Mac addresses age out of the CAM table every 5 mins by default if no traffic with this source MAC hasn't been  when the timer ends then when traffic is seen the timer is reset.So this is normal behaviour.

One possible issue is that as you are using dhcp relay on a router, first for the relay to work the dhcp service must be on( service dhcp global config  command).Next the DHCP server must have a route to the giaddr which is the address of your relay agent where you put the ip helper command.

You could do a debug ip dhcp server packets and debug ip dhcp server events on the router and post results.

Here is also an interesting document for troubleshooting DHCP:

http://www.cisco.com/en/US/tech/tk648/tk361/technologies_tech_note09186a00800f0804.shtml

Regards.

Alain.

[ling20125]

Hi Alain,

  Suppose that all you've said are true, and I've made sure I used 'service dhcp' on the router. I enabled the debugs, then I plugged out the power and network cable of a known device that uses DHCP, and I waited for some times (with the debug open), and nada. I haven't received a single DHCP request packet (at least not from the debugs of the router). So, does this mean that the devices are somehow broken and do not request DHCP traffic?

Regards,

Angela

[ling20125]

Hi Alain,

  I'm still not seeing DHCP debug packets, but somehow, a few devices are starting to get IP address. However, there is still a large portion that isn't. I'll be updating the post.

Thanks,

Angela

[cadet alain]

Hi angela,

How were you logged into the router? if it was not from console but by telnet/ssh then you have to issue the following commands to see debugs:

- terminal monitor

-logging terminal debug

But to prevent from being overflowed from debug messages I suggest sending debugs to the buffer with these commands:

-logging buffered 7

-logging terminal 6

-logging console 6

-logging buffered 100000

Then you'll have to do show log to see the debugs

You can also post a sanitized version of your configs from switch and router.

1 point I don't understand is why you're doing router on a stick when you have a L3 switch, you could do all on this switch.

Regards.

Alain.

[ling20125]

Hi Alain,

  Making the switch Layer 2 is by design, nothing too much I can do about it. Well, I've reloaded the switch, and still, only a few devices (including some Cisco stuff) got IP. Others haven't got anything on them yet. And yes, I was using telnet (sorry about the blast in my last post), and 'terminal monitor' showed these messages.

  And I keep getting these repeated BOOTP REQUEST from this address.

*Aug 12 09:36:16.396: DHCPD: unicasting BOOTREPLY to client 000f.4401.ebd4 (10.3.101.34).

*Aug 12 09:36:16.396: DHCPD: removing ARP entry (10.3.101.34 vrf default).

*Aug 12 09:36:20.452: DHCPD: client's VPN is .

*Aug 12 09:36:20.452: DHCPD: Sending notification of DISCOVER:

*Aug 12 09:36:20.452:   DHCPD: htype 1 chaddr 000f.4401.ebd4

*Aug 12 09:36:20.452:   DHCPD: remote id 020a00000a03652100000184

*Aug 12 09:36:20.452:   DHCPD: circuit id 00000000

*Aug 12 09:36:20.456: DHCPD: Seeing if there is an internally specified pool class:

*Aug 12 09:36:20.456:   DHCPD: htype 1 chaddr 000f.4401.ebd4

*Aug 12 09:36:20.456:   DHCPD: remote id 020a00000a03652100000184

*Aug 12 09:36:20.456:   DHCPD: circuit id 00000000

*Aug 12 09:36:20.456: DHCPD: setting giaddr to 10.3.101.33.

*Aug 12 09:36:20.456: DHCPD: BOOTREQUEST from 0100.0f44.01eb.d4 forwarded to 192.168.2.100.

*Aug 12 09:36:20.456: DHCPD: client's VPN is .

*Aug 12 09:36:20.456: DHCPD: forwarding BOOTREPLY to client 000f.4401.ebd4.

*Aug 12 09:36:20.456: DHCPD: Forwarding reply on numbered intf

*Aug 12 09:36:20.456: DHCPD: creating ARP entry (10.3.101.34, 000f.4401.ebd4, vrf default).

*Aug 12 09:36:20.456: DHCPD: unicasting BOOTREPLY to client 000f.4401.ebd4 (10.3.101.34).

Regards,

Angela

Edited: I've also added '(config)#ip dhcp relay information option' command to the router relaying DHCP requests.

Message was edited by: angela zou

[Alexander Maroukian]

Hi Angela,

You can try to use global config command "ip dhcp smart-relay" as this is the relay for secondary networks.

Best regards,

Alex

[ling20125]

Hi Alex,

  I added the command, I will update if there are any new information. The biggest concern I have right now is that these devices are non-Cisco, and those device that did get an IP is either a Cisco device or has an operating system. Some of them, I can't even check their IP address from their local end, so this is kind of frustrating.

Regards,

Angela

[cadet alain]

angela zou a écrit:
Some of them, I can't even check their IP address from their local end, so this is kind of frustrating.

you could  do a SPAN session on your switch and then sniff with wireshark on the destination SPAN port to see the DHCP packets sent/received by these devices.

Regards.

Alain.

[Alexander Maroukian]

Hi Angela,

Some sort of ip scanner, your NMS or the DHCP server would tell you if your devices are getting their ip addresses properly. They should probably be restarted to get new addresses.

Regards,

Alex

[ling20125]

Yes, I did restart them, even the switch itself. Not a lot of progress though.

Regards,

Angela

[cadet alain]

Hi,

Could you try without this :

ip dhcp relay information option

Regards.

Alain.

[ling20125]

I certainly could, but I just want to see what would make the config work...

Regards,

Angela

[Alexander Maroukian · Accepted solution]

What is your router you have written 2960 which is switch?

Are you getting ip addresses in all vlans with the devices which are working properly and can get ip address?

How many hops is the DHCP server from the hosts?

You should not have "no ip directed-broadcast" on the outbond interface where the broadcast need to traverse if "ip helper-address" is subnet broadcast address of the DHCP server. This should be used with caution but is required if this is the config.

regards,

Alex

[ling20125]

  So, I get to look at these devices this morning, and most of the devices are getting an IP, most, not all. The remaining devices that are not getting IP are all located in one VLAN, one that has a different subnet mask from the rest of the VLANs. (These DHCP configurations are set on a Windows 2008 server)

  My assumption is that there may be something wrong with the scope, causing it to affect all the devices (with manual binding) in it. Some of the devices are able to get their IP, but none of them are never pingable. In addition, some of these addresses even claimed to be taken already. (how could the DHCP server declare an address taken even though the DHCP server can probably, not ping it?).

  However, if I hook up a cable between my laptop and that particular device, its IP is pingable.

  Another thing is, I added a few hosts to dynamically get their IP address, and these addresses are the only ones that are pingable. So I'm a little confused, I checked the configuration settings of this scope. It's no different from other scopes that are working correctly. What could be the cause of this? (I'm guessing something to do with the default gateway, which is, surprisingly, pingable).

Regards,

Angela