Solved: 3750 not handing out IP Addresses

mano.hernandez · ‎01-11-2013

We have a 3750 running IOS ver 12.2 (44) SE, it has performed great and we have never had a problem with it. However we have noticed that when we had an outage some of our Wireless APs didn't come up as they get DHCP from the 3750. The DHCP scope said IP was depleted although there were IPs to give. We had to delete and recreate the DHCP Pool. However two days later we got the same problem and then had to do the same thing over again. Any Ideas?

Gregory Snipes · ‎01-11-2013

You are getting conflicts in at least two different subnets on this device I see them in both the 192.168.12.X and the 192.168.4.X. The impact these conflicts will have on the subnets will vary greatly depending on: the size of the subnets, the number of clients attempting to obtain addresses, and the order in witch the clients attempt to renew their addresses.

Basically the problem goes as follows:

-The switch hands out addresses and gives: address 1 to client a, address 2 to client b, address 3 to client c, ect...

-The switch reloads and loeses its database, it now has no idea that it has given any address to any client.

-Client d attempts to obtain an IP address, the switch says "lets give him address 1" but then it finds that address 1 is in use (by client a) and says "I will mark that as a bad address and not give that out" it then repeats this process until it finds an address that is not in use, marking everyone it can ping as bad along the way.

-Client a then attempts to renew its IP address, but the switch says " no you can't have that, it's a badd address" and gives it the first good address it has.

Depending on the utilization of the subnet, the switch can end up filling up a whole pool with bad addresses and not have any good ones to hand out. If the utilization is low in the subnet, it may have enough to give everyone a new address on top of the bad addresses and you would not know you have a problem.

View solution in original post

stephen.stack · ‎01-11-2013

Hi,

Any specific log entires or errors to go by? What was the output of the following commands?

sh ip dhcp server statistics

sh ip dhcp pool

sh ip dhcp binding

sh ip dhcp conflict

You may also want to run a

deb ip dhcp server events when this issue happens for more verbose logging.

Regards

Stephen

==========================
http://www.rConfig.com

A free, open source network device configuration management tool, customizable to your needs!

- Always vote on an answer if you found it helpful

========================== http://www.rconfig.com A free, open source network device configuration management tool, customizable to your needs! - Always vote on an answer if you found it helpful

mano.hernandez · ‎01-11-2013

Library_3750_L3#sho ip dhcp server statistics

Memory usage 9419

Address pools 2

Database agents 0

Automatic bindings 18

Manual bindings 0

Expired bindings 355

Malformed messages 30

Message Received

BOOTREQUEST 162818

DHCPDISCOVER 679538

DHCPREQUEST 717559

DHCPDECLINE 204

DHCPRELEASE 7015

DHCPINFORM 791935

Message Sent

BOOTREPLY 0

DHCPOFFER 16669

DHCPACK 97891

DHCPNAK 887

Library_3750_L3#sh ip dhcp binding

IP address Client-ID/ Lease expiration Type

Hardware address

192.168.4.101 0100.1fca.2843.d8 Jan 12 2013 01:23 PM Automatic

192.168.4.102 0100.1fca.2844.42 Jan 12 2013 01:23 PM Automatic

192.168.4.104 0100.1819.bdbb.4a Jan 12 2013 01:23 PM Automatic

192.168.4.105 0100.1ae2.531a.82 Jan 12 2013 01:23 PM Automatic

192.168.4.109 0100.2414.d23f.e8 Jan 12 2013 01:23 PM Automatic

192.168.4.110 0100.1fca.2844.1c Jan 12 2013 01:23 PM Automatic

192.168.4.114 0100.1ae2.5315.de Jan 12 2013 01:23 PM Automatic

192.168.4.119 01d4.8cb5.b005.96 Jan 12 2013 01:23 PM Automatic

192.168.4.120 0100.1aa2.3354.4a Jan 12 2013 01:23 PM Automatic

192.168.4.122 01c8.4c75.a989.74 Jan 12 2013 01:23 PM Automatic

192.168.4.125 0100.1aa2.3350.2c Jan 12 2013 01:23 PM Automatic

192.168.4.127 0100.1fca.27e5.0c Jan 12 2013 01:23 PM Automatic

192.168.4.128 0100.1fca.2843.78 Jan 12 2013 01:24 PM Automatic

192.168.4.131 0100.1aa2.3350.38 Jan 12 2013 01:24 PM Automatic

192.168.4.133 0130.f70d.e65f.d9 Jan 12 2013 01:24 PM Automatic

192.168.14.95 0100.175a.df7e.81 Jan 12 2013 01:24 PM Automatic

192.168.14.96 0100.2414.b446.27 Jan 12 2013 01:27 PM Automatic

192.168.14.97 0100.1759.3f49.87 Jan 12 2013 01:29 PM Automatic

Library_3750_L3#sh ip dhcp conflict

IP address Detection method Detection time

192.168.12.29 Gratuitous ARP Mar 01 1993 12:11 AM

192.168.12.26 Gratuitous ARP Mar 01 1993 12:11 AM

192.168.12.75 Ping Mar 01 1993 12:12 AM

192.168.12.27 Gratuitous ARP Mar 01 1993 12:12 AM

192.168.12.130 Ping Mar 01 1993 12:53 AM

192.168.12.131 Ping Mar 01 1993 12:53 AM

192.168.12.138 Ping Mar 01 1993 12:54 AM

192.168.12.168 Ping Mar 01 1993 01:30 AM

192.168.12.187 Ping Mar 01 1993 03:14 AM

192.168.12.188 Ping Mar 01 1993 03:14 AM

192.168.12.189 Ping Mar 01 1993 03:14 AM

192.168.12.190 Ping Mar 01 1993 03:14 AM

192.168.12.191 Ping Mar 01 1993 03:14 AM

192.168.12.192 Ping Mar 01 1993 03:14 AM

192.168.12.193 Ping Mar 01 1993 03:14 AM

192.168.12.194 Ping Mar 01 1993 03:14 AM

192.168.12.195 Ping Mar 01 1993 03:14 AM

192.168.12.196 Ping Mar 01 1993 03:14 AM

192.168.4.100 Ping Jan 11 2013 01:23 PM

192.168.4.103 Ping Jan 11 2013 01:23 PM

192.168.4.106 Ping Jan 11 2013 01:23 PM

192.168.4.107 Ping Jan 11 2013 01:23 PM

192.168.4.108 Ping Jan 11 2013 01:23 PM

192.168.4.111 Ping Jan 11 2013 01:23 PM

192.168.4.112 Ping Jan 11 2013 01:23 PM

192.168.4.113 Ping Jan 11 2013 01:23 PM

192.168.4.115 Ping Jan 11 2013 01:23 PM

192.168.4.116 Ping Jan 11 2013 01:23 PM

192.168.4.117 Ping Jan 11 2013 01:23 PM

192.168.4.118 Ping Jan 11 2013 01:23 PM

192.168.4.121 Ping Jan 11 2013 01:23 PM

192.168.4.123 Ping Jan 11 2013 01:23 PM

192.168.4.124 Ping Jan 11 2013 01:23 PM

192.168.4.126 Ping Jan 11 2013 01:23 PM

192.168.4.129 Ping Jan 11 2013 01:23 PM

192.168.4.130 Ping Jan 11 2013 01:23 PM

192.168.4.132 Ping Jan 11 2013 01:23 PM

Library_3750_L3#sh ip dhcp ?

binding DHCP address bindings

conflict DHCP address conflicts

database DHCP database agents

import Show Imported Parameters

relay Miscellaneous DHCP relay information

server Miscellaneous DHCP server information

snooping DHCP snooping

Gregory Snipes · ‎01-11-2013

Are you storing the DHCP file in persistent memory? The defualt location for a Cisco device to store its DHCP database is in RAM. When the device reloads it loses the database and you wind up with conflicts as the device trys to hand out IPs that are already in use.

You can move the file with "dhcp database Flash:dhcp.txt"

mano.hernandez · ‎01-11-2013

I am unsure where to look for where the database is stored, but I do know our other DHCP Pool has no problem.

Gregory Snipes · ‎01-11-2013

From your output, you definitely have a bunch of conflicts. This strongly supports my theory of what your issue is. I would recommend moving the DHCP file to persistent memory and clearing out the conflicts that have been created with the "clear ip dhcp conflict * " command repeatedly until you stop seeing them.

mano.hernandez · ‎01-11-2013

Can you explain why it is happening to one DHCP Pool and not the other?

Gregory Snipes · ‎01-11-2013

You are getting conflicts in at least two different subnets on this device I see them in both the 192.168.12.X and the 192.168.4.X. The impact these conflicts will have on the subnets will vary greatly depending on: the size of the subnets, the number of clients attempting to obtain addresses, and the order in witch the clients attempt to renew their addresses.

Basically the problem goes as follows:

-The switch hands out addresses and gives: address 1 to client a, address 2 to client b, address 3 to client c, ect...

-The switch reloads and loeses its database, it now has no idea that it has given any address to any client.

-Client d attempts to obtain an IP address, the switch says "lets give him address 1" but then it finds that address 1 is in use (by client a) and says "I will mark that as a bad address and not give that out" it then repeats this process until it finds an address that is not in use, marking everyone it can ping as bad along the way.

-Client a then attempts to renew its IP address, but the switch says " no you can't have that, it's a badd address" and gives it the first good address it has.

Depending on the utilization of the subnet, the switch can end up filling up a whole pool with bad addresses and not have any good ones to hand out. If the utilization is low in the subnet, it may have enough to give everyone a new address on top of the bad addresses and you would not know you have a problem.

mano.hernandez · ‎01-14-2013

This was from TAC

Thanks for your e-mail.

This problems seems to be related to DHCP conflict issue. Let me explain what it is:

Before allocating an ip address to the device requesting for it the router sends an echo request packet to check if the ip address is available. If the router receives ICMP Echo Reply message the address is in use. If the DHCP conflict logging is enabled (default), the router will log the conflict with a syslog message (not in a separate log file) and put the address on the list of conflicts. The addresses on that list (displayed with show ip dhcp conflict) are not used in the future (similar to the addresses configured with the ip dhcp excluded-addresses command). To reuse a conflicting address, we need to remove it from the list with the clear ip dhcp conflict address (or * for all addresses) command.

Cisco recommends disabling the conflict logging by executing the command 'no ip dhcp conflict logging' on the device. This does not affect the DHCP functionality in any way since the server will always ping before allocating an IP address. The only difference would be that if it receives a reply then it will not put that IP in the conflict table and will move on to the next available IP address in the pool and
allocate that if there is no conflict.

As we can see you already have a lot of IP addresses in conflict list:

I would highly recommend you to disable HDCP conflict logging using 'no ip dhcp conflict logging' command. This should fix the issue you are facing. Feel free to revert in case you have any questions for me.