Hi, hopefully this is the right place for this question
We have been having issue with DHCP Ip address assignment for the past 3-4 weeks now where by some users pc/laptop (both LAN and Wireless) are not getting IP address. Its getting an APIPA.
We have 4 Cisco 3860 (stack) configured as a collapsed core switch (it is Core and Distribution switch). Current OS version: Gibraltar - 16.12.03aSwitching
Our initial set up when the issue suddenly occured:
DHCP Server (Core Switch) ---> IP phone (vlan812) ---> PC (vlan808)
All IP phones are ok. no issue getting IP address. Only the user vlan are having issues
We even have other machines that are "directly" connected to the switch. no ip phone in between but still having issue getting a valid ip address.
DHCP snooping is disabled.
Troubleshooting done:
disabled windows firewall on affected pc
disabled sleep/hybernate and power management option
ipv6 disabled
reinstall network driver
no port security configured on switch port
bounced switch port
flush dns
ipconfig /release and renew
restart pc
PC OS (win 10 and win 7)
user pc will be able to connect after a couple of tries doing the above steps. but will take at least 30 mins or more before they can connect and then after a while they will disconnect again and somehow releases the ip address.
from core switch arp table and dhcp binding table the ip address is still assigned to the PC mac address but for some reason the PC "had release" the IP and instead getting an APIPA. no issue if we statically assigned an IP address to the PC.
We have run a dhcp debug on the core switch and did a packet capture on some of the affected PC. we noticed that the 4 way handshake for the DHCP (D.O.R.A) are not being completed. Meaning we can see the PC are sending the Discover broadcast and the core switch received it and it sends the Offer reply but then never received the user Request reply but from the PC wireshark capture we can see it is continuously sending the Request reply. Or sometimes the core will received the Request reply from PC but there is no ACK reply being seen from either the core switch or the PC.
We have already did an IOS upgrade on the Core Switch as we initially thought it might be a bug on the cisco ios but after doing so the issue is still there.
After a few weeks of no luck, we have decided to move the DHCP server to our AD. here is our current setup now.
DHCP Server ---> Core Switch ---> IP Phones ----> PC
after the migration, seems like the issue is still there. we have already removed the pool configuration on the core switch. dhcp snooping is still disabled. what we have noticed so far is that when we check the core switch arp table we can see IP address assigned to the PC but somehow on the DHCP server that IP address is not showing as "leased IP address". so either there is a rouge dhcp server that is assigning the ip address but we dont have any other DHCP server running aside from the current one. and the weird part is only the User vlan (data) is having this issue and the voice vlan is not.
Below is the current configuration of both the user and voice vlan:
interface Vlan808
description USERVLAN
ip address 10.8.8.1 255.255.252.0
ip helper-address 172.20.1.1
ip helper-address 172.20.1.3
end
interface Vlan812
description VOIPVLAN
ip address 10.8.12.1 255.255.252.0
ip helper-address 172.20.1.1
ip helper-address 172.20.1.3
end
Appreciate if anyone can shed some light on what might be causing this issue. we are running out of option to check now.
another thing is when i run show platform hardware fed switch 1 qos queue stats internal cpu policer, i can see the dhcp snooping is enabled. not sure if that might have something to do with the issue we are facing.
CPU Queue Statistics
============================================================================================
(default) (set) Queue
QId PlcIdx Queue Name Enabled Rate Rate Drop(Bytes)
-----------------------------------------------------------------------------
0 11 DOT1X Auth Yes 1000 1000 0
1 1 L2 Control Yes 2000 2000 0
2 14 Forus traffic Yes 4000 4000 0
3 0 ICMP GEN Yes 600 200 0
4 2 Routing Control Yes 5400 1800 0
5 14 Forus Address resolution Yes 4000 4000 0
6 0 ICMP Redirect Yes 600 200 468269673006
7 16 Inter FED Traffic Yes 2000 2000 0
8 4 L2 LVX Cont Pack Yes 1000 1000 0
9 16 EWLC Control Yes 2000 2000 0
10 16 EWLC Data Yes 2000 2000 0
11 13 L2 LVX Data Pack Yes 1000 1000 0
12 0 BROADCAST Yes 600 200 316499752
13 10 Openflow Yes 100 100 0
14 13 Sw forwarding Yes 1000 1000 188404
15 8 Topology Control Yes 13000 6200 0
16 12 Proto Snooping Yes 2000 2000 0
17 6 DHCP Snooping Yes 500 500 0
18 13 Transit Traffic Yes 1000 1000 0
19 10 RPF Failed Yes 100 100 0
20 15 MCAST END STATION Yes 2000 2000 0
21 13 LOGGING Yes 1000 1000 0
22 7 Punt Webauth Yes 1000 1000 0
23 10 High Rate App Yes 100 100 0
24 10 Exception Yes 100 100 0
25 3 System Critical Yes 1000 1000 0
26 10 NFL SAMPLED DATA Yes 100 100 0
27 2 Low Latency Yes 5400 1800 0
28 10 EGR Exception Yes 100 100 0
29 5 Stackwise Virtual OOB Yes 8000 8000 0
30 9 MCAST Data Yes 500 500 0
31 3 Gold Pkt Yes 1000 1000 0
* NOTE: CPU queue policer rates are configured to the closest hardware supported value
CPU Queue Policer Statistics
====================================================================
Policer Policer Accept Policer Drop
Index Bytes Bytes
------------------------------------------------
0 30036055987 468586207518
1 62670684 0
2 1280 0
3 0 0
4 0 0
5 0 0
6 0 0
7 0 0
8 77033496 0
9 0 0
10 406860 0
11 1024 0
12 0 0
13 76832406 188404
14 243711982 0
15 2297790 0
16 0 0
17 0 0
CPP Classes to queue map
======================================================================================
PlcIdx CPP Class : Queues
--------------------------------------------------------------------------------------
0 system-cpp-police-data : ICMP GEN/ BROADCAST/ ICMP Redirect/
10 system-cpp-police-sys-data : Openflow/ High Rate App/ Exception/ EGR Exception/ NFL SAMPLED DATA/ RPF Failed/
13 system-cpp-police-sw-forward : Sw forwarding/ LOGGING/ L2 LVX Data Pack/ Transit Traffic/
9 system-cpp-police-multicast : MCAST Data/
15 system-cpp-police-multicast-end-station : MCAST END STATION /
7 system-cpp-police-punt-webauth : Punt Webauth/
1 system-cpp-police-l2-control : L2 Control/
2 system-cpp-police-routing-control : Routing Control/ Low Latency/
3 system-cpp-police-system-critical : System Critical/ Gold Pkt/
4 system-cpp-police-l2lvx-control : L2 LVX Cont Pack/
8 system-cpp-police-topology-control : Topology Control/
11 system-cpp-police-dot1x-auth : DOT1X Auth/
12 system-cpp-police-protocol-snooping : Proto Snooping/
6 system-cpp-police-dhcp-snooping : DHCP Snooping/
14 system-cpp-police-forus : Forus Address resolution/ Forus traffic/
5 system-cpp-police-stackwise-virt-control : Stackwise Virtual OOB/
16 system-cpp-default : Inter FED Traffic/ EWLC Control/ EWLC Data/
