DHCP Severe Problem on Cisco Catalyst 9K Switches

[Rami Ibrahim]

Hello,

Please help me because I have a serious problem that I don't know the cause of it yet. I have a general enterprise network with Collapse Core Design where clients connected on access switches are not able to get IP addresses through DHCP from the DHCP server.

When endpoints request IP through DHCP sometimes it works sometimes not, some IP Phones able to renew others won't, some endpoints even couldn't get an IP at all. I have been troubleshooting this for 2 days and upgrade Access SW IOS-XE from 17.3.4 to the latest 17.9.3 and the same problem persists.

My network setup is as follows:

Endpoints (Cisco IP Phone, PCs) >> Access Switch (2 x 9300) Stack >> Core Switch (2 x 9606) Stackwise-Virtual >> DHCP server (Microsoft Windows Server 2012) connected also directly on the Core SW.

I have Access SW connected to the Core SW through a port-channel (2 x 10G) links and finally I realized that when I shutdown the second uplink (between member no 2 in the Access SW stack & Standby Core SW) all endpoints were able to get IP through DHCP and all start to work as expected. When I revert both links to the port-channel the issue repeated again. 

Any ideas on what could result in this behavior since the port-channel considered a single link logically from an STP perspective and it should not cause this issue. 

Note: TAC case opened 2 days ago and the engineer still checking with no findings yet.

Thank you,

Kindest Regards,  

[Leo Laohoo]

---

@Rami Ibrahim wrote:
I have Access SW connected to the Core SW through a port-channel (2 x 10G) links and finally I realized that when I shutdown the second uplink (between member no 2 in the Access SW stack & Standby Core SW) all endpoints were able to get IP through DHCP and all start to work as expected. When I revert both links to the port-channel the issue repeated again. 

---

What is the configuration of the two physical ports? 

Post the complete output to the command "sh logs". 

---

@Rami Ibrahim wrote:
TAC case opened 2 days ago and the engineer still checking with no findings yet.

---

What "region" is the TAC Case assigned to, EUMA, Asia or LATAM/RTP?

[Rami Ibrahim]

Hi,

 The configuration of member ports are just trunk nothing else. 

 Actually, I didn't know which region was assigned.

Thanks

[Leo Laohoo]

---

@Rami Ibrahim wrote:
Actually, I didn't know which region was assigned.

---

Look at the signature of the TAC Agent.  It will state what TAC region he/she is from.

[MHM Cisco World]

show platform software fed switch active punt cause summary <<- first share this 

second the SW forwarding Queue drop packets,  one of traffic server by this Queue is transit traffic, which I think issue here, 
what is SVL BW you use is it 40 M?

[Rami Ibrahim]

Please find the requested output attached sir. 

We are running two twentyfiveGig links between core SWs and SVL BW is 50G.

Regards,

[Rami Ibrahim]

Let me just add that this was working before perfect, but we had only one Cat9300 SW connected via Port-channel to both Core SWs, when we build a stack of 2 Cat9300 and connect via Port-Channel the problem appears.

I don't know if this is related honestly. 

[MHM Cisco World]

can you share the same output but for Access SW 9300 no Core SW
thanks 
MHM

[Rami Ibrahim]

Absolutely please find attached.

Yesterday, a new TAC case opened with severity 1 and product team connect with us those were very very skilled and experience. they check for about 5 hours, and they determined the drop is happening on standby core sw, they did a PSV capture to trace it further, but it was not supported in our release, so they recommended a software upgrade since they suspected a software defect. What we are going to do is to reboot the standby core switch first in a try to eliminate the option of any MACs (suspected ffff.ffff.ffff) are stuck into the ASICs and if it didn't work then upgrade and continue troubleshooting. 

I am also suspecting one thing that is BUM optimization feature, only me who suspect this, and I just wanted to know based on your experience sir is this something which can result in this issue preventing the Broadcast to go over the SVL and for some interesting reason the switch couldn't flood locally the broadcast to DHCP, so it drops the packet.  What do you think.

Thank you,

[MHM Cisco World]

As I mention it can from SVL drop or the queue drop packet.

What I see glean adjunacy drop in one Core' I need to sure it svl not queue 

So I need below also 

Show controllers ethernet-control

For c9300.

For SVL hope upgrade solve issue' 

If not then we need to increase bw of svl to reduce transit traffic drop' and if the server is connect to one core we try to connect to both corw via port-channel.

All above is to reduce the traffic pass through svl.

[Rami Ibrahim]

In fact, TAC team mentioned the SW forwarding queue drops are so small and not related. by the way we have two twentyfive Gig links SVL so total of 50 Gigs on the SVL and the server is connected to both core SWs (2 x 10G) links on each Core.

I left the office now so tomorrow I will share the request output

Thank you so much 

[Rami Ibrahim]

Hi,

here is the output on SVL links on both Core SW and on C9300 Uplinks.

Core C9600
============

show controllers ethernet-controller twe1/2/0/47 (1st SVL link on Active Core SW)
Transmit TwentyFiveGigE1/2/0/47 Receive
179759960929 Total bytes 4050315375849 Total bytes
232024152 Unicast frames 5200173083 Unicast frames
179745165115 Unicast bytes 4050306541312 Unicast bytes
89777 Multicast frames 53844 Multicast frames
14795814 Multicast bytes 8834537 Multicast bytes
0 Broadcast frames 0 Broadcast frames
0 Broadcast bytes 1 Broadcast bytes
0 System FCS error frames 0 IpgViolation frames
0 MacUnderrun frames 0 MacOverrun frames
0 Pause frames 0 Pause frames
0 Cos 0 Pause frames 0 Cos 0 Pause frames
0 Cos 1 Pause frames 0 Cos 1 Pause frames
0 Cos 2 Pause frames 0 Cos 2 Pause frames
0 Cos 3 Pause frames 0 Cos 3 Pause frames
0 Cos 4 Pause frames 0 Cos 4 Pause frames
0 Cos 5 Pause frames 0 Cos 5 Pause frames
0 Cos 6 Pause frames 0 Cos 6 Pause frames
0 Cos 7 Pause frames 0 Cos 7 Pause frames
0 Oam frames 0 OamProcessed frames
0 Oam frames 0 OamDropped frames
0 Minimum size frames 0 Minimum size frames
0 65 to 127 byte frames 0 65 to 127 byte frames
76662091 128 to 255 byte frames 2327730094 128 to 255 byte frames
47662598 256 to 511 byte frames 463516153 256 to 511 byte frames
9175465 512 to 1023 byte frames 110423417 512 to 1023 byte frames
78144415 1024 to 1518 byte frames 675636470 1024 to 1518 byte frames
20469360 1519 to 2047 byte frames 1622920793 1519 to 2047 byte frames
0 2048 to 4095 byte frames 0 2048 to 4095 byte frames
0 4096 to 8191 byte frames 0 4096 to 8191 byte frames
0 8192 to 16383 byte frames 0 8192 to 16383 byte frames
0 16384 to 32767 byte frame 0 16384 to 32767 byte frame
0 > 32768 byte frames 0 > 32768 byte frames
0 Late collision frames 0 SymbolErr frames
0 Excess Defer frames 0 Collision fragments
0 Good (1 coll) frames 0 ValidUnderSize frames
0 Good (>1 coll) frames 0 InvalidOverSize frames
0 Deferred frames 0 ValidOverSize frames
0 Gold frames dropped 0 FcsErr frames
0 Gold frames truncated
0 Gold frames successful
0 1 collision frames
0 2 collision frames
0 3 collision frames
0 4 collision frames
0 5 collision frames
0 6 collision frames
0 7 collision frames
0 8 collision frames
0 9 collision frames
0 10 collision frames
0 11 collision frames
0 12 collision frames
0 13 collision frames
0 14 collision frames
0 15 collision frames
0 Excess collision frames

LAST UPDATE 1376 msecs AGO

show controllers ethernet-controller twe1/2/0/48 (2nd SVL link on Active Core SW)
Transmit TwentyFiveGigE1/2/0/48 Receive
252398224847 Total bytes 4857632364026 Total bytes
401181877 Unicast frames 5719775771 Unicast frames
251948807472 Unicast bytes 4857390731732 Unicast bytes
3444799 Multicast frames 1824329 Multicast frames
449417375 Multicast bytes 241632294 Multicast bytes
0 Broadcast frames 0 Broadcast frames
0 Broadcast bytes 1 Broadcast bytes
0 System FCS error frames 0 IpgViolation frames
0 MacUnderrun frames 0 MacOverrun frames
0 Pause frames 0 Pause frames
0 Cos 0 Pause frames 0 Cos 0 Pause frames
0 Cos 1 Pause frames 0 Cos 1 Pause frames
0 Cos 2 Pause frames 0 Cos 2 Pause frames
0 Cos 3 Pause frames 0 Cos 3 Pause frames
0 Cos 4 Pause frames 0 Cos 4 Pause frames
0 Cos 5 Pause frames 0 Cos 5 Pause frames
0 Cos 6 Pause frames 0 Cos 6 Pause frames
0 Cos 7 Pause frames 0 Cos 7 Pause frames
0 Oam frames 0 OamProcessed frames
0 Oam frames 0 OamDropped frames
0 Minimum size frames 0 Minimum size frames
0 65 to 127 byte frames 4 65 to 127 byte frames
201656236 128 to 255 byte frames 2211112078 128 to 255 byte frames
48447970 256 to 511 byte frames 479077927 256 to 511 byte frames
24169686 512 to 1023 byte frames 337049752 512 to 1023 byte frames
108929189 1024 to 1518 byte frames 704573524 1024 to 1518 byte frames
21423595 1519 to 2047 byte frames 1989786815 1519 to 2047 byte frames
0 2048 to 4095 byte frames 0 2048 to 4095 byte frames
0 4096 to 8191 byte frames 0 4096 to 8191 byte frames
0 8192 to 16383 byte frames 0 8192 to 16383 byte frames
0 16384 to 32767 byte frame 0 16384 to 32767 byte frame
0 > 32768 byte frames 0 > 32768 byte frames
0 Late collision frames 0 SymbolErr frames
0 Excess Defer frames 0 Collision fragments
0 Good (1 coll) frames 0 ValidUnderSize frames
0 Good (>1 coll) frames 0 InvalidOverSize frames
0 Deferred frames 0 ValidOverSize frames
0 Gold frames dropped 0 FcsErr frames
0 Gold frames truncated
0 Gold frames successful
0 1 collision frames
0 2 collision frames
0 3 collision frames
0 4 collision frames
0 5 collision frames
0 6 collision frames
0 7 collision frames
0 8 collision frames
0 9 collision frames
0 10 collision frames
0 11 collision frames
0 12 collision frames
0 13 collision frames
0 14 collision frames
0 15 collision frames
0 Excess collision frames

LAST UPDATE 75 msecs AGO

show controllers ethernet-controller twe2/2/0/47 (1st SVL link on Standby Core SW)
Transmit TwentyFiveGigE2/2/0/47 Receive
4055223258481 Total bytes 179847203962 Total bytes
5205548872 Unicast frames 232236347 Unicast frames
4055214425142 Unicast bytes 179832408030 Unicast bytes
53881 Multicast frames 89824 Multicast frames
8833339 Multicast bytes 14795932 Multicast bytes
0 Broadcast frames 0 Broadcast frames
0 Broadcast bytes 1 Broadcast bytes
0 System FCS error frames 0 IpgViolation frames
0 MacUnderrun frames 0 MacOverrun frames
0 Pause frames 0 Pause frames
0 Cos 0 Pause frames 0 Cos 0 Pause frames
0 Cos 1 Pause frames 0 Cos 1 Pause frames
0 Cos 2 Pause frames 0 Cos 2 Pause frames
0 Cos 3 Pause frames 0 Cos 3 Pause frames
0 Cos 4 Pause frames 0 Cos 4 Pause frames
0 Cos 5 Pause frames 0 Cos 5 Pause frames
0 Cos 6 Pause frames 0 Cos 6 Pause frames
0 Cos 7 Pause frames 0 Cos 7 Pause frames
0 Oam frames 0 OamProcessed frames
0 Oam frames 0 OamDropped frames
0 Minimum size frames 0 Minimum size frames
0 65 to 127 byte frames 0 65 to 127 byte frames
2329133897 128 to 255 byte frames 76746972 128 to 255 byte frames
464527860 256 to 511 byte frames 47754662 256 to 511 byte frames
110562734 512 to 1023 byte frames 9185349 512 to 1023 byte frames
676443051 1024 to 1518 byte frames 78156602 1024 to 1518 byte frames
1624935211 1519 to 2047 byte frames 20482586 1519 to 2047 byte frames
0 2048 to 4095 byte frames 0 2048 to 4095 byte frames
0 4096 to 8191 byte frames 0 4096 to 8191 byte frames
0 8192 to 16383 byte frames 0 8192 to 16383 byte frames
0 16384 to 32767 byte frame 0 16384 to 32767 byte frame
0 > 32768 byte frames 0 > 32768 byte frames
0 Late collision frames 0 SymbolErr frames
52388 Excess Defer frames 0 Collision fragments
0 Good (1 coll) frames 0 ValidUnderSize frames
0 Good (>1 coll) frames 0 InvalidOverSize frames
0 Deferred frames 0 ValidOverSize frames
0 Gold frames dropped 0 FcsErr frames
0 Gold frames truncated
0 Gold frames successful
0 1 collision frames
0 2 collision frames
0 3 collision frames
0 4 collision frames
0 5 collision frames
0 6 collision frames
0 7 collision frames
0 8 collision frames
0 9 collision frames
0 10 collision frames
0 11 collision frames
0 12 collision frames
0 13 collision frames
0 14 collision frames
0 15 collision frames
0 Excess collision frames

LAST UPDATE 866 msecs AGO

show controllers ethernet-controller twe2/2/0/48 (2nd SVL link on Standby Core SW)
Transmit TwentyFiveGigE2/2/0/48 Receive
4862020454331 Total bytes 252545316941 Total bytes
5725209365 Unicast frames 401498427 Unicast frames
4861778671453 Unicast bytes 252095560108 Unicast bytes
1825528 Multicast frames 3447469 Multicast frames
241782878 Multicast bytes 449756833 Multicast bytes
0 Broadcast frames 0 Broadcast frames
0 Broadcast bytes 1 Broadcast bytes
0 System FCS error frames 0 IpgViolation frames
0 MacUnderrun frames 0 MacOverrun frames
0 Pause frames 0 Pause frames
0 Cos 0 Pause frames 0 Cos 0 Pause frames
0 Cos 1 Pause frames 0 Cos 1 Pause frames
0 Cos 2 Pause frames 0 Cos 2 Pause frames
0 Cos 3 Pause frames 0 Cos 3 Pause frames
0 Cos 4 Pause frames 0 Cos 4 Pause frames
0 Cos 5 Pause frames 0 Cos 5 Pause frames
0 Cos 6 Pause frames 0 Cos 6 Pause frames
0 Cos 7 Pause frames 0 Cos 7 Pause frames
0 Oam frames 0 OamProcessed frames
0 Oam frames 0 OamDropped frames
0 Minimum size frames 0 Minimum size frames
4 65 to 127 byte frames 0 65 to 127 byte frames
2213005797 128 to 255 byte frames 201799464 128 to 255 byte frames
479820996 256 to 511 byte frames 48539811 256 to 511 byte frames
337576392 512 to 1023 byte frames 24202318 512 to 1023 byte frames
705387695 1024 to 1518 byte frames 108967680 1024 to 1518 byte frames
1991244009 1519 to 2047 byte frames 21436623 1519 to 2047 byte frames
0 2048 to 4095 byte frames 0 2048 to 4095 byte frames
0 4096 to 8191 byte frames 0 4096 to 8191 byte frames
0 8192 to 16383 byte frames 0 8192 to 16383 byte frames
0 16384 to 32767 byte frame 0 16384 to 32767 byte frame
0 > 32768 byte frames 0 > 32768 byte frames
0 Late collision frames 0 SymbolErr frames
46623 Excess Defer frames 0 Collision fragments
0 Good (1 coll) frames 0 ValidUnderSize frames
0 Good (>1 coll) frames 0 InvalidOverSize frames
0 Deferred frames 0 ValidOverSize frames
0 Gold frames dropped 0 FcsErr frames
0 Gold frames truncated
0 Gold frames successful
0 1 collision frames
0 2 collision frames
0 3 collision frames
0 4 collision frames
0 5 collision frames
0 6 collision frames
0 7 collision frames
0 8 collision frames
0 9 collision frames
0 10 collision frames
0 11 collision frames
0 12 collision frames
0 13 collision frames
0 14 collision frames
0 15 collision frames
0 Excess collision frames

LAST UPDATE 1170 msecs AGO

C9300
=========
show controllers ethernet-controller tenGigabitEthernet 1/1/1 (1st Uplink to Active Core SW)
Transmit TenGigabitEthernet1/1/1 Receive
28722211439 Total bytes 314112702205 Total bytes
145904771 Unicast frames 256812741 Unicast frames
28562199717 Unicast bytes 311865426663 Unicast bytes
438891 Multicast frames 10186172 Multicast frames
116825629 Multicast bytes 1588982244 Multicast bytes
617626 Broadcast frames 8398215 Broadcast frames
43186175 Broadcast bytes 658293298 Broadcast bytes
0 System FCS error frames 0 IpgViolation frames
0 MacUnderrun frames 0 MacOverrun frames
0 Pause frames 0 Pause frames
0 Cos 0 Pause frames 0 Cos 0 Pause frames
0 Cos 1 Pause frames 0 Cos 1 Pause frames
0 Cos 2 Pause frames 0 Cos 2 Pause frames
0 Cos 3 Pause frames 0 Cos 3 Pause frames
0 Cos 4 Pause frames 0 Cos 4 Pause frames
0 Cos 5 Pause frames 0 Cos 5 Pause frames
0 Cos 6 Pause frames 0 Cos 6 Pause frames
0 Cos 7 Pause frames 0 Cos 7 Pause frames
0 Oam frames 0 OamProcessed frames
0 Oam frames 0 OamDropped frames
463360 Minimum size frames 1692004 Minimum size frames
115522472 65 to 127 byte frames 40322095 65 to 127 byte frames
10413906 128 to 255 byte frames 19589298 128 to 255 byte frames
4742207 256 to 511 byte frames 6846673 256 to 511 byte frames
9172707 512 to 1023 byte frames 7567341 512 to 1023 byte frames
3382649 1024 to 1518 byte frames 33235568 1024 to 1518 byte frames
3263987 1519 to 2047 byte frames 166144149 1519 to 2047 byte frames
0 2048 to 4095 byte frames 0 2048 to 4095 byte frames
0 4096 to 8191 byte frames 0 4096 to 8191 byte frames
0 8192 to 16383 byte frames 0 8192 to 16383 byte frames
0 16384 to 32767 byte frame 0 16384 to 32767 byte frame
0 > 32768 byte frames 0 > 32768 byte frames
0 Late collision frames 0 SymbolErr frames
0 Excess Defer frames 0 Collision fragments
0 Good (1 coll) frames 0 ValidUnderSize frames
0 Good (>1 coll) frames 0 InvalidOverSize frames
0 Deferred frames 0 ValidOverSize frames
0 Gold frames dropped 0 FcsErr frames
0 Gold frames truncated
0 Gold frames successful
0 1 collision frames
0 2 collision frames
0 3 collision frames
0 4 collision frames
0 5 collision frames
0 6 collision frames
0 7 collision frames
0 8 collision frames
0 9 collision frames
0 10 collision frames
0 11 collision frames
0 12 collision frames
0 13 collision frames
0 14 collision frames
0 15 collision frames
0 Excess collision frames

LAST UPDATE 3004 msecs AGO

show controllers ethernet-controller tenGigabitEthernet 2/1/1 (2nd Uplink to Standby Core SW)
Transmit TenGigabitEthernet2/1/1 Receive
3443125 Total bytes 4659589620 Total bytes
0 Unicast frames 2832836 Unicast frames
0 Unicast bytes 3218528842 Unicast bytes
12154 Multicast frames 6346246 Multicast frames
3443125 Multicast bytes 1021624907 Multicast bytes
0 Broadcast frames 5425594 Broadcast frames
0 Broadcast bytes 419435871 Broadcast bytes
0 System FCS error frames 0 IpgViolation frames
0 MacUnderrun frames 0 MacOverrun frames
0 Pause frames 0 Pause frames
0 Cos 0 Pause frames 0 Cos 0 Pause frames
0 Cos 1 Pause frames 0 Cos 1 Pause frames
0 Cos 2 Pause frames 0 Cos 2 Pause frames
0 Cos 3 Pause frames 0 Cos 3 Pause frames
0 Cos 4 Pause frames 0 Cos 4 Pause frames
0 Cos 5 Pause frames 0 Cos 5 Pause frames
0 Cos 6 Pause frames 0 Cos 6 Pause frames
0 Cos 7 Pause frames 0 Cos 7 Pause frames
0 Oam frames 0 OamProcessed frames
0 Oam frames 0 OamDropped frames
4700 Minimum size frames 1098126 Minimum size frames
123 65 to 127 byte frames 8671948 65 to 127 byte frames
0 128 to 255 byte frames 1775384 128 to 255 byte frames
7331 256 to 511 byte frames 417521 256 to 511 byte frames
0 512 to 1023 byte frames 734011 512 to 1023 byte frames
0 1024 to 1518 byte frames 127072 1024 to 1518 byte frames
0 1519 to 2047 byte frames 1780614 1519 to 2047 byte frames
0 2048 to 4095 byte frames 0 2048 to 4095 byte frames
0 4096 to 8191 byte frames 0 4096 to 8191 byte frames
0 8192 to 16383 byte frames 0 8192 to 16383 byte frames
0 16384 to 32767 byte frame 0 16384 to 32767 byte frame
0 > 32768 byte frames 0 > 32768 byte frames
0 Late collision frames 0 SymbolErr frames
0 Excess Defer frames 0 Collision fragments
0 Good (1 coll) frames 0 ValidUnderSize frames
0 Good (>1 coll) frames 0 InvalidOverSize frames
0 Deferred frames 0 ValidOverSize frames
0 Gold frames dropped 0 FcsErr frames
0 Gold frames truncated
0 Gold frames successful
0 1 collision frames
0 2 collision frames
0 3 collision frames
0 4 collision frames
0 5 collision frames
0 6 collision frames
0 7 collision frames
0 8 collision frames
0 9 collision frames
0 10 collision frames
0 11 collision frames
0 12 collision frames
0 13 collision frames
0 14 collision frames
0 15 collision frames
0 Excess collision frames

LAST UPDATE 5246 msecs AGO

[MHM Cisco World]

all is OK, there is no issue 
still the traffic pass SVL 
so can you try match hash of all port-channel in all SW, this prevent the frame from asymmetric route and pass svl.

[Rami Ibrahim]

Hello,

 Tomorrow we can have a change request and I will try that. Next, I'm going to reload the standby box if that didn't fix the issue. 

 If the reload also didn't make effect, then We're going for an upgrade as a last resort.

Thanks 

Kind Regards,

[Rami Ibrahim]

Hello, 

 Reloading Standby Core switch didn't solve the issue, and also matching the same ether-channel load-balance algorithm between access & core didn't make any effect. 

 I also tried to disable BUM Opt (no svl l2-bum optimization) and same issue happens. Whenever the link to standby core is up from access sw side we experience drops in DHCP broadcast packets. 

 I am getting crazy because flushed the ARP on one PC and ping the GW again only when link to standby is running and all works fine. it seems Broadcast is flooded normally on standby core sw it is just the DHCP discover broadcast packets getting dropped. 

We will go for an upgrade after 2 weeks as per the management decision.

Thanks 

Best regards,

[Rajan R]

Hi Rami,

We are facing the exact same issue on the same version. What happened after this ?