10-22-2017 08:12 AM - edited 03-08-2019 12:27 PM
Hi all,
I assume this is not an easy at least if you are not a real expert.
Background information: We have real high cpu now for about 3 weeks on our 3750X-Stack with 6 switches. The high CPU came from out of nowhere. It is about 70% most of the time and 25-30% are from the Hulc DAI process. However we do not have DAI activated. After a little deep diving I found a way to find the cause of it.
"show controllers cpu-interface" shows a strong growing routing-protocol Queue. That is strange as the stack isn't a router anymore for about one year now. Before it was routing between different VLANs but that role was migrated to a 4500X VSS.
"debug platform cpu-queues routing-protocol-q"brings really a lot of - if not a flood - of messages like these:
021559: Oct 20 16:31:48.017: RT-Q:DAI Consume: Remote Port Blocked L3If:Vlan1 L2If:GigabitEthernet2/0/37 DI:0xAC, LT:1, Vlan:1 SrcGPN:93, SrcGID:93, ACLLogIdx:0x0, MacDA:ffff.ffff.ffff, MacSA: 0cc4.7aa9.f2ea ARP: 00010800_06040001_0CC47AA9_F2EA0A02_96050000_00000000_0A029701
TPFFD:CDC0005D_00010001_00800040-000000AC_1C1C0000_00000000
021560: Oct 20 16:31:48.017: RT-Q:DAI Consume: Local Port Fwding L3If: L2If:TenGigabitEthernet5/1/2 DI:0xFC, LT:1, Vlan:405 SrcGPN:584, SrcGID:584, ACLLogIdx:0x0, MacDA:ffff.ffff.ffff, MacSA: 901b.0ec0.848a ARP: 00010800_06040001_901B0EC0_848A0A0A_05C80000_00000000_0A0A0525
TPFFD:CD800248_01950195_00880040-000000FC_17170000_00000000
Now we see here the "RT-Q:DAI" which I assume is the root cause of the problem. The messages mention different VLANs and different uplinks while most of them show "Local Port Fwding", some of the "Remote Port Blocked".
I have no idea where the packets come from. I mean I followed one up to it's source port (according to the MAC address in the log message) but could not see why such a packet should be generated there.
Any help would be greatly appreciated.
Thanks
Daniel
Solved! Go to Solution.
04-14-2020 03:22 AM
Hi,
I just came across with the same logs when I was trying to do the lab repro for one of my customer. These logs are seen for the DAI process (DAI would enable automatically in case you have 802.1x configured or you have IPDT enabled) which is receiving the ARP broadcast packets from Gig 2/0/37 and Ten 5/1/2. These packets would have been seen in the stack while the stack master would have been the switches other than switch 2 and switch 5. Packets received from the other stack member to the master would be using the routing protocol queue.
04-14-2020 03:22 AM
Hi,
I just came across with the same logs when I was trying to do the lab repro for one of my customer. These logs are seen for the DAI process (DAI would enable automatically in case you have 802.1x configured or you have IPDT enabled) which is receiving the ARP broadcast packets from Gig 2/0/37 and Ten 5/1/2. These packets would have been seen in the stack while the stack master would have been the switches other than switch 2 and switch 5. Packets received from the other stack member to the master would be using the routing protocol queue.
09-15-2020 11:12 PM
Hi,
thanks for getting back to this. I just saw your response. Meanwhile we have replaced the 3750X stack with some Catalyst 9300 so the initial issue has resolved itself somehow. But what you write makes sense, we already had 802.1x in place when the issue occurred so thanks for clarifying.
09-15-2020 11:27 PM
Thanks for your feedback. I am sure you would enjoy working on C9300 which do help you explore more on your programming skills.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide