DAI disabled - High CPU RT-Q

[the_kirschi]

Hi all,

I assume this is not an easy at least if you are not a real expert.

Background information: We have real high cpu now for about 3 weeks on our 3750X-Stack with 6 switches. The high CPU came from out of nowhere. It is about 70% most of the time and 25-30% are from the Hulc DAI process. However we do not have DAI activated. After a little deep diving I found a way to find the cause of it.

"show controllers cpu-interface" shows a strong growing routing-protocol Queue. That is strange as the stack isn't a router anymore for about one year now. Before it was routing between different VLANs but that role was migrated to a 4500X VSS.

"debug platform cpu-queues routing-protocol-q"brings really a lot of - if not a flood - of messages like these:

021559: Oct 20 16:31:48.017: RT-Q:DAI Consume: Remote Port Blocked L3If:Vlan1 L2If:GigabitEthernet2/0/37 DI:0xAC, LT:1, Vlan:1   SrcGPN:93, SrcGID:93, ACLLogIdx:0x0, MacDA:ffff.ffff.ffff, MacSA: 0cc4.7aa9.f2ea   ARP: 00010800_06040001_0CC47AA9_F2EA0A02_96050000_00000000_0A029701

   TPFFD:CDC0005D_00010001_00800040-000000AC_1C1C0000_00000000

021560: Oct 20 16:31:48.017: RT-Q:DAI Consume: Local Port Fwding L3If: L2If:TenGigabitEthernet5/1/2 DI:0xFC, LT:1, Vlan:405   SrcGPN:584, SrcGID:584, ACLLogIdx:0x0, MacDA:ffff.ffff.ffff, MacSA: 901b.0ec0.848a   ARP: 00010800_06040001_901B0EC0_848A0A0A_05C80000_00000000_0A0A0525

   TPFFD:CD800248_01950195_00880040-000000FC_17170000_00000000

Now we see here the "RT-Q:DAI" which I assume is the root cause of the problem. The messages mention different VLANs and different uplinks while most of them show "Local Port Fwding", some of the "Remote Port Blocked".

I have no idea where the packets come from. I mean I followed one up to it's source port (according to the MAC address in the log message) but could not see why such a packet should be generated there.

Any help would be greatly appreciated.

Thanks

Daniel

[virparma · Accepted solution]

Hi, 

I just came across with the same logs when I was trying to do the lab repro for one of my customer. These logs are seen for the DAI process (DAI would enable automatically in case you have 802.1x configured or you have IPDT enabled) which is receiving the ARP broadcast packets from Gig 2/0/37 and Ten 5/1/2. These packets would have been seen in the stack while the stack master would have been the switches other than switch 2 and switch 5. Packets received from the other stack member to the master would be using the routing protocol queue. 

[the_kirschi]

Hi,

thanks for getting back to this. I just saw your response. Meanwhile we have replaced the 3750X stack with some Catalyst 9300 so the initial issue has resolved itself somehow. But what you write makes sense, we already had 802.1x in place when the issue occurred so thanks for clarifying.

[virparma]

Thanks for your feedback. I am sure you would enjoy working on C9300 which do help you explore more on your programming skills.