topic Re: IHQ constant packets in incoming queue in Switching

IHQ constant packets in incoming queue

Chrizhitz — Wed, 08 May 2024 07:37:37 GMT

Good morning

I am facing the following problem on a customer location with main and backup router. cisco ISR4331/K9

We see constant incoming IHQ packets on the WAN Interfaces on both routers. At the same time there is almost no traffic on the WAN Interface Gi0/0/2 on the standby router.

we lost TACACS connection to the main router and "fixed" it by increasing the hold-que in to 2000 pakets. I do not see any other irregularities like high cpu load or massive packet counts.

has anyone encountered the same problem or can tell me in which direction I have to investigate.

sh interface summary (see attached doc)

thanks in advance

Chris

Re: IHQ constant packets in incoming queue

Mark Elsen — Wed, 08 May 2024 08:38:18 GMT

- Check software version being used on the ISR and or use latest advisory (if applicable) ; check if that can help ,

Re: IHQ constant packets in incoming queue

Chrizhitz — Wed, 08 May 2024 08:46:10 GMT

Hi marce1000

we are currently using isr4300-universalk9.03.16.03.S.155-3.S3-ext.SPA.bin

I am not able to change iOS unless it has been evaluated from Service Development Engineering .

Are you aware of any iOS Problems which are causing this behavior.

thanks

Chris

Re: IHQ constant packets in incoming queue

Mark Elsen — Wed, 08 May 2024 08:57:13 GMT

- Not exactly , but such issues should be evaluated against a (the) advisory release , if it is all possible. Note that on Cisco software downloads site , if the release is gold-starred it's considered to be safe to use (too) ,

Re: IHQ constant packets in incoming queue

Georg Pauwen — Wed, 08 May 2024 10:20:03 GMT

Hello,

post the output of:

show buffers

show interfaces GigabitEthernet0/0/2

PS: Use a .txt file to post the output, as not everybody has a paid Word subscription.

Re: IHQ constant packets in incoming queue

Chrizhitz — Wed, 08 May 2024 11:33:03 GMT

Hello Georg

Please find attached the requested outputs

thanks Chris

Re: IHQ constant packets in incoming queue

Joseph W. Doherty — Wed, 08 May 2024 14:01:01 GMT

Is the WAN connection Internet? If so, running BGP?

Re: IHQ constant packets in incoming queue

Chrizhitz — Wed, 08 May 2024 14:06:11 GMT

VPN not Internet but we are using BGP

Re: IHQ constant packets in incoming queue

Joseph W. Doherty — Wed, 08 May 2024 17:15:11 GMT

@Chrizhitz wrote:

VPN not Internet but we are using BGP

Ah, then likely not what I had in mind.

IHQ, in my experience, is rare. Assuming you're not dealing with some bug, it can be caused by a bunch of packets, that arrive quickly, which the device NIC is able to queue, but the device is "slow" emptying the ingress queue.

Your router is just a 4331 supporting gig interfaces, and the 4331 is far, far from being a gig capable router.

I notice in the second stat posting, g0/0/0 is also showing 124 packets in its ingress queue.

So, it's very possible, some gig bursts are delayed processed by the CPU.

Personally, I've seen this with Internet BGP route updates, that "flood" the receiving the device, but it's unable to keep up with the burst.

Sometimes, mitigation is as simple as increasing ingress queue size to handle the burst, to avoid packet drops. (Which if I'm reading your OP correctly, "fixed" the TACAS issue.)

I'm surprised that you note CPU doesn't show high load. For that, you mean you CPU history doesn't shown any 100% spikes? I.e. it doesn't have to be a sustained overall high average load.

Since you increased ingress queue depth, having any other issues?

Re: IHQ constant packets in incoming queue

Chrizhitz — Thu, 09 May 2024 15:30:08 GMT

Yes, very rare. I think I only saw it in combination with the 4331. Meanwhile the constant input increased to 412 packets.

Main Router

GigabitEthernet0/0/2 412 0 0 41029 1033000 879 21028000 1869 0

Backup Router Gig0/0/0 now 138 Packets / Gig0/0/2 still 368

GigabitEthernet0/0/0 138 0 0 0 6000 11 2000 4 0

GigabitEthernet0/0/2 368 0 0 0 0 1 5000 4 0

Yes CPU history does not show any dramatic spikes the 72 hours average is somewhat around 10%

To me it looks like some sort of loop. Could it be useful to configure site-of-origin?

No further issues observed since I increased the ingress queue depth.

Re: IHQ constant packets in incoming queue

Joseph W. Doherty — Thu, 09 May 2024 16:45:10 GMT

BTW, what's the capacity license on this 4331?

I'm wondering, as overrunning license capacity shapes, whether that might queue ingress traffic.

Re: IHQ constant packets in incoming queue

Chrizhitz — Fri, 10 May 2024 05:12:22 GMT

Main Router#sh platform hardware throughput level
Load for five secs: 0%/0%; one minute: 1%; five minutes: 1%
Time source is NTP, 07:07:04.189 CEST Fri May 10 2024
The current throughput level is 300000 kb/s

Two shaping policies of 212mbps and 21mbps (which of course is not optimal but should in my opinion not be the reason for this problem)

Main Router#sh platform hardware throughput-license-enforcement
Load for five secs: 1%/0%; one minute: 1%; five minutes: 1%
Time source is NTP, 07:07:52.124 CEST Fri May 10 2024
Throughput License Mode: Oversubscription-buffered

Re: IHQ constant packets in incoming queue

Joseph W. Doherty — Fri, 10 May 2024 15:44:52 GMT

Again, don't know "where" capacity shaping is queued.

Consider, two gig interfaces can submit 2 Gbps but you're limited to 300 Mbps throughput. The 300 Mbps, depending on traffic "kind" and config, may require little CPU loading. I.e. low CPU usage does not exclude this as a possible issue.

Also consider, the time needed to handle 368 packets at "only" 300 Mbps. If all were 1500 bytes, that would be 4,416,000 bits, and at 300 Mbps, it would take about 15 ms to process.

What would be an interesting test, if possible, is run with a "boost" (evaluation) license, and see if traffic still is shown in ingress queues.

Re: IHQ constant packets in incoming queue

Chrizhitz — Sat, 11 May 2024 08:43:54 GMT

on the standby Router we also do have a constant IHQ of 159 but traffic is only some 2000 bits/sec (RXBS and TXBS)

no boost license available.

another "fun fact" when I admin shut the gig0/0/0 (LAN) the IHQ remains 159 (clear counters done) Don't know if this is a feature or a bug.

however I am going to reload the standby Router to see if this is going to change anything.

Re: IHQ constant packets in incoming queue

Chrizhitz — Sat, 11 May 2024 09:11:42 GMT

21 minutes after reload of Standby Router

IHQ on both Gig Interfaces 0

Main Router Gig 0/0/2 still IHQ 450 (no reload)

Re: IHQ constant packets in incoming queue

Joseph W. Doherty — Sat, 11 May 2024 09:48:26 GMT

If the count remains the same and occasionally increases, remaining the same, basically behaving like a "high water mark", that may be a bug as I believe it's supposed to show the current ingress queue depth, not the highest seen to date.

Re: IHQ constant packets in incoming queue

Chrizhitz — Sat, 11 May 2024 15:02:11 GMT

after 5 hours 56 minutes no new IHQ packets.

IHQ (if working normal) always shows the current situation on the ingress queue with every refresh.

If there are 0 the IHQ shows 0 and so on.

I agree it behaves like a bug but then how come we lost TACACS access when Ingress queue exceeded approx. 300 packets and we where able to fix it by increasing the hold-queue in?

Re: IHQ constant packets in incoming queue

Joseph W. Doherty — Sat, 11 May 2024 16:14:21 GMT

The possible bug would have been the IHQ never showing lower values than prior values. If that's not happening, I misunderstood what you described as happening.

As to having IHQ non-zero values. even if usages appears low, much can happen in milliseconds that won't be seen in longer term averages.

Generically, if unfamiliar, look up "microburstimg".