IHQ constant packets in incoming queue

[Chrizhitz]

Good morning

I am facing the following problem on a customer location with main and backup router. cisco ISR4331/K9

We see constant incoming IHQ packets on the WAN Interfaces on both routers. At the same time there is almost no traffic on the WAN Interface Gi0/0/2 on the standby router. 

we lost TACACS connection to the main router and "fixed" it by increasing the hold-que in to 2000 pakets. I do not see any other irregularities like high cpu load or massive packet counts.

has anyone encountered the same problem or can tell me in which direction I have to investigate.

sh interface summary (see attached doc)

thanks in advance

Chris

[marce1000]

 - Check software version being used on the ISR and or use latest advisory (if applicable) ; check if that can help , 

 M.

[Chrizhitz]

Hi marce1000

we are currently using isr4300-universalk9.03.16.03.S.155-3.S3-ext.SPA.bin

I am not able to change iOS unless it has been evaluated from Service Development Engineering .

Are you aware of any iOS Problems which are causing this behavior.

thanks 

Chris

[marce1000]

 - Not exactly , but such issues should be evaluated against a (the) advisory release , if it is all possible. Note that on Cisco software downloads site , if the release is gold-starred it's considered to be safe to use (too) , 

 M.

[Georg Pauwen]

Hello,

post the output of:

show buffers

show interfaces GigabitEthernet0/0/2

PS: Use a .txt file to post the output, as not everybody has a paid Word subscription.

[Chrizhitz]

Hello Georg

Please find attached the requested outputs

thanks Chris

[Joseph W. Doherty]

Is the WAN connection Internet?  If so, running BGP?

[Chrizhitz]

VPN not Internet but we are using BGP

[Joseph W. Doherty]

---

@Chrizhitz wrote:

VPN not Internet but we are using BGP

---

Ah, then likely not what I had in mind.

IHQ, in my experience, is rare.  Assuming you're not dealing with some bug, it can be caused by a bunch of packets, that arrive quickly, which the device NIC is able to queue, but the device is "slow" emptying the ingress queue.

Your router is just a 4331 supporting gig interfaces, and the 4331 is far, far from being a gig capable router.

I notice in the second stat posting, g0/0/0 is also showing 124 packets in its ingress queue.

So, it's very possible, some gig bursts are delayed processed by the CPU.

Personally, I've seen this with Internet BGP route updates, that "flood" the receiving the device, but it's unable to keep up with the burst.

Sometimes, mitigation is as simple as increasing ingress queue size to handle the burst, to avoid packet drops.  (Which if I'm reading your OP correctly, "fixed" the TACAS issue.)

I'm surprised that you note CPU doesn't show high load.  For that, you mean you CPU history doesn't shown any 100% spikes?  I.e. it doesn't have to be a sustained overall high average load.

Since you increased ingress queue depth, having any other issues?

[Chrizhitz]

Yes, very rare. I think I only saw it in combination with the 4331. Meanwhile the constant input increased to 412 packets.

Main Router

GigabitEthernet0/0/2 412 0 0 41029 1033000 879 21028000 1869 0

Backup Router Gig0/0/0 now 138 Packets / Gig0/0/2 still 368

GigabitEthernet0/0/0 138 0 0 0 6000 11 2000 4 0 

GigabitEthernet0/0/2 368 0 0 0 0 1 5000 4 0

Yes CPU history does not show any dramatic spikes the 72 hours average is somewhat around 10%

To me it looks like some sort of loop. Could it be useful to configure site-of-origin?

No further issues observed since I increased the ingress queue depth.

[Joseph W. Doherty]

BTW, what's the capacity license on this 4331?

I'm wondering, as overrunning license capacity shapes, whether that might queue ingress traffic.

[Chrizhitz]

Main Router#sh platform hardware throughput level
Load for five secs: 0%/0%; one minute: 1%; five minutes: 1%
Time source is NTP, 07:07:04.189 CEST Fri May 10 2024
The current throughput level is 300000 kb/s

Two shaping policies of 212mbps and 21mbps (which of course is not optimal but should in my opinion not be the reason for this problem)

Main Router#sh platform hardware throughput-license-enforcement
Load for five secs: 1%/0%; one minute: 1%; five minutes: 1%
Time source is NTP, 07:07:52.124 CEST Fri May 10 2024
Throughput License Mode: Oversubscription-buffered

[Joseph W. Doherty]

Again, don't know "where" capacity shaping is queued.

Consider, two gig interfaces can submit 2 Gbps but you're limited to 300 Mbps throughput.  The 300 Mbps, depending on traffic "kind" and config, may require little CPU loading.  I.e. low CPU usage does not exclude this as a possible issue.

Also consider, the time needed to handle 368 packets at "only" 300 Mbps.  If all were 1500 bytes, that would be 4,416,000 bits, and at 300 Mbps, it would take about 15 ms to process.

What would be an interesting test, if possible, is run with a "boost" (evaluation) license, and see if traffic still is shown in ingress queues.

[Chrizhitz]

on the standby Router we also do have a constant IHQ of 159 but traffic is only some 2000 bits/sec (RXBS and TXBS)

no boost license available.

another "fun fact" when I admin shut the gig0/0/0 (LAN) the IHQ remains 159 (clear counters done) Don't know if this is a feature or a bug.

however I am going to reload the standby Router to see if this is going to change anything.

[Chrizhitz]

21 minutes after reload of Standby Router

IHQ on both Gig Interfaces 0

Main Router Gig 0/0/2 still IHQ 450 (no reload)