cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
3277
Views
35
Helpful
16
Replies

Many Output Queue drops and 255 txload 255 rxload on ASR1002

N3t W0rK3r
Level 3
Level 3

We're troubleshooting an issue with our Internet performance.  We are provisioned with a 1Gbps symmetrical link from our ISP and currently the ASR outside interface (gi0/0/0) shows around 30% utilization:

 

INTERNET-ROUTER#sh int gi0/0/0
GigabitEthernet0/0/0 is up, line protocol is up
Hardware is 4XGE-BUILT-IN, address is 001f.ca15.ed00 (bia 001f.ca15.ed00)
Description: OUTSIDE TO ISP
Internet address is a.b.c.d
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 49/255, rxload 61/255
Encapsulation ARPA, loopback not set
Keepalive not supported
Full Duplex, 1000Mbps, link type is auto, media type is T
output flow-control is on, input flow-control is on
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:11, output 00:01:24, output hang never
Last clearing of "show interface" counters never
Input queue: 0/375/2497/1030 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 239675000 bits/sec, 49780 packets/sec
5 minute output rate 193607000 bits/sec, 44036 packets/sec
108538977078 packets input, 55832140065374 bytes, 0 no buffer
Received 326948 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
99430287225 packets output, 38558895093668 bytes, 0 underruns
0 output errors, 0 collisions, 3 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

 

Now looking at the inside interface (Gi0/0/3) we see problems with txload, rxload and output queue drops:

 

INTERNET-ROUTER#sh int gi0/0/3
GigabitEthernet0/0/3 is up, line protocol is up
Hardware is 4XGE-BUILT-IN, address is 001f.ca15.ed03 (bia 001f.ca15.ed03)
Description: INSIDE TO ASA
Internet address is e.f.g.h
MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,
reliability 255/255, txload 255/255, rxload 255/255
Encapsulation ARPA, loopback not set
Keepalive not supported
Full Duplex, 1000Mbps, link type is force-up, media type is T
output flow-control is on, input flow-control is on
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:01, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 800781112
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 110959000 bits/sec, 28364 packets/sec
5 minute output rate 231570000 bits/sec, 39173 packets/sec
50404755765 packets input, 21397119353119 bytes, 0 no buffer
Received 398901 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 2940620 multicast, 0 pause input
65949758049 packets output, 56288732167788 bytes, 0 underruns
0 output errors, 0 collisions, 6 interface resets
270970 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

 

So my question is, txload/rxload seems to indicate the interface is saturated but the input rate/output rate doesnt indicate anything close to the 1 Gbps link speed capable.  How is this possible?  Is this a bug or a defect in the hardware?

 

Our ASR 1002 is running version 03.16.10.S.

 

Our observations on speed tests run using Speedtest.net is that our download speed is often a third (or more) of our upload.

 

Thanks in advance for any light you can shed on this issue.

1 Accepted Solution

Accepted Solutions

Well, we had our maintenance window last night and tried changing the neighboring ports to auto/auto.  Although the ports came back successfully, this did not correct the bandwidth reported issue, the output drops or improve the speed test results.

We then provisioned a new port, with a new SFP on the 5-port SPA card with a new patch cable.  This worked perfectly in all aspects.

Out of curiosity, we replaced the patch cable with the original cable and observed the same positive results across the board (1 Gbps reported, 0 output drops and almost 900 Mbps download in speed test).

So, it looks like perhaps the problem was with the SFP or SPA port itself.

 

Thanks again for all of your input.  I am relieved to finally put this behind me!

 

Cheers and stay safe.

 

View solution in original post

16 Replies 16

Naresh Murali
Cisco Employee
Cisco Employee

Hi,

 

I just verified your output. It seems you have set the Bandwidth on the interface level. 

 

From the output you have attached:

INTERNET-ROUTER#sh int gi0/0/0
GigabitEthernet0/0/0 is up, line protocol is up
Hardware is 4XGE-BUILT-IN, address is 001f.ca15.ed00 (bia 001f.ca15.ed00)
Description: OUTSIDE TO ISP
Internet address is a.b.c.d
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,   >>> Here if you look the BW is 1000000 Kbit/sec
reliability 255/255, txload 49/255, rxload 61/255
Encapsulation ARPA, loopback not set
Keepalive not supported
Full Duplex, 1000Mbps, link type is auto, media type is T

 

However if you look the problematic interface its different:

 

INTERNET-ROUTER#sh int gi0/0/3
GigabitEthernet0/0/3 is up, line protocol is up
Hardware is 4XGE-BUILT-IN, address is 001f.ca15.ed03 (bia 001f.ca15.ed03)
Description: INSIDE TO ASA
Internet address is e.f.g.h
MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,   >>> Here is it 100000 Kbit/sec
reliability 255/255, txload 255/255, rxload 255/255
Encapsulation ARPA, loopback not set
Keepalive not supported
Full Duplex, 1000Mbps, link type is force-up, media type is T

 

Check if you have give any BW command on the interface level. Removing that should resolve your issue. However make sure the other side also has the same configuration. From the output you attached seems to be a normal behavior as the BW is less compared to the other interface.

 

Hope this helps.

 

Naresh M

Naresh,

 

Thank you for your keen observation. But this actually makes things even more confusing because there is no bandwidth command on that interface.

Here are the two adjacent ports' configs, FYI:

 

ASR1002

interface GigabitEthernet0/0/3
description INSIDE ASA
ip address a.b.c.d 255.255.255.0
ip nat inside
ip route-cache policy
ip policy route-map GUEST-Internet
speed 1000
no negotiation auto
no mop enabled
end

 

C3750X Switch

interface GigabitEthernet1/0/25
description INTERNET ASR 64
switchport access vlan 64
switchport mode access
speed 1000
duplex full
end

 

 

Hi  N3t W0rK3r,

 

Thank you for providing the show run interface output.

 

There is one more thing which I noticed. May I why the auto negotiation was removed?

 

I could see that the link type is forced:

 

Full Duplex, 1000Mbps, link type is force-up, media type is T

 

Also auto negotiation is disabled:

 

interface GigabitEthernet0/0/3
speed 1000
no negotiation auto
no mop enabled

 

In 3750 you have manually give the speed and duplex:

 

interface GigabitEthernet1/0/25
description INTERNET ASR 64
switchport access vlan 64
switchport mode access
speed 1000
duplex full

 

Auto negotiation did not work on 3750?

 

May I know whats the running config of Gi0/0/0 that will give a better idea as i see the auto negotiation is on with respect to the output. Also does the interface connected to a 3750 as well?

 

I understand that you disabled auto negotiation on ASR to enable the speed command.

 

Additionally Can you add the duplex full command and check if that increase the bandwidth?

 

Regards

Naresh 

 

Naresh,

 

This ASR was recently put in to replace a lower performing router as we upgraded our Internet bandwidth to support a surge in remote workers.  At the time of the cutover, we had some problems with one of the interfaces and I think the omission of the duplex full on the one interface was just an oversight.  Next week we will plan a maintenance window and try to set both sides to auto/auto and see what happens.


Thanks.

Hi N3t W0rK3r,

 

Thanks for the reply.

If the Auto negotiation works, you should not be facing any issue after that.

Also keep in mind when changing speed or other factors make sure it is configured on both the sides.

Let me know how it went after the MW.

 

Regards

Naresh M

Giuseppe Larosa
Hall of Fame
Hall of Fame

Hello @N3t W0rK3r ,

let us focus on the output drops as the txload and rxload might be a "cosmetic bug",

Looking at show interface gi0/0/3 we see that:

there are 800781112 output drops

These have to be compare to the total output packets = output drops + output packets.

The output packets are 65949758049 packets output.

The drop probability is roughly  1,2 % that is on the border of acceptable.

Being queuiing FIFO it looks like no QoS is applied to this interface.

However, flow control is enabled:

>> output flow-control is on, input flow-control is on

 

So these output drops may be increased / caused by the fact of receiving IEEE pause frames from the downstream device that attempts to slow down the sending in this way to avoid to have input errors of type ignored / overrun ( in cisco terms)

 

Check if the downstream device is sending IEEE pause frames.

 

Hope to help

Giuseppe

 

Hello Giuseppe,

 

Thank you for your reply.

 

On the adjacent 3750X switchport, input flowcontrol is off and output is unsupported.  Or should I be looking at the ASA interface also connected to the same switch? I just checked and it also is not configured/supported for flowcontrol.

 

Since the media is copper, do you think this may be an issue with one of the pairs?

 

 

Hello @N3t W0rK3r ,

I don't think there is a cabling issue otherwise the error ratio would be higher and we could see other type of errors also in input like runt ( too small frames ).

At this point either the traffic is very burstly or the two GE ports are managed by the same ASIC but in an ASR 1002 even in this case I would not expect performance issues.

 

Hope to help

Giuseppe

 

As Giuseppe correctly notes, you're drop rate is just a tad over the typically recommended threshold of 1%, but what you could probably safely do is increase your input and egress queue sizes. Cisco's default of 40 for egress is generally suitable for 10 Mbps LAN Ethernet or T-1 WAN, so for gig you could increase by quite a bit. (Start by doubling egress queue size, and monitoring results. What you're looking for in a dramatic decrease in drop rate while not detecting a jump in latency. [You might also see your average bandwidth load increase and/or an increase in your Speedtest results.])

The default for ingress doesn't always allow for bursts that peg the CPU. It's generally safe to increase that to allow the CPU to catch-up with ingress packet bursts rather than dropping them.

Hi Joseph,

 

Thank you for this suggestion.  We will definitely give this a try. Do you expect the application of the hold-queue command to the interface will impact service at all?

 

One question that came with up my colleagues as we discussed this is that we are never seeing the output queue size value change from 0 at all with repeated show int commands.  If the queue size is in fact an issue, wouldn't we be seeing non-zero values there more frequently?  Just a thought?


Thanks again.

 

A possible issue of using increasing the size of a queue, you'll also increase the changes some traffic will have additional latency. Or course, the alternative is dropping the traffic. Generally, but not always, an occasional increase in latency is worth a occasional decrease in drops.

As to your question about not seeing the interface with queued traffic (assuming there's no IOS bug), the queue is only 40 packets deep, so it takes a very brief time to fill it, overflow it and drain the 40 packets at gig rate. With occasional bursty traffic, it can be difficult, in this situation, to "catch" seeing the queue with packets queued.

Hello,

 

on a side note, you might want to try some basic QoS:

 

policy-map SHAPE_ALL
shape average 1000000000
class class-default
fair-queue
!
interface GigabitEthernet0/0/0
service-policy output SHAPE_ALL
!
interface GigabitEthernet0/0/3
service-policy output SHAPE_ALL

 

Also, I think the ASR supports the global command 'qos queue-softmax-multiplier 1200', if so, you might want to give that a try as well.

BTW, whenever you're using QoS with an interface with its full native bandwidth, there shouldn't be any advantage to using a shaper.

N3t W0rK3r
Level 3
Level 3

Thank you everyone for your responses to this post.

We will be having a maintenance window on Sunday evening to address this issue.  Hopefully we can get it sorted out within the hour.

I will report back on Monday with the outcome.

Cheers.

 

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: