cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
3739
Views
0
Helpful
10
Replies

GRE Tunnel Output Queue Drops

joet8591
Level 1
Level 1

Running a GRE tunnel between two 3845s with full DS3s. DS3/tunnel only running at around 8Mbps and picking up significant "output drops" on the Tunnel interface but not on the DS3 which is carrying the tunnel. Trying to see what these drops represent. Not seeing any problems with Apps but may want to run voice and need to understand these drops. Standard GRE tunnel with FIFO. No IPSec.

Thanks

Regards

Joe

10 Replies 10

royalblues
Level 10
Level 10

What is the IOS version you are running. This may be a cosmetic bug

Also what is the bandwidth configured under the tunnel interface?

By default the tunnel assumes a bandwidth of just 9K and hence can show drops. Make sure you have the correct bandwidth configured under the tunnel interface

Narayan

joet8591
Level 1
Level 1

3845 Enterprise Version 12.4(12)

Bandwidth is 44210 to reflect the DS3.

Tunnel1 is up, line protocol is up

Hardware is Tunnel

Internet address is 10.14.249.97/30

MTU 1514 bytes, BW 44210 Kbit, DLY 500000 usec,

reliability 255/255, txload 21/255, rxload 18/255

Encapsulation TUNNEL, loopback not set

Keepalive not set

Tunnel source 1.x.x.x (Serial1/0.1), destination 2.x.x.x

Tunnel protocol/transport GRE/IP

Key disabled, sequencing disabled

Checksumming of packets disabled

Tunnel TTL 255

Fast tunneling enabled

Tunnel transmit bandwidth 8000 (kbps)

Tunnel receive bandwidth 8000 (kbps)

Last input 00:00:04, output 00:00:04, output hang never

Last clearing of "show interface" counters 09:35:07

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 122496

Queueing strategy: fifo

Output queue: 0/0 (size/max)

30 second input rate 3125000 bits/sec, 1624 packets/sec

30 second output rate 3734000 bits/sec, 1523 packets/sec

7059033 packets input, 1565205618 bytes, 0 no buffer

Received 0 broadcasts, 0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

5990442 packets output, 2562730255 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 output buffer failures, 0 output buffers swapped out

Thanks

Joe

Joe,

From your output

Tunnel transmit bandwidth 8000 (kbps)

Tunnel receive bandwidth 8000 (kbps)

This should show the DS3 bandwidth

Can you post the interface configuration

As said earlier, it could also be a cosmetic bug.

HTH

Narayan

Joseph W. Doherty
Hall of Fame
Hall of Fame

What's your CPU look like when you peak out at 8 Mbps?

[edit]

Do you use the ip tcp mss-adjust?

CPU averages 10% regardless of load and CPU history shows no spikes.

I am trying to look for commands for any fragmentation/reassembly etc....

Show IP traffic is for the whole router. Checking the code for any cosmetic bugs.

One thing I omitted is that the Serial carrying the tunnel has a policy applied but shows no drops.

interface Serial1/0

no ip address

encapsulation frame-relay

load-interval 30

dsu bandwidth 44210

scramble

serial restart-delay 0

service-policy output WAN_EDGE_OUT

class-map match-any SERVICE_CONTROL

match ip dscp af31

match ip dscp cs3

class-map match-any NETWORK_CONTROL

match ip dscp cs6

class-map match-all REAL_TIME

match ip dscp ef

class-map match-any BULK_DATA

match ip dscp af11

!

!

policy-map WAN_EDGE_OUT

class REAL_TIME

priority percent 34

class SERVICE_CONTROL

bandwidth percent 5

class NETWORK_CONTROL

bandwidth percent 3

class BULK_DATA

bandwidth percent 33

class class-default

fair-queue

queue-limit 350

interface Tunnel1

Tunnel1

bandwidth 44210

ip address 10.

ip mtu 1400

ip tcp adjust-mss 1400

load-interval 30

tunnel source Serial1/0.1

tunnel destination

Thanks Again

Joe

Reason I asked about CPU, believe GRE is often (always?) not in the fast switch path.

I too have seen drops on GRE tunnels, when they shouldn't be stressed. Never discovered exactly why. Have suspected during bursts CPU drops some. I've also tried increasing outbound queue on tunnel, but seems to make little difference.

PS:

I wonder about:

interface Tunnel1

Tunnel1

bandwidth 44210

ip address 10.

ip mtu 1400

ip tcp adjust-mss 1400

load-interval 30

tunnel source Serial1/0.1

tunnel destination

Shouldn't mss be at least 40 bytes less than mtu?

Why mtu so small? I don't recall GRE using 100 bytes.

PPS:

Also recall there's a dynamic MTU probe you can configure for tunnel; don't recall the exact command.

Several problems here.

First, you don't want to adjust the ip mtu at all, particularly on a GRE tunnel. You should only be changing the ip tcp adjust-mss to 1460 or 1380. By changing both the ip mtu and adjust-mss to the same value, you broke tcp across this link because you rendered useless the benefit of tcp adjust-mss on this interface. Also, depending on IOS version and hardware platform, you may need the tcp adjust-mss statement on the ingress interface of the router, not the tunnel.

Second, never set the queue-limit to anything greater than 100 packets, particularly as a part of a service-policy. Remember, every class has a 64 packet queue-limit, so the total hold-queue for the interface will be the sum of all the class queue-limits. By making this number crazy high, you are essentially allowing duplicate packets to get into the hold-queue. In other words, by the time the tcp sender timed out and resent the packet, the first packet may not have been sent!

This is bad, the tcp mss-adjust misconfiguration only made it worse.

Richard, you make a great point about not setting the tcp adjust-mss value to the same value as ip mtu, but my thinking is a bit different on some of your other points, so thought I present them for your feedback.

First, about not wanting to set ip mtu at all, as I mentioned in my prior post, I would lean toward using the dynamic tunnel mtu probe(, using the command tunnel path-mtu-discovery [if supported]). Both, however, are recommended by Cisco in the following whitepaper http://www.cisco.com/en/US/tech/tk827/tk369/technologies_white_paper09186a00800d6979.shtml#more, in the "More Recommendations" section (near the bottom of the document). Could you expand upon why not to use ip mtu?

On your second point about never setting queue-limit greater than 100 packets, and that every class has a (default) 64 packet limit, and the interface total would be the sum of all, you make another great point about TCP needlessly resending packets that have timed out, but are actually still sitting in a queue.

(Item to note, most platforms CBWFQ classes do default to 64 packets, but not all do. I know the 7500s don't. I suspect the 6500/7600 FlexWAN or SIP-200 may not either, since they appear to support 7500 QoS.)

TCP's RTO (retranmission timeout) should be dynamically adjusted while it's measuring RTT (round-trip time). So, TCP might not send lots of needless packet retransmissions just because the packet is sitting in a deep queue, although I too think it is more likely, especially if the active queue depth is very variable.

Sizing overall queue buffer depth, in packets, should depend on link's bandwidth ("faster" can drain a larger packet queue in the same time), whether BDP (bandwidth delay product) is to be supported, and expected packets sizes (e.g. VoIP packets generally are smaller than typical FTP packets). So, I believe proper sizing queue parameters can be a bit complex, but unsure providing or exceeding some total value, such as never greater than 100, is always bad or "crazy". Is there a particular reason for limiting overall queue depth to 100 or less in all cases, or do you really mean to emphasis that queues can be sized too large? (If the latter, I agree; there's also sometimes the consideration of buffer resources available on the platform.)

Are you running GRE inside an IPSEC tunnel? Your configurations posted so far do not reflect this. That would be an entirely different scenerio.

If not, and you are simply running a GRE tunnel between these two endpoints, and the end-to-end ip mtu OUTSIDE the tunnel is 1500 bytes (the tunnel's ip path) then you do not (and must not) change the ip mtu under the tunnel interface. The document you reference only mentions changing the default ip mtu on the tunnel in two scenerios: 1) the path mtu of the tunnel is less than 1500 bytes (ipsec, pppoe, etc.) or 2) the entire network supports large frames and can accept GRE packets of 1500 bytes (and therefore GRE/IP frames of 1524 bytes. In either case, this should be deterministic and incorporated into your design so should be hard set and not rely on path-mtu discovery, which rarely works outside the gre tunnel.

You do, however, need to use tcp adjust-mss on the ingress interface of this router, using this command on the tunnel does not always work depending on IOS version. I use a value of 1380 but values up to 1460 should work. This is why certain tcp transactions will fail across the tunnel, udp packets are rarely larger than 1200 bytes and work by default.

Regarding the inbound hold-queue size, you will not find *any* document on Cisco's web site that lists an inbound hold-queue of anything greater than 100, particularly if using CBWFQ. Ignore any documents that are more than 4 years old or refer to modems, isdn, etc.

Usually, you will set the hold-queue or CBWFQ queue-limit *SMALLER* to more aggressively tail-drop traffic in the default class so that tcp makes the right decision about bandwidth.

Remember, the input hold-queue does not drain at all unless the output interface tx-limit and tx queue are nearly empty. IOS puts back-pressure on the input interface when the output interface is saturated and you want the input interface to tail-drop, not buffer a bunch of packets unrealistically.

Richard, again you're quite correct where IP MTU shouldn't need to be configured on a pure GRE tunnel supporting and end-to-end MTU of 1500 bytes (assuming standard Ethernet?), although you now note a couple of exceptions. It was your usage of "at all", in your earlier post's, "First, you don't want to adjust the ip mtu at all, particularly on a GRE tunnel.", I thought the statement might be somewhat misleading, i.e. that there weren't any cases where the command should be used. I agree with "hard set" in what you now write "In either case, this should be deterministic and incorporated into your design so should be hard set . . .". (Another instance I could see using it [IP MTU], would be on the LAN side interface of a WAN router which only supports traffic to/from a GRE tunnel.)

In cases of running GRE tunnels over something like the Internet, I'm unsure I agree with not using tunnel path-mtu-discovery since often the end-to-end path MTU is never guaranteed. (I do agree PMTU, itself, can be an issue.)

I'm a bit lost about your references to "inbound" hold-queues, unless you meant outbound.

I disagree with making outbound queues "*SMALLER*" to aggressively tail-drop so that TCP makes the right decision about bandwidth. Just as you're correct, I believe, to be concerned about oversizing queues; making them too small can have a very adverse impact with TCP performance. Too small queues will often not allow TCP flows to correctly determine and utilize bandwith. Lots of tail drops, besides also causing retransmissions, can often force a TCP flow back into slow start instead of just congestion avoidance.

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: