Re: Problem with Output drops on Catalyst 6509

burleyman · ‎06-17-2009

I have alot of interfaces that are seeing very high Total output drops and was looking for some direction on what we can do to help this. Here is what I am seeing on one interface....

GigabitEthernet2/3/25 is up, line protocol is up (connected)

Hardware is C6k 1000Mb 802.3, address is 001d.45fc.b8cc (bia 001d.45fc.b8cc)

Description: The NetApp

MTU 9216 bytes, BW 1000000 Kbit, DLY 10 usec,

reliability 255/255, txload 16/255, rxload 1/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full-duplex, 1000Mb/s, media type is 10/100/1000BaseT

input flow-control is on, output flow-control is on

Clock mode is auto

ARP type: ARPA, ARP Timeout 04:00:00

Last input never, output 00:00:55, output hang never

Last clearing of "show interface" counters 17:43:31

Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 9806017

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 0 bits/sec, 0 packets/sec

5 minute output rate 64799000 bits/sec, 1323 packets/sec

1385 packets input, 124522 bytes, 0 no buffer

Received 1197 broadcasts (1063 multicasts)

0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 watchdog, 0 multicast, 0 pause input

0 input packets with dribble condition detected

366307768 packets output, 1640603031216 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier, 0 PAUSE output

0 output buffer failures, 0 output buffers swapped out

Here is the config for that interface...

interface GigabitEthernet2/3/25

description The NetApp

switchport

switchport access vlan 33

switchport mode access

mtu 9216

flowcontrol receive on

flowcontrol send on

spanning-tree portfast

Thanks,

Mike

Giuseppe Larosa · ‎06-17-2009

Hello Mike,

may you add a sh module ?

What type of linecard is this?

supervisor type and IOS image ?

I see g2/3/25 triple index so I wonder what type of linecard is this.

One possible direction to investigate is the fact that you have mtu 9216 this can reduce the size of interface buffers and can be a source of drops.

flowcontrol is enabled but we don't see pause frames sent or received.

Hope to help

Giuseppe

burleyman · ‎06-17-2009

6509-VSS#show mod

Mod Ports Card Type Model Serial No.

--- ----- -------------------------------------- ------------------ -----------

1 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX SAD120606A0

2 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX SAD0834084X

3 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX SAD102105KG

4 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX SAL08486CJF

5 5 Supervisor Engine 720 10GE (Active) VS-S720-10G SAD1216042F

9 16 CEF720 16 port 10GE WS-X6716-10GE SAD120804WX

Mod MAC addresses Hw Fw Sw Status

--- ---------------------------------- ------ ------------ ------------ -------

1 001e.4a6f.a99c to 001e.4a6f.a9cb 2.8 12.2(14r)S5 12.2(33)SXH3 Ok

2 0011.9299.bb20 to 0011.9299.bb4f 2.5 12.2(14r)S5 12.2(33)SXH3 Ok

3 0017.0ee2.2a3c to 0017.0ee2.2a6b 2.3 12.2(14r)S5 12.2(33)SXH3 Ok

4 0012.7f29.7cf8 to 0012.7f29.7d27 2.1 12.2(14r)S5 12.2(33)SXH3 Ok

5 001e.4aab.2138 to 001e.4aab.213f 2.0 8.5(2) 12.2(33)SXH3 Ok

9 001f.6cbd.b2e0 to 001f.6cbd.b2ef 1.0 12.2(18r)S1 12.2(33)SXH3 Ok

Mod Sub-Module Model Serial Hw Status

---- --------------------------- ------------------ ----------- ------- -------

1 Centralized Forwarding Card WS-F6700-CFC SAL1209HJ4Z 4.0 Ok

2 Centralized Forwarding Card WS-F6700-CFC SAL1032XFJF 2.1 Ok

3 Centralized Forwarding Card WS-F6700-CFC SAD1025028B 2.0 Ok

4 Centralized Forwarding Card WS-F6700-CFC SAL085073F1 2.0 Ok

5 Policy Feature Card 3 VS-F6K-PFC3C SAD121806US 1.0 Ok

5 MSFC3 Daughterboard VS-F6K-MSFC3 SAD121808N0 1.0 Ok

9 Distributed Forwarding Card WS-F6700-DFC3C SAL1214L973 1.0 Ok

Mod Online Diag Status

---- -------------------

1 Pass

2 Pass

3 Pass

4 Pass

5 Pass

9 Pass

IOS - s72033-advipservicesk9_wan-vz.122-33.SXH3.bin

Question...

What do you mean about mtu 9216 can reduce the interface buffer size? I have this config on a lot of interfaces.

Thanks,

Mike

Giuseppe Larosa · ‎06-17-2009

Hello Mike,

quite a powerful device this and it is a VSS so now I understand the triple index.

note:

low level buffers are programmed based on the expected maximum frame size so given a fixed memory space an increase to 9216 bytes of MTU makes this space segmented in a smaller number of buffers.

Also to be noted you have CFC on all 6748 modules and DFC3C on 10GE.

Hope to help

Giuseppe

burleyman · ‎06-17-2009

So correct me if I am wrong....the more interfaces I have configured with mtu 9216 I will get more output drops because of the amount of memory I have? Should I look at increasing the memory? would that help resolve the issue?

Mike

jbarger · ‎06-17-2009

From what I understand your going to have issues until all your devices use jumbo frames. A switched layer 2 frame is 1500 bytes. As far a I know there is no mechanism to fragment jumbo frames back to standard 1500 byte frames.

I have not configured jumbo frames and only know this from asking others so you should really read up on jumbo frame support.

burleyman · ‎06-17-2009

I am reviewing jumbo frame info now. I am also checing to make sure everything connected has jumbo frames on and are supported.

Mike

Giuseppe Larosa · ‎06-17-2009

Hello Mike,

this is a good check.

However, the effect of shorter queues (bigger single buffers for the MTU 9216) on ports might be that frames to be transferred from a 10GE port to a 1GE port are "lost in translation" even if the whole data path supports jumbo frames.

The 10GE can try to move N frames to 1GE port but if N is greater then the number of slots in these low level buffers some are lost.

in other words if the queue can keep less then 10 frames that is the ratio between 10GE speed and 1Gbps speed this can be a problem.

This is just a wild guess but on other platforms (c7500) we had problems caused by a similar scenario.

Hope to help

Giuseppe

burleyman · ‎06-17-2009

I see what you mean....I have the 10G trying to send data to the 1G it could get conjested and cause drops, even though both can handle jumbo frames.....correct.

Thanks,

Mike