Output Drop on ether channel L2 trunk 3560-4500

[Dr.X]

Hello Folks ,

We have an L2 trunk between cisco 3560 and cisco 4500.

we have 7 ports bundled in that channel.

we have both sides flow control disabled in/out

real traffic can reach to 5.5 G, but we still see output Drops only on 3560.

At 4500 we don't see any Drops at all in Etherchannel.

Both sides have( ip-src-dst) hashing algorithms for load balancing.

I searched almost all posts in the forum but not able to find something can help me.

i tested PPS and switch limitations, all is normal and under thresholds.

I'm just wondering does ether channel port like 7 G with real traffic 4-5.5 G loaded require different settings?

could be a bug? why is it only on one side of ether channel not on both of them ?!!

###############################

below show commands :

1- Cisco 3560 :

3560G#sh etherchannel summary
Flags: D - down P - bundled in port-channel
I - stand-alone s - suspended
H - Hot-standby (LACP only)
R - Layer3 S - Layer2
U - in use f - failed to allocate aggregator

M - not in use, minimum links not met
u - unsuitable for bundling
w - waiting to be aggregated
d - default port

Number of channel-groups in use: 5
Number of aggregators: 5

Group Port-channel Protocol Ports
------+-------------+-----------+-----------------------------------------------
45 Po45(SU) - Gi0/6(P) Gi0/7(P) Gi0/24(P)
Gi0/32(P) Gi0/34(P) Gi0/45(P)
Gi0/48(P)

3560G#sh run int port-channel 45
Building configuration...

Current configuration : 124 bytes
!
interface Port-channel45
description 3560--4500 Bundle
switchport trunk encapsulation dot1q
switchport mode trunk
end

3560G#sh int port-channel 45
Port-channel45 is up, line protocol is up (connected)
Hardware is EtherChannel, address is 0023.ea72.ba06 (bia 0023.ea72.ba06)
Description: 3560--4500 Bundle
MTU 1500 bytes, BW 7000000 Kbit, DLY 10 usec,
reliability 255/255, txload 123/255, rxload 196/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, link type is auto, media type is unknown
input flow-control is off, output flow-control is unsupported
Members in this channel: Gi0/6 Gi0/7 Gi0/24 Gi0/32 Gi0/34 Gi0/45 Gi0/48
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 01:45:41
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 71866200
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 5405805000 bits/sec, 673407 packets/sec
5 minute output rate 3398399000 bits/sec, 520771 packets/sec
4072242825 packets input, 4102195296979 bytes, 0 no buffer
Received 842862805 broadcasts (150897 multicasts)
0 runts, 0 giants, 0 throttles
2 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 150897 multicast, 0 pause input
0 input packets with dribble condition detected
3185468252 packets output, 2632031579409 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out

3560G#sh etherchannel load-balance
EtherChannel Load-Balancing Configuration:
src-dst-ip

EtherChannel Load-Balancing Addresses Used Per-Protocol:
Non-IP: Source XOR Destination MAC address
IPv4: Source XOR Destination IP address
IPv6: Source XOR Destination IP address

3560G#sh ve
3560G#sh version
Cisco IOS Software, C3560 Software (C3560-IPSERVICESK9-M), Version 12.2(53)SE2, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2010 by Cisco Systems, Inc.
Compiled Wed 21-Apr-10 05:33 by prod_rel_team
Image text-base: 0x01000000, data-base: 0x02E00000

ROM: Bootstrap program is C3560 boot loader
BOOTLDR: C3560 Boot Loader (C3560-HBOOT-M) Version 12.2(44)SE5, RELEASE SOFTWARE (fc1)

3560G uptime is 1 year, 27 weeks, 7 hours, 42 minutes
System returned to ROM by power-on
System image file is "flash:/c3560-ipservicesk9-mz.122-53.SE2.bin"

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco WS-C3560G-48TS (PowerPC405) processor (revision D0) with 131072K bytes of memory.
Processor board ID FOC1245W1F1
Last reset from power-on
21 Virtual Ethernet interfaces
52 Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address : 00:23:EA:72:BA:00
Motherboard assembly number : 73-10214-04
Power supply part number : 341-0107-01
Motherboard serial number : FOC12450UNW
Power supply serial number : AZS124310CD
Model revision number : D0
Motherboard revision number : C0
Model number : WS-C3560G-48TS-S
System serial number : FOC1245W1F1
Top Assembly Part Number : 800-26849-01
Top Assembly Revision Number : C0
Version ID : V03
CLEI Code Number : CNMWY00ARC
Hardware Board Revision Number : 0x09

Switch Ports Model SW Version SW Image
------ ----- ----- ---------- ----------
* 1 52 WS-C3560G-48TS 12.2(53)SE2 C3560-IPSERVICESK9-M

Configurations 

2- on 4500 Side :

4500E#sh etherchannel su
4500E#sh etherchannel summary
Flags: D - down P - bundled in port-channel
I - stand-alone s - suspended
H - Hot-standby (LACP only)
R - Layer3 S - Layer2
U - in use f - failed to allocate aggregator

M - not in use, minimum links not met
u - unsuitable for bundling
w - waiting to be aggregated
d - default port

Number of channel-groups in use: 3
Number of aggregators: 3

Group Port-channel Protocol Ports
------+-------------+-----------+-----------------------------------------------
45 Po45(SU) - Gi1/42(P) Gi1/43(P) Gi1/44(P)
Gi1/45(P) Gi1/46(P) Gi1/47(P)
Gi1/48(P)

4500E#sh int port-channel 45
Port-channel45 is up, line protocol is up (connected)
Hardware is EtherChannel, address is 0007.7d64.0529 (bia 0007.7d64.0529)
Description: 4500---3560 Bundle
MTU 1500 bytes, BW 7000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 197/255, rxload 124/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is N/A
input flow-control is off, output flow-control is unsupported
Members in this channel: Gi1/42 Gi1/43 Gi1/44 Gi1/45 Gi1/46 Gi1/47 Gi1/48
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:01, output never, output hang never
Last clearing of "show interface" counters 02:18:08
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 49122
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 3417084000 bits/sec, 522307 packets/sec
5 minute output rate 5415228000 bits/sec, 674449 packets/sec
4139848421 packets input, 3419495488468 bytes, 0 no buffer
Received 53945 broadcasts (25299 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 input packets with dribble condition detected
5286307877 packets output, 5330349202928 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
4500E#sh int port-channel 45
Port-channel45 is up, line protocol is up (connected)
Hardware is EtherChannel, address is 0007.7d64.0529 (bia 0007.7d64.0529)
Description: 4500---3560 Bundle
MTU 1500 bytes, BW 7000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 197/255, rxload 124/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is N/A
input flow-control is off, output flow-control is unsupported
Members in this channel: Gi1/42 Gi1/43 Gi1/44 Gi1/45 Gi1/46 Gi1/47 Gi1/48
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:01, output never, output hang never
Last clearing of "show interface" counters 02:18:14
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 3414069000 bits/sec, 521786 packets/sec
5 minute output rate 5409966000 bits/sec, 673824 packets/sec
4142893680 packets input, 3421981735262 bytes, 0 no buffer
Received 53990 broadcasts (25320 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 input packets with dribble condition detected
5290299377 packets output, 5334375403783 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out

4500E#sh version
Cisco IOS Software, Catalyst 4500 L3 Switch Software (cat4500e-IPBASEK9-M), Version 15.2(2)E8, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2018 by Cisco Systems, Inc.
Compiled Mon 22-Jan-18 06:32 by prod_rel_team

ROM: 12.2(44r)SG9
4500E uptime is 24 weeks, 10 hours, 22 minutes
System returned to ROM by power-on
System image file is "bootflash:cat4500e-ipbasek9-mz.152-2.E8.bin"
Hobgoblin Revision 20, Fortooine Revision 1.40

Last reload reason: power-on

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco WS-C4948E (MPC8548) processor (revision 5) with 1048576K bytes of memory.
Processor board ID CAT1524S1KJ
MPC8548 CPU at 1GHz, Cisco Catalyst 4948E
Last reset from PowerUp
4 Virtual Ethernet interfaces
48 Gigabit Ethernet interfaces
4 Ten Gigabit Ethernet interfaces
511K bytes of non-volatile configuration memory.

Configuration register is 0x2101

4500E#sh run int port-channel 45
Building configuration...

Current configuration : 99 bytes
!
interface Port-channel45
description 4500---3560 Bundle
switchport
switchport mode trunk
end

4500E#

Hope to find some help .

Thanks in advance .

[Giuseppe Larosa]

Hello @Dr.X ,

the best way to investigate this kind of issues is to look at the statistics of each member interfaces.

You have seven member links on each side . For the way that Cisco ether-channel load balancing actually work a fair per flow distribution is achieved only with 2,4, o r 8 member links.

In short the etherchannel hashing function to find the member link to use for each flow uses a table with 16 slots numbered binary 0000 to 1111 and maps the member links to the 16 slots table sequentially.

The hash function is calculated using the binary EXOR of the last 4 less significant bits of IP SA and IP DA the result that can be 0000 to 1111 is used to fetch the member link from the 16 slot table.

In your case the first two member links are used slightly more as they are used three times and the other member links two times each

0000 member link1

0001 member link2

0010 member link3

0011 member link4

0100 member link5

0101 member link6

0111 member link7

1000 member link1

1001 member link2

[ouput omitted ]

in addition to this you need to understand that the hashing function does not take in account the amount of traffic generated on each flow fair results are achieved when comparable flows are spread among the member links.

if there are specific flows that move very great volume of traffic because for example they represent traffic between servers to build a database replica the fair load balancing does not happen anymore as these heavy flows can move several gigabyte of traffic and they can stress your network.

On routers there is the ip accounting command that allows to find out the top talkers.

In your case you should verify if you can enable Netflow on the c4500 side and to enable it on L2 switched traffic. This may be possible or not depending on the speciic supevisor model you have and the software licenses.

Edit:

I see you have a C4948 switch that is /has been a good switch in the datacenter as top of the rack TOR switch. But I am afraid it does not support Netflow

>>cisco WS-C4948E (MPC8548) processor (revision 5) with 1048576K bytes of memory.

at this point I would suggest to add another member to the bundle taking care of ASIC to ports mapping on C3560.

Hope to help

Giuseppe

[Dr.X]

Thank you for your reply ,

Totally understand your point .

Now we added 8-th port .

3560G#sh int port-channel 45
Port-channel45 is up, line protocol is up (connected)
Hardware is EtherChannel, address is 0023.ea72.ba06 (bia 0023.ea72.ba06)
Description: 3560--4500 Bundle
MTU 1500 bytes, BW 8000000 Kbit, DLY 10 usec,
reliability 255/255, txload 90/255, rxload 143/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, link type is auto, media type is unknown
input flow-control is off, output flow-control is unsupported
Members in this channel: Gi0/6 Gi0/7 Gi0/24 Gi0/32 Gi0/34 Gi0/39 Gi0/45 Gi0/48
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:01, output hang never
Last clearing of "show interface" counters 00:00:45
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 71030169
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 4501611000 bits/sec, 538787 packets/sec
5 minute output rate 2839950000 bits/sec, 422960 packets/sec
24952326 packets input, 25919676128 bytes, 0 no buffer
Received 842651573 broadcasts (1088 multicasts)
0 runts, 0 giants, 0 throttles
2 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 1088 multicast, 0 pause input
0 input packets with dribble condition detected
19632125 packets output, 16603728068 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out
3560G#

normal traffic now is about 4.5 G but we still have output Drops on the 8G EtherChannel.

FYI the 4500 side don't have Drop at all, only drops occur on the 3560 Side.

Could we look at something else?

[Joseph W. Doherty]

The 3560 (and other Catalyst 2K and 3K switches) is a bit infamous for egress drops as the switch doesn't have much in the way of egress buffer resources which is compounded by its default buffer management. The Catalyst 4K series is much, much better in this aspect (which is likely why you don't see if dropping packets).

Anything you can do? Yes, first physical RAM (on the 3560) is allocated to/per 24 copper egress ports and to/per the SFP ports, so if you spread your 8 Etherchannel ports, across banks of copper and/or the SFP ports, they will have more RAM to work from. (NB: however, they will still contend with other ports using the same bank of RAM.)

Next, if you enable QoS (if not already enabled), you can "tweak" buffer settings, which can make a huge difference in egress drops (although, again, if other ports are "busy", there will again be contention for buffer RAM).

First thing to try as a buffer tweak is to push the logical drop thresholds up from their defaults. I've even/often used maximum values, which doesn't usually seem otherwise adverse.

Next thing to try would be to reduce per port "reservations" (allows the buffers to go to the shared pool). Again, I've even/often used minimum per port reservation values. Also again, doesn't seem adverse if your other ports aren't much demanding of buffers too.