packet drop on etherchannel

[Marko Rodic]

Hello,

We have a problem with etherchannel... We have two 4500 in VSS, and around 20 2960 switches connected with 4500 via etherchannel(one port on one 4500, another port on second 4500). Everything seemed to be working fine, however, we have discovered that we have occasional random packet dropping. When we ping firewall from some of 2960 switches we get 1ms replay for ~15 times, then time out, then everything resume to work normally for another 20 pings, then another time out, etcetc..

When I shut down one of two ports in ehterchannel, everything starts to work without any drops.

I have only configure port channel with broadcast control and left everything else by default, so config on 2960 looks like this:

interface Port-channel1
 switchport mode trunk
 storm-control broadcast level 5.00

interface GigabitEthernet1/0/25
 switchport mode trunk
 shutdown
 storm-control broadcast level 5.00
 channel-group 1 mode passive

interface GigabitEthernet1/0/26
 switchport mode trunk
 storm-control broadcast level 5.00
 channel-group 1 mode passive

and on 4500(random port channel, every single one is same):

interface Port-channel15
 switchport
 switchport mode trunk

interface GigabitEthernet1/5/7
 switchport mode trunk
 channel-group 15 mode active

interface GigabitEthernet2/5/7
 switchport mode trunk
 channel-group 15 mode active

Any idea what might be causing random ping drops? Thanks!

[Markus Benz]

Hi Marko,

your config looks good so far. I can only assume that there is a problem with the interfaces or the ether-channels. Sounds a bit like bouncing interfacs..

Do you see any links flapping in the log or the like?

Do you have this behavior on all 2960ies? Or just on one?

Do you see the ping losses only if you ping the Firewall or also on other destinations?

Regards,

Markus

[Marko Rodic]

Same behavior on all 2960ies. Well, I checked 6 or 7, same problem with every single one. I presumed its same with other 15 or so... 

Ping lost on other destinations as well. 

Unfortunately, haven't been able to see the full logs. It was urgent to recover the system asap. But there was no flapping as message that interface went down would popup. No L1 message popped up. 

[Markus Benz]

Ok.. It will be difficult to solve that if the setup is not existing anymore and we can't get additional logs. (how did you recover it?)

The config snipped you've provided looks correct to me. So without additional information, I doubt we can solve that here...

Regards,
Markus

[Marko Rodic]

Saw that all the problems are only in new part of system(old one worked like a charm). The only difference was that old part is not using fiberchannel, so thought it might be a problem. Shut down one interface on one problematic switch, everything started working fine. Then just shut down rest...

I cant try out anything now. I might be able to set something up, try it out when I went there(I'm not in same city), but would like to know what kind of logs should I look for? All the switches, including 4500, are working at L2. all L3 and above traffic is handled by firewall.  

[Markus Benz]

I would say you should enable one of the etherchannels and capture the standards logs druing the test process ("show log").

Additionally yhou should look at "show etherchannel summary" check if the etherchannel is properly up. And if there is a change during the period you cant ping.

You should also look at LACP: (Example for Port-Channel 15)

show lacp 15 internal
show lacp 15 counters

Please also check the following debugs:

debug etherchannel [ all | error | event ]
debug lacp [ all | event | fsm ]

If the problem is bound to the etherchannel, we should be able to find it with that.
If that shows no result, it is probably not an LACP / Etherchannel Problem

Regards,
Markus