Cisco 2960 Etherchannel gets intermittent ping drops to a specific Mac

[Ramesh Prabhu]

There are lots of intermittent ping loss from 57.31.201.216 to 57.31.203.211 only between these two. 

I debuged I found all the ping from 57.31.203.211 are received fine in Swtich 57.31.201.216 however its the reply packet or forward packet towards PAKL321:- 57.31.203.211 there will be drops. Quite unusual I captured the ICMP counters and found that for every ping packet sent from Switch:- 57.31.203.216 to Router PAKL321 I see 3 packet in ACL counter on Router, I wonder why.

Maybe the there is flood of packet sent towards PAKL321 from in between switch causing packet loss. Router is absolutely fine.

I suspect the switch. Please advise how to find these intermittent packet loss problem towards particular MAC/IP.

Config of both switch attached.

Diag:-

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
AKLE-APH-COR-1#ping 57.31.203.211 re 100
Type escape sequence to abort.
Sending 100, 100-byte ICMP Echos to 57.31.203.211, timeout is 2 seconds:
.!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!...!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!....!!!!!!!!!!!!
Success rate is 91 percent (91/100), round-trip min/avg/max = 1/2/9 ms
pakl321#sh ip access-lists TEST
Extended IP access list TEST
10 permit icmp host 57.31.203.216 host 57.31.203.211 (273 matches) <<< 273/91 = 3
20 permit ip any any (26 matches)
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
AKLE-APH-COR-1#ping 57.31.203.211 re 200
Type escape sequence to abort.
Sending 200, 100-byte ICMP Echos to 57.31.203.211, timeout is 2 seconds:
.!!!!!!!!!!!!!!!!!!!!....!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.....
.......!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!
Success rate is 90 percent (181/200), round-trip min/avg/max = 1/2/9 ms
pakl321#sh ip access-lists TEST
Extended IP access list TEST
10 permit icmp host 57.31.203.216 host 57.31.203.211 (543 matches) <<<543/181 = 3

Thank you!!!

Ramesh

[Ramesh Prabhu]

Kindly suggest all

[MHM Cisco World]

only icmp packet drop, other traffic is fine ?

[Ramesh Prabhu]

I tested with ping as I can count with that only, there is service issue so expected other packet would also behave same ..

[MHM Cisco World]

I think you face some kind of bottle neck, 
what is core 1 platform you use ?

[Ramesh Prabhu]

Just for particular IP or Mac it's a problem i use Cisco WS 2970 switch

[MHM Cisco World]

please check the point I share, 
can you share the output I ask in photo

[Ramesh Prabhu]

Config is attached in the top post. Status of interface is in the attachment. Standby is on Router side. Router side I don't suspect anything.

[MHM Cisco World · Accepted solution]

AKLE-APH-COR-1#sh int fa0/1
... ....
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 88266 

try 
port-channel load-balance src-dst-mac or dst-mac

and check the output drop. 

[Ramesh Prabhu]

Tried src-dst-mac it fixed now so far no drops. However I don't get the concept behind the previous behaviour of 3 packets duplication and wonder why fa0/1 drops are ok other side more where packet should not go there ...

[MHM Cisco World]

the three packet, as I know and need to test is number of probe per one count 
you specify ping with repeat 200 
the ping will send three probe for each count. 
but I will more sure by doing lab and test this case. 

[MHM Cisco World]

as I guess, each count in ping source router is actually three probe. (probe meaning icmp echo).
you can see four ping success in R1 but in ACL of R2 the ACL count 12 (3 probe X 4 ping count = 12 count in ACL).
so what you see is normal 

[Ramesh Prabhu]

Great thank you mate, so why were the drops occur to particular MAC/IP with loadbalancing of etherchannel?

[MHM Cisco World]

first I will ask you ? are you port-channel is L2 or L3 ? your PO is L2 and hence the IP is not check here the mac is check, 
and you mention particular MAC/IP I will correct here particular MAC. 
now I see your traffic drop and assume the issue is hash using source mac, 
hash using source mac make one SW always use same port member to send traffic to other SW via PO, 
so we need to make SW use both port member, we clear the idea of using IP in L2 PO so we only have now two option 
using des.mac 
or 
using scr.des.mac 

and wow I was right this solve issue of hash and now SW use two port member instead of one. 
this make us use the benefit of PO which is increase the BW (200000). 
and as you get the drop is decrease since now the BW from 100000 to 200000. 

thanks.

[Ramesh Prabhu]

Hi there, 

Now I got another problem ping drops occurs between routers now pakl320 and pakl321