Solved: Packet loss when pinging from/to a cisco 3560e switch

veerareddy.dubbaka1 · ‎03-29-2015

I see Packet loss when pinging from/to a cisco 3560e switch. CPU utilization is normal.

Switches are running with IOS c3560e-universalk9-mz.122-35.SE5.bin.

Packet loss is observed for all the devices irrespective of directly connected or remote devices.

If i do self pinging, there are no packet loss.

I don't see any error on interface.

Can anyone please help me in resolving this issue.

Madhukrishnan Gopinathan Nair · ‎04-01-2015

Hi Veera,

One more thing you can check if there is any stuck tcp connections.

sh tcp brief

Thanks,

Madhu

View solution in original post

Madhukrishnan Gopinathan Nair · ‎04-01-2015

This seems to be the issue. This is a lot of them.

And all are to port 49 which is to your TACACS server. I can see only one connection in ESTAB. Can you verify what is that connection?

The simplest way to remove them is clear all at once.

clear tcp tcb *

But it would close down your ESTAB too.. Also your current session and you have to login back.

Else if you have patience you can close them which are in close_wait one by one.

clear tcp tcb 03737C48

clear tcp tcb 039ACDC4 etc...

I am very positive this is going to fix this issue.

Thanks,

Madhu

View solution in original post

Madhukrishnan Gopinathan Nair · ‎03-30-2015

Hi what devices you are trying to ping ?

Do you see any pattern for drops while pinging ?

Thansk,

Madhu

veerareddy.dubbaka1 · ‎03-30-2015

Thanks Madhu for taking up my question.

I am trying to ping directly connected and remote devices like switches(6500) and computers.

I see ping packet loss from 6500 switch to any 3560e switch but not to other switches like 4500 and 6500.

Here is ping packet drops.

Type escape sequence to abort.
Sending 20, 100-byte ICMP Echos to X.X.X.X, timeout is 2 seconds:
!!.!!.!!!!.!!!.!.!!!
Success rate is 75 percent (15/20), round-trip min/avg/max = 1/1/9 ms

Type escape sequence to abort.
Sending 20, 100-byte ICMP Echos to X.X.X.X, timeout is 2 seconds:
!.!!.!!!!.!!!.!!!!.!
Success rate is 75 percent (15/20), round-trip min/avg/max = 1/3/25 ms

a.klingenberg · ‎03-30-2015

Do you have a large L2 environment?
Could be a STP issue,...

veerareddy.dubbaka1 · ‎03-31-2015

Hi,

We have L2 environment but i don't see anything which we can suspect that STP is causing the issue. CPU utilization is normal ,no switching loops and traffic on the trunk interfaces is normal. .

Madhukrishnan Gopinathan Nair · ‎03-31-2015

Do you see any impact due to this?

Can you ping the hosts connected behind 3560 without loss from 6500?

Thanks

Madhu

veerareddy.dubbaka1 · ‎03-31-2015

These devices are monitored in solarwinds and because of the icmp packet loss some time device status shown down.

I can ping the host from 6500 without any packet loss but not from 3560.

Madhukrishnan Gopinathan Nair · ‎03-31-2015

Hi Veera,

This means there is no problem in packet forwarding at hardware but only control plane is affected(when you ping the switch itself, its handled at software). I am quite not sure why many switches are showing the same behaviour. But what is the uptime of these switches. Probably you may consider reloading one of the switch to see it clears this. Also please share a show tech from one of the switch.

Thanks,

Madhu.

Madhukrishnan Gopinathan Nair · ‎04-01-2015

Hi Veera,

One more thing you can check if there is any stuck tcp connections.

sh tcp brief

Thanks,

Madhu

veerareddy.dubbaka1 · ‎04-01-2015

I checked sh tcp brief and most of the TCP connections are in CLOSEWAIT and some of them are in TIMEWAIT and ESTAB.

Madhukrishnan Gopinathan Nair · ‎04-01-2015

How many are in CLOSEWAIT. Can you please share the outptut.

That could be one reason for the slow responses.

Thanks,

Madhu.

veerareddy.dubbaka1 · ‎04-01-2015

TCB       Local Address           Foreign Address        (state)
03737C48 10.47.0.229.60053       10.41.81.55.49         CLOSEWAIT
039ACDC4 10.47.0.229.61929       10.41.35.250.49        CLOSEWAIT
03B316C0 10.47.0.229.27544       10.41.81.55.49         CLOSEWAIT
038228F0 10.47.0.229.16506       10.41.35.250.49        CLOSEWAIT
039C3D04 10.47.0.229.15207       10.41.81.55.49         CLOSEWAIT
039A9BD0 10.47.0.229.52983       10.41.81.55.49         CLOSEWAIT
0394152C 10.47.0.229.22425       161.61.35.250.49       CLOSEWAIT
037D811C 10.47.0.229.21117       10.41.81.55.49         CLOSEWAIT
039C12BC 10.47.0.229.37437       10.41.81.55.49         CLOSEWAIT
03933B84 10.47.0.229.34085       161.61.35.250.49       TIMEWAIT
03B32340 10.47.0.229.45729       10.41.81.55.49         CLOSEWAIT
038247D0 10.47.0.229.32816       10.41.81.55.49         CLOSEWAIT
039A92D8 10.47.0.229.38680       161.61.35.250.49       CLOSEWAIT
037370F0 10.47.0.229.13212       10.41.81.55.49         CLOSEWAIT
037D85F0 10.47.0.229.38728       10.41.81.55.49         CLOSEWAIT
03B2B284 10.47.0.229.23428       10.41.81.55.49         CLOSEWAIT
03B2ADB0 10.47.0.229.56836       10.41.81.55.49         CLOSEWAIT
0394BFF0 10.47.0.229.23257       161.61.35.250.49       CLOSEWAIT
036604DC 10.47.0.229.44437       10.41.81.55.49         CLOSEWAIT
0394C700 10.47.0.229.22          192.37.184.211.61639   ESTAB
039B9A68 10.47.0.229.20543       10.41.81.55.49         CLOSEWAIT
03739B28 10.47.0.229.15392       10.41.81.55.49         CLOSEWAIT
TCB       Local Address           Foreign Address        (state)
0392EA48 10.47.0.229.13862       10.41.81.55.49         CLOSEWAIT
0365E23C 10.47.0.229.27856       10.41.81.55.49         CLOSEWAIT
03817C0C 10.47.0.229.64929       10.41.81.55.49         CLOSEWAIT
039357C8 10.47.0.229.22088       10.41.81.55.49         CLOSEWAIT
037375C4 10.47.0.229.21832       10.41.81.55.49         CLOSEWAIT
039C20E8 10.47.0.229.18169       10.41.81.55.49         CLOSEWAIT
03716D08 10.47.0.229.61993       10.41.81.55.49         CLOSEWAIT
039A74E4 10.47.0.229.62948       10.41.81.55.49         CLOSEWAIT
03655480 10.47.0.229.14052       10.41.81.55.49         CLOSEWAIT
039407F0 10.47.0.229.49643       161.61.35.250.49       CLOSEWAIT
039A53AC 10.47.0.229.13233       10.41.81.55.49         CLOSEWAIT
03739FFC 10.47.0.229.16605       10.41.81.55.49         CLOSEWAIT
039B82B8 10.47.0.229.16458       10.41.35.250.49        CLOSEWAIT
039BEBA4 10.47.0.229.64377       10.41.81.55.49         CLOSEWAIT
03741980 10.47.0.229.13866       10.41.81.55.49         CLOSEWAIT
03B3ABF8 10.47.0.229.19365       10.41.81.55.49         CLOSEWAIT
039B5810 10.47.0.229.24768       10.41.81.55.49         CLOSEWAIT
03956E48 10.47.0.229.55980       161.61.35.250.49       CLOSEWAIT
03946820 10.47.0.229.65053       161.61.35.250.49       CLOSEWAIT
037DBE94 10.47.0.229.15283       10.41.81.55.49         CLOSEWAIT
039A4854 10.47.0.229.48562       10.41.81.55.49         CLOSEWAIT
TCB       Local Address           Foreign Address        (state)
03B33320 10.47.0.229.29803       10.41.81.55.49         CLOSEWAIT
03B3B79C 10.47.0.229.12142       10.41.81.55.49         CLOSEWAIT
03713C9C 10.47.0.229.63799       10.41.81.55.49         CLOSEWAIT
039BBECC 10.47.0.229.14763       10.41.81.55.49         CLOSEWAIT
03656E40 10.47.0.229.16357       10.41.81.55.49         CLOSEWAIT
0362A73C 10.47.0.229.62450       10.41.81.55.49         CLOSEWAIT
039B878C 10.47.0.229.64402       161.61.35.250.49       CLOSEWAIT
03826CFC 10.47.0.229.16108       10.41.81.55.49         CLOSEWAIT
03B2CA34 10.47.0.229.17634       10.41.81.55.49         CLOSEWAIT
03AD78D0 10.47.0.229.15249       161.61.35.250.49       CLOSEWAIT
03AD967C 10.47.0.229.20389       161.61.35.250.49       CLOSEWAIT
03B2C560 10.47.0.229.37079       10.41.81.55.49         CLOSEWAIT
039C5128 10.47.0.229.24711       10.41.81.55.49         CLOSEWAIT
03822F74 10.47.0.229.54866       10.41.81.55.49         CLOSEWAIT
0372C5FC 10.47.0.229.13298       10.41.81.55.49         CLOSEWAIT
0372D278 10.47.0.229.12407       10.41.81.55.49         CLOSEWAIT
039A33D0 10.47.0.229.36573       10.41.81.55.49         CLOSEWAIT
039BCEF8 10.47.0.229.53853       10.41.81.55.49         CLOSEWAIT
039C02D8 10.47.0.229.53725       10.41.81.55.49         CLOSEWAIT
039B5CE4 10.47.0.229.58027       10.41.81.55.49         CLOSEWAIT
0381866C 10.47.0.229.17100       10.41.81.55.49         CLOSEWAIT
TCB       Local Address           Foreign Address        (state)
039BB374 10.47.0.229.53148       10.41.81.55.49         CLOSEWAIT
03AD3634 10.47.0.229.19716       161.61.35.250.49       CLOSEWAIT
0362DAA4 10.47.0.229.19479       10.41.81.55.49         CLOSEWAIT
0365AE60 10.47.0.229.62209       10.41.81.55.49         CLOSEWAIT
0362D5D0 10.47.0.229.41327       10.41.81.55.49         CLOSEWAIT
037D7C48 10.47.0.229.58283       10.41.81.55.49         CLOSEWAIT
03955474 10.47.0.229.33810       161.61.35.250.49       CLOSEWAIT
0373B15C 10.47.0.229.23331       10.41.81.55.49         CLOSEWAIT
036628D0 10.47.0.229.46856       10.41.81.55.49         CLOSEWAIT
03819584 10.47.0.229.19861       10.41.81.55.49         CLOSEWAIT
0394D000 10.47.0.229.64732       10.41.35.250.49        CLOSEWAIT
0394B760 10.47.0.229.19967       161.61.35.250.49       CLOSEWAIT
039B6BD4 10.47.0.229.40096       10.41.81.55.49         CLOSEWAIT
03AD7150 10.47.0.229.65184       10.41.35.250.49        CLOSEWAIT
039BC3A0 10.47.0.229.64702       10.41.81.55.49         CLOSEWAIT
03B3A724 10.47.0.229.60399       10.41.81.55.49         CLOSEWAIT
037145E0 10.47.0.229.43951       10.41.81.55.49         CLOSEWAIT
03955EDC 10.47.0.229.29015       161.61.35.250.49       TIMEWAIT
0365FB34 10.47.0.229.13961       10.41.81.55.49         CLOSEWAIT
03828D54 10.47.0.229.12743       10.41.81.55.49         CLOSEWAIT
037DB40C 10.47.0.229.23708       10.41.81.55.49         CLOSEWAIT
TCB       Local Address           Foreign Address        (state)
039AF814 10.47.0.229.15100       10.41.81.55.49         CLOSEWAIT
0392E344 10.47.0.229.23399       10.41.35.250.49        CLOSEWAIT
0393DC3C 10.47.0.229.15393       161.61.35.250.49       CLOSEWAIT
03AD85D0 10.47.0.229.40932       161.61.35.250.49       TIMEWAIT
039574CC 10.47.0.229.25935       10.41.35.250.49        CLOSEWAIT
03738B74 10.47.0.229.58656       10.41.81.55.49         CLOSEWAIT
039AD91C 10.47.0.229.56760       10.41.81.55.49         CLOSEWAIT
03B3BC70 10.47.0.229.15058       10.41.81.55.49         CLOSEWAIT
03B2DC54 10.47.0.229.51131       161.61.35.250.49       CLOSEWAIT
03B393F0 10.47.0.229.11957       10.41.35.250.49        CLOSEWAIT
039B2610 10.47.0.229.33728       10.41.81.55.49         CLOSEWAIT
03B311EC 10.47.0.229.18047       10.41.81.55.49         CLOSEWAIT
039A8E04 10.47.0.229.52022       161.61.35.250.49       CLOSEWAIT
0365D460 10.47.0.229.12241       10.41.81.55.49         CLOSEWAIT
03B33E78 10.47.0.229.47640       10.41.81.55.49         CLOSEWAIT
0372C128 10.47.0.229.60323       10.41.81.55.49         CLOSEWAIT
03661CD8 10.47.0.229.39923       10.41.81.55.49         CLOSEWAIT
0393C73C 10.47.0.229.41864       10.41.35.250.49        CLOSEWAIT
03829584 10.47.0.229.56673       161.61.35.55.49        CLOSEWAIT
0362AC10 10.47.0.229.31952       10.41.81.55.49         CLOSEWAIT
039BF078 10.47.0.229.22636       10.41.81.55.49         CLOSEWAIT
TCB       Local Address           Foreign Address        (state)
0365CF8C 10.47.0.229.14476       10.41.81.55.49         CLOSEWAIT
039B443C 10.47.0.229.59226       10.41.81.55.49         CLOSEWAIT
0393E794 10.47.0.229.56282       10.41.35.250.49        CLOSEWAIT
03657740 10.47.0.229.25769       10.41.81.55.49         CLOSEWAIT
03B2F6E8 10.47.0.229.19328       10.41.81.55.49         CLOSEWAIT
0373AC88 10.47.0.229.25766       10.41.81.55.49         CLOSEWAIT
039B213C 10.47.0.229.28882       10.41.81.55.49         CLOSEWAIT
039C07AC 10.47.0.229.38201       10.41.81.55.49         CLOSEWAIT
03AD8DD0 10.47.0.229.23002       10.41.35.250.49        CLOSEWAIT
03739048 10.47.0.229.29572       10.41.35.250.49        CLOSEWAIT
039BA464 10.47.0.229.32273       10.41.81.55.49         CLOSEWAIT
03B31E6C 10.47.0.229.32521       10.41.81.55.49         CLOSEWAIT
0365EBE0 10.47.0.229.41319       10.41.81.55.49         CLOSEWAIT
03938804 10.47.0.229.62841       10.41.35.250.49        CLOSEWAIT
039A1AF8 10.47.0.229.12758       10.41.81.55.49         CLOSEWAIT
039B7DE4 10.47.0.229.20921       10.41.81.55.49         CLOSEWAIT
036549F8 10.47.0.229.51903       10.41.81.55.49         CLOSEWAIT
03714CC8 10.47.0.229.45145       10.41.81.55.49         CLOSEWAIT
037425F8 10.47.0.229.56492       10.41.81.55.49         CLOSEWAIT
03B39D74 10.47.0.229.18174       10.41.81.55.49         CLOSEWAIT

Madhukrishnan Gopinathan Nair · ‎04-01-2015

This seems to be the issue. This is a lot of them.

And all are to port 49 which is to your TACACS server. I can see only one connection in ESTAB. Can you verify what is that connection?

The simplest way to remove them is clear all at once.

clear tcp tcb *

But it would close down your ESTAB too.. Also your current session and you have to login back.

Else if you have patience you can close them which are in close_wait one by one.

clear tcp tcb 03737C48

clear tcp tcb 039ACDC4 etc...

I am very positive this is going to fix this issue.

Thanks,

Madhu

Madhukrishnan Gopinathan Nair · ‎04-01-2015

A likely hit of the bug

https://tools.cisco.com/bugsearch/bug/CSCub04965/?reffering_site=dumpcr

Note : Kindly rate helpful posts

Thanks,

Madhu

veerareddy.dubbaka1 · ‎04-01-2015

Thanks Madhu , It should resolve my issue.I 'll try and let you know.