Packet loss across Catalyst

[tenbucker]

Looking for a bit of guidance.

I have a customer that is using a Catalyst as their CPE, it also appears they are using this device as one of their core switches as there are over 20+ VLANs connected to the device through a number of trunks. The customer is reporting slow performance, this is evident from a large number of drops on the output queue of one of the trunks. Cable has been replaced but is still re-occuring. 

I conducted a number of Ping Plotter tracers, the tracers take the same path, but obviously 3 & 4 have an additional hop. All were conducted at the same time. The four tracers were:

1. To the WAN IP of the CPE

2. To one of the SViI's IP on the CPE

3. Host IP on one of the VLANs

4. Another host IP on a different VLAN.

I conducted these persistent tracers over an hour or so, and for tracers 3 & 4 I was seeing up to 25% packet loss, however the packet loss was occurring at the WAN IP, but at the same time, for tracers 1 & 2 there was no packet loss at the WAN IP, and no packet loss overall?

I need to find some sort of distinction here. My peers think this could be the Catalyst is low on resources in respect to operating as a switch & a router, i.e some sort of contention for resources, however I need to provide some evidence to support this.

If anyone can shed some light as to what may be happening that would be appreciated.

[paul driver]

Hello

That switch is quite old! - Have you checked the health of it?
Would you be able to post the output of the following into a file?

sh process cpu sorted/history
sh memory summary
sh int x/x controllers
show interface | include line|\/sec
show ip interface | in switch

sh version | in Ba
sh spanning-tree root
sh spanning-tree summary
show spanning-tree detail | in ieee|root|top|link|bpdu

res
Paul

[tenbucker]

Hey Paul,

Yeah, it wasn't looking too bad, please find attached.

Ben

[paul driver]

Hello Ben

Thanks for the output 

im traveling at present so kind of hard to read that output - however may I suggest a few tweaks?

on the L3 core 

1) my sure that switch is the stp root for all vlans (even vlan 1)

2) enable stp loop-guard uplinkfast globally on all switches  And stp  portfast-bpduguard on all access ports

3) enable fast switching on the l3 interfaces  ( ip route-cache)

res

Paul

[Joseph W. Doherty]

The 3560/3750 series of switches are a bit infamous on having little RAM to buffer interface bursts, and the series buffer management, by "reserving" buffers for ports, often makes the lack of buffer RAM acute, especially with QoS enabled.

Assuming your interface buffer needs are transient, and most interfaces aren't "busy" concurrently, you can "tune" buffer management from a reserved model to a dynamically acquire resource model, which I've found, will often dramatically reduce interface output drops.