3750 switch stack: How to troubleshoot performance issue

[Difan Zhao]

Hi experts,

I have a network with 3750 switch stack as the core. Durring the installation everything was fine and working. However when more users start to come in, I start to have issues such as choppy VOIP audio, delay for the IPTV, ...etc.

I was troubleshooting the CPU usage on the switch since it's at 50% all the time. However nothing was dropped by the CPU or can't be handled. I followed the following link for the troubleshooting.

http://www.cisco.com/en/US/products/hw/switches/ps5023/products_tech_note09186a00807213f5.shtml

Now I'm thinking if it's because that the switch hardware can't handle the amount traffic required to be switched or routed. I read the datasheet and it says it can handle 160Gbps switching fabric capability. What command can I use to check how much is being used? What's the routing capability of the switch? Here is my stack:

*    1 30    WS-C3750E-24TD     12.2(53)SE2           C3750E-UNIVERSALK9-M
     2 30    WS-C3750E-24TD     12.2(53)SE2           C3750E-UNIVERSALK9-M
     3 26    WS-C3750V2-24PS    12.2(53)SE2           C3750-IPSERVICESK9-M
     4 26    WS-C3750V2-24TS    12.2(53)SE2           C3750-IPSERVICESK9-M
     5 12    WS-C3750G-12S      12.2(53)SE2           C3750-IPSERVICESK9-M
     6 12    WS-C3750G-12S      12.2(53)SE2           C3750-IPSERVICESK9-M

I need to determine if it's performance issue and decide whether replacing the core with next level 4500 switch.

Thanks!

[sdheer]

Hi,

Check the following on your switch and if possible paste it here.

>> sh int | i line |rate

>> sh platform tcam utilisation

>> sh platform port-asic stats drop

Do you have qos configured on your switches .If not and in case you have a mix of traffic ie data as well as voice and video you could go with configuring qos on the switch which will prioritize voice and video traffic over data traffic and prevent the issues which you are facing.

Regards,

Swati

Please rate if you find content useful

[Shashank Singh]

Hi,

Please share the following outputs from the stack:

show platform port-asic stats drop

show controller utilization

show mls qos

Cheers,

Shashank

P.S. Please rate helpful posts

[Difan Zhao]

Thank you guys so much for quick reply! I will reply you both in one post.

sh int | i line |rate
The physical interfaces are having the normal volume of traffic. I also realized that the VLAN interface doesn't record the traffic caused by multicast! We have IPTV so multicast is quite big! Is there way to check the routed multicast volume?

sh platform tcam utilization
...
Unicast mac addresses:                       3292/3292       5838/5838
...

I think this is bad! I also did a "show mac add count" and available MAC is only like 40!!

sh platform port-asic stats drop
All 0. What's port Asic?

show controller utilization
...

Total Ports : 136
Switch Receive Bandwidth Percentage Utilization  : 1
Switch Transmit Bandwidth Percentage Utilization : 1

Stack Ring Percentage Utilization : 1

Not too bad...

I don't have the QoS enabled yet. I worry that it will stress the network even more...

Thanks guys!

[Shashank Singh]

Hi,

As far as I know, you can get the ouput of "sh platform port-asic stats drop" without specifying any port and asic also. This output will be long and would be for all ASICs on the switch.

Also, share "sh platform pm if-num" from the same switch.

Cheers,

Shashank

P.S.Pleae rate helpful posts

[Shashank Singh]

Do you see output drops on any of the stack switches? You can concentrate on the uplinks for checking on drops. If you do not see any drops in the "sh platform port-asic stats drop" output, check the same in following command:

"remote command all sh platform port-asic stats drop" (for all the stack member switches).

Instead you can session in the stack members one by one and do the command individually:

master#session switch#

(switch#) sh platfo port-as stats drop

exit

If there is any traffic which is getting dropped due to hardware overutilization, there should be some drops seen in this output.

Cheers,

Shashank

Please rate helpful posts

[Difan Zhao]

Hi Shashank,

Sorry for late reply. We replaced the core with the 4507E and everything is fine now. Those commands don't seem to show the packets dropped during the routing process, especially the multicast routing packets. I will create a new post and try to gather more troubleshooting commands.

Thanks!

Difan