01-25-2011 06:41 AM - edited 03-04-2019 11:12 AM
Hello,
we have a 6509 with the following modules:
Mod Ports Card Type Model Serial No.
--- ----- -------------------------------------- ------------------ -----------
1 2 Catalyst 6000 supervisor 2 (Active) WS-X6K-SUP2-2GE
2 2 Catalyst 6000 supervisor 2 (Standby) WS-X6K-SUP2-2GE
3 16 16 port 1000mb GBIC ethernet WS-X6416-GBIC
4 16 16 port 1000mb GBIC ethernet WS-X6416-GBIC
5 48 SFM-capable 48 port 10/100/1000mb RJ45 WS-X6548-GE-TX
6 48 SFM-capable 48 port 10/100/1000mb RJ45 WS-X6548-GE-TX
7 48 48 port 10/100 mb RJ45 WS-X6348-RJ-45
8 48 48 port 10/100 mb RJ45 WS-X6348-RJ-45
Mod Sub-Module Model Serial Hw Status
--- --------------------------- --------------- --------------- ------- -------
1 Policy Feature Card 2 WS-F6K-PFC2 2.0 Ok
1 Cat6k MSFC 2 daughterboard WS-F6K-MSFC2 1.2 Ok
2 Policy Feature Card 2 WS-F6K-PFC2 2.0 Ok
2 Cat6k MSFC 2 daughterboard WS-F6K-MSFC2 1.2 Ok
The problem is that the CPU usage is around 40% and the switch is doing normal routing with BGP but it has only the default route to
the ISP and no full BGP routing table. Other services are not enabled (QoS, ACL...)
There are max. 8000 subscribers (Cable and WiFi) and 800Mbit/s link top, but normaly the link utilization is around 500-600 Mbit/s.
Is the Supervisor to powerless for this much subscribers?
I tried to enable NBAR on one interface and the CPU usage jumped to 99%.
After enabling NBAR I found out that there is a lot of Bittorrent traffic, on second place after HTTP.
And we have packet loss when doing normal ping to the ISP (directly connected over optical cable) on a windows machine.
About every 15th is a lost packet.
And what is more strange is that I am losing packets when I ping the Cat6509 when I am in the same VLAN as the 6509 WAN interface (connected over a 3550 GE swtich with CPU usage around 3%). It should not lose any packets.
I put the sh proc cpu output in the attachment. I typed the command after we disconnected the cable to the ISP for testing so it is a little bit less cpu usage.
I think that the SUP2 is just to weak for this kind of operation.
I whould like to hear your opinions.
Thank you!
01-26-2011 03:07 AM
Anyone??
01-26-2011 07:49 AM
Hello,
It sounds like your MSFC is getting hit with high level of traffic at maybe for very short periods and dificult to trace if this are microburst.
You can SPAN the MSFC to a sniffer port as bellow and apply ACL to filter unwanted traffic. This way unwanted traffic will not hit MSFC CPU (it will be done on PFC).
on SP
remote login switch
test monitor add 1 rp-inband both
test monitor del 1 rp-inband both
ON RP
create a monitor session with source using an admin down/unused port - using same monitor session number as SP monitor
create a destination monitor session to sniffer with same session number a SP session
Hopefuly you have an sniffer that can summarise top talkers. It is quite common to have unwanted multicast/broadcast traffic hammering your MSFC CPU.
I hope it helps.
Regards
Gonçalo
01-26-2011 08:40 AM
Thank your, I will try that as soon as possible.
with
"You can SPAN the MSFC to a sniffer port as bellow and apply ACL to filter unwanted traffic. This way unwanted traffic will not hit MSFC CPU (it will be done on PFC)."
you mean that I apply an ACL AFTER I found out what traffic is overloading the MSFC?
on SP
remote login switch
test monitor add 1 rp-inband both
test monitor del 1 rp-inband both
ON RP
SP is the Supervisor?
RP is the MFSC?
Do you have a link for the configuration guide? I dont want to mess something up. I do not have any experience on the 6500 series.
01-27-2011 12:44 AM
Hello,
There is no formal documentation for this procedure (as far I am aware). I learned this technique a while back.
There is a document highlighting most common issues for high CPU utilization on 6500. This can give you a pretty good idea:
http://www.cisco.com/en/US/products/hw/switches/ps708/products_tech_note09186a00804916e0.shtml
Back to your other questions:
You will need to apply ACL AFTER you find found out what traffic is overloading the MSFC. This is housekeeping post SPAN traffic analysis.
SP=Switch Processor (mainboard on Sup card and is use for L2 services "mainly")
RP= Router Processor (daughter card on Sup card and it is used for L3 routing "mainly")
Good luck :-)
Gonçalo
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide