Multicast routing on C2960XR

[triton.milossedlacek]

Dear All,

We are testing the WS-C2960XR-24TS-I for use in video applications dealing with IP multicasts. I want to ask if these switches (or routers) are suitable for routing of multicasts. We did notice that if we route multicasts between VLANs (inter-VLAN routing) then the CPU load gets significanlty higher (up to 100%).

Is what we experience the normal behavior and the switch does not suit the multicast routing applications? 

[Mark Malone]

Hi These are low end access switches capable of stripped down versions of routing , its probably pushing it a bit using intervlan multicast  depending what else switch is configured for too but it still shouldn't push it to 100% , was it definitely the multicast spiking the cpu at that time. Did you collect the show proc cpu sorted

What ios version are you on currently and was this only between 2 svis or much more ?

[triton.milossedlacek]

Hi Mark, thank you for your interest.

I have no access to the switch right now so I could only give you this information (will follow the actual one).

ios 15.2 (switch purchased a month ago)

show proc cpu sorted attached

[Mark Malone]

ok so the 2nd process HULC is a bug you hit , it effects multiple platforms 2900s,3750s,3560s etc very common problem

Heres just one of the related BUG ids on some 2960s,

https://tools.cisco.com/bugsearch/bug/CSCtg86211/?referring_site=bugquickviewredir

The ip input is high though, do you know if cef is running ? If it is switch may be getting over utilized cef filling up and punting packets to the cpu to process which will spike it an cause high ip input

Also small things like access-list log at the end can cause it too on smaller switches

if you do get access again try this script may show more information it will collect the commands after hitting 80% , if eem is not available on these platforms just try collect some of the outputs below especially cef and show int switching  ,  I know on the pure layer 2 ones there not supporting eem  but maybe on routed version it has the feature

event manager applet High_CPU

    event snmp oid 1.3.6.1.4.1.9.9.109.1.1.1.1.4.1 get-type exact entry-op ge entry-val 80" exit-time 10 poll-interval 5

    action 0.1 syslog msg "CPU Utilization is high"

    action 0.2 cli command "enable"

    action 0.4 cli command "show log | append flash:CPU_Profile.txt"

    action 0.5 cli command "show process cpu sorted | append flash:CPU_Profile.txt"

    action 0.6 cli command "show interfaces | append flash:CPU_Profile.txt"

    action 0.7 cli command " show ip cef switching stat | append flash:CPU_Profile.txt"

    action 0.8 cli command " show ip traffic | append flash:CPU_Profile.txt"

    action 0.9 cli command " show int switching | append flash:CPU_Profile.txt"

    action 1.0 cli command "no event manager applet High_CPU"

    action 1.1 cli command "end"

This is the safe harbour image for your switch on the Cisco website most recommended and tested by Cisco , this may get rid of HULC issue c2960x-universalk9-mz.152-2.E3.bin 

https://software.cisco.com/download/release.html?mdfid=284832157&flowid=43522&softwareid=280805680&release=15.2.2E3&relind=AVAILABLE&rellifecycle=MD&reltype=latest

[triton.milossedlacek]

Hi Gyus,

in case you still keep track here, I'm attaching the information Mark was asking for. It was taken at 100% CPU. 

[Mark Malone]

Hi Triton

it does not look like cef is turned on at all which may help here , when cef is not enabled everything will be process switched which on lower end switches can put pressure on them and nothing is cached to take the burden off your device and create a hardware forwarding table first

That link Jon posted is very good was not aware of it myself great info in it

Have a quick look at your running config and see if you can see ip cef in it should be near the top usually

This may not resolve it completely but may help in certain situations with your multicast traffic

I think Jon is onto something here as well with the TTL when I look at your ip traffic output I see this 15152 bad hop count, this can be normal output when you have multicast traffic set with very low TTLs

[triton.milossedlacek]

Hi Mark,

thank you. There is no record of ip cef in our running config. I'm reading the configuration guide for C2960XR and it says the ip cef or distributed cef shoud be enabled by default. In any case, we'll try to enable it explicitly. 

Do you know the difference between ip cef vs ip cef distributed vs ip route-chache cef?

Yes, we'll check the TTL parameter for our multicasts but I believe it's high enough.

[Mark Malone]

Hi

ok if that's the case that's good as you should nearly always have cef enabled its just one of those features that helps

You should really only need standard ip cef enabled in running config , if its enabled by default it may be hidden in the config show run all you might see it , all the commands that Cisco have in the background will show up with that

from what I have seen dCEF is really only for higher end switches with line cards and asrs etc that sync the route-processor to the fib table maybe its available on 2960xr not sure , basically if you take a 6509 with line cards rather the the RP doing it the line card itself can which again takes the overall burden off the RP

ip route-cache cef is when you just want the interface to do it and not enable it globally on every interface , in some instances you may not want cef enabled on every int and this allows you to turn it off when enabled globally on the device per inetrface

EDIT: Unfortunately if you already have everything set right with TTLs and CEF is enabled you may just be over utilizing the switch and need something with a bit more overall power , these are good switches but they are at the end of the day low end enough compared to some of the dist switches , saying that once your not constantly sitting over 60% I wouldn't be too concerned switch will still work ok

[triton.milossedlacek]

Hi Mark,

I have not looked into sh run all. The cif command might be hidden there.

Thank you for your help so far. I'll let you know next week about our progress.

[triton.milossedlacek]

Hi

after looking into sh run all, I have found these CEF commands:

cef table consistency-check IPv4 type lc-detect count 71 period 60
cef table consistency-check IPv4 type scan-lc-rp count 71 period 60
cef table consistency-check IPv4 type scan-rp-lc count 71 period 60
cef table consistency-check IPv4 type scan-rib-ios count 1000 period 60
cef table consistency-check IPv4 type scan-ios-rib count 1000 period 60
no cef table consistency-check IPv4 data-checking
no cef table consistency-check IPv4 error-message
cef table consistency-check IPv4 auto-repair delay 10 holddown 300
cef table consistency-check IPv6 type lc-detect count 26 period 60
cef table consistency-check IPv6 type scan-lc-rp count 26 period 60
cef table consistency-check IPv6 type scan-rp-lc count 26 period 60
cef table consistency-check IPv6 type scan-ios-rib count 1000 period 60
no cef table consistency-check IPv6 data-checking
no cef table consistency-check IPv6 error-message
cef table consistency-check IPv6 auto-repair delay 10 holddown 300
cef table vrf tree IPv4 type MTRIE short-mask-protection 4 stride-pattern 8-8-8-8 hardware-api-notify off
cef table vrf tree IPv6 type RTREE
cef table output-chain build favor default
cef table rate-monitor-period 5
cef table download recursive-dependents priority 1
cef table download default-route priority 1
cef table download connected-route priority 2
cef table download receive-route priority 2
cef table download route-in-vrf priority 3
cef table download catch-all priority 4

...

ip cef optimize neighbor resolution
no ip cef accounting
ip cef load-sharing algorithm universal 8EF4184C

...

ipv6 cef optimize neighbor resolution
ipv6 cef load-sharing algorithm universal 8EF4184C

...

interface FastEthernet0
  ipv6 mfib cef input
  ipv6 mfib cef output

...

interface Vlan1
  ip route-cache cef
  ipv6 mfib cef input
  ipv6 mfib cef output

...

interface Vlan100
  ip route-cache cef
  ipv6 mfib cef input
  ipv6 mfib cef output

...

interface Vlan200
  ip route-cache cef
  ipv6 mfib cef input
  ipv6 mfib cef output

[Mark Malone]

Yes so cef is enabled on your vlan interfaces which is good I wouldn't change any those other commands there usually set based on what the hardware can do , this morning are you still running a high cpu , is it still ip input the cause ?

[triton.milossedlacek]

Yes, we're still running the 100% CPU. Still the IP input the cause. Does it make sense to apply "ip cef" and "ip cef distributed" commands on global level in this case? Is it possible to check the TTL value of the input multicasts?

[Mark Malone]

Your interface is already part of cef you can still enable it globally but it should be working in the background anyway , the issue was the last time when you posted this show ip cef switching stat there was none , is that still the same output ?

you can check per interface its working with show ip cef vlan x

You may be able to check the ttl in an actual wireshark capture if you do a local  span some of the multicast traffic when that's being sent

if it was a higher end switch we could do a netdr which is a capture of cpu in switch see what exactly is being punted to it unfortunately 29s don't support it

[triton.milossedlacek]

We finally shutdown the ports the majority of multicasts came in and reloaded the switch. After that the CPU is stable at ~35%. I have to mention that we had removed "no routing dynamic" command for both VLANS prior the reload.

So it seems there was a stale process or something similar. Now the IP input process is low.

Thanks guys

[Mark Malone]

That's good to hear , did you re-enable the ports after reboot ? if not could be a case of overload on the switch if shutting down the multicast ports stopped the issue but if they have been re-enabled like you said could have been a jammed process

no routing dynamic just makes an interface passive but you never know what triggered it