Multicasting performance Issues

brettevans34 · ‎10-26-2015

Hi , I hope you can help?

I have two 6509's connected together. Using STP and HSRP, I currently have around 80 vlans on the network with odd vlans going to core1 and even Vlans going to Core2. Performance of the 6509's has always been pretty good until I created a further 80 vlans. Terminal emulation to the cores started to go slow. Looking at all the interface vlans I noticed a lot of traffic being dropped, roughly every minute which coincided with terminal emulation going slow.

Using 'Netdr' I captured the traffic being dropped, which resulted in the majority of traffic captured being multicast traffic 224.0.1.40.

Each interface vlan is configured for PIM sparse-dense-mode. The below configuration show multicasting on our cores:

Core1:

interface Loopback100
ip address 1.1.1.2 255.255.255.255
ip pim sparse-dense-mode

ip pim autorp listener
ip pim send-rp-announce 1.1.1.2 scope 20
ip pim send-rp-discovery scope 20

Core2:

interface Loopback100
description PIM-RP-ADDRESS
ip address 1.1.1.1 255.255.255.255
ip pim sparse-dense-mode

ip pim autorp listener
ip pim send-rp-announce 1.1.1.1 scope 20
ip pim send-rp-discovery scope 20
!

ip route 1.1.1.2 255.255.255.255 192.168.1.6 (192.168.1.6 being the physical IP address of core 1 vlan1)

Question: A static route on core 1 is not present to get to 1.1.1.1 (core2). If I add this static route will it have a knock on affect? and/or will it stop the multicast traffic bursts.

Thanks

Brett

Jon Marshall · ‎10-26-2015

Brett

So core 1 does not have a route in it's table for core 2's loopback interface ?

When you did the packet capture was that on both switches or just core 1 ?

As a side note why are you running sparse-dense mode and also the autorp listener feature ?

Jon

brettevans34 · ‎10-26-2015

Hi Jon, Thanks for taking the time to reply to my query.

So core 1 does not have a route in it's table for core 2's loopback interface?

No, Core 1 doesn't have a route to the loopback interface of Core2

When you did the packet capture was that on both switches or just core 1 ?

The Captures I took were from Core2.

As a side note why are you running sparse-dense mode and also the autorp listener feature ?

Core configuration for multicasting hasn't changed since the Cores were installed 5 years ago by our 3rd party provider at the time. My multicast knowledge is limited so I never queried the config

Thanks

Brett

Jon Marshall · ‎10-26-2015

Brett

Okay I thought perhaps because core 1 didn't have a route it was something to do with the RPF check and perhaps this was causing traffic to be sent to the RP but obviously that isn't the case as you did the packet capture on core 2.

Thinking about it some more if you are seeing lots of traffic drops I am not sure this ties in with the packet capture ie. the multicast announcements are not large so they may not be related.

Do you have other L3 devices connected to the 6500 ?

What does the CPU usage on the 6500 look like ie. "sh process cpu" ?

Jon

brettevans34 · ‎10-26-2015

Hi Jon,

The cat 6509's are the only L3 devices on the network. Whilst I was capturing traffic from Core 2. CPU utilisation was very high. the average I would say was 45% but on burst (every minute) it would be over 80%.

I disabled all the new vlans I created and the CPU reduced dramatically, at this moment both cores are running average between 10% and 20% and burst maximum to around 60%

From the captures I took from core2, each vlan physical ip address sent 156 auto-rp packets to 224.0.1.40. with 160 vlans that's nearly 25K packets

thanks

Brett

Jon Marshall · ‎10-26-2015

Brett

Did you create the new vlans with the exact same configuration as the existing ones.

Are there any clients on the new vlans requesting the multicast streams or any new sources of multicast streams on those new vlans ?

Jon

brettevans34 · ‎10-27-2015

Hi Jon,

Please see the outputs from the 'sh ip pim rp mapping' command:

Core 1:

Core_1#sh ip pim rp mapping
PIM Group-to-RP Mappings
This system is an RP (Auto-RP)
This system is an RP-mapping agent

Group(s) 224.0.0.0/4
RP 1.1.1.2 (?), v2v1
    Info source: 1.1.1.2 (?), elected via Auto-RP
         Uptime: 1y2w, expires: 00:02:42
RP 1.1.1.1 (?), v2v1
    Info source: 1.1.1.1 (?), via Auto-RP
         Uptime: 1w0d, expires: 00:02:36

Core 2:

Core2#sh ip pim rp mapping
PIM Group-to-RP Mappings
This system is an RP (Auto-RP)
This system is an RP-mapping agent

Group(s) 224.0.0.0/4
RP 1.1.1.2 (?), v2v1
    Info source: 1.1.1.2 (?), elected via Auto-RP
         Uptime: 1d16h, expires: 00:02:00
RP 1.1.1.1 (?), v2v1
    Info source: 1.1.1.1 (?), via Auto-RP
         Uptime: 2y14w, expires: 00:01:59

The new vlans I created were exactly the same as the old ones, where all odd vlans for L2 & L3 are serviced by Core1 and even vlans by Core 2. The new vlans I created are not in service yet for access layer users.

Brett

Arjun Sarkar · ‎10-27-2015

Hello Brett,

Just curious at the time of high CPU which process was high? Was it "PIM process"? Also in debug netdr capture do you see any packets with TTL=1?

Also I would like to understand is there any impact for not receiving multicast feed?

Best Regards
Arjun

brettevans34 · ‎10-29-2015

Hi Arjun

From the captures the TTL=19

The PIM processes were low but the IP input process was high 30% plus

from the logs i'm seeing a lot of the following messages which do stop when i disable the new vlans i created

Oct 29 03:28:19.013: %SCHED-3-STUCKTMR: Sleep with expired timer 47AC4020, time
0x7ADEDF3E0 (00:00:00 ago). -Process= "IP Input", ipl= 5, pid= 262 -Traceback= 413A15E0 413A19C4 409B06CC 41358804 413587F0

Thanks

Brett

Arjun Sarkar · ‎10-29-2015

Hello Brett ,

Can you please post comeplete log along with show version? Also you see high CPU after configuring new SVI on the switch ONLY?

Jon Marshall · ‎10-26-2015

Brett

I can't see how adding that route would affect any existing communications but worth doing it in a quiet time.

If it is the RP announcements causing the issue and the 6500s are the only L3 devices then you could simply configure static RPs on each switch which would mean each switch did not need to send RP announcements.

What does a "sh ip pim rp-mapping" on each switch currently show ?

Jon