Introduction:

Kureli Sankar · ‎09-08-2010

Introduction:
Documentation:
Support:
Configuring multicast:
Shared tree and Source specific tree:
Where to add these commands:
igmp join-group:
igmp static-group:
igmp forward interface:
Terminology:
What is a first hop router:
What is a last hop router:
OIL:
In coming interface and out going interface for (*,G):
In coming interface and out going interface for (S,G):
DR - How to increase the pim priority:
PIM Chart:
How to interpret the "sh mroute x.x.x.x"
sample 1:
sample 2:
sample 3:
How to interpret the "sh mfib x.x.x.x"
sample 1:
sample 2:
Common Problems:
syslog 106010:
mfib limit (5000) reached:
Failed to locate egress interface:
HSRP:
TTL (Time to Live Value) too low or set to 1:
Multlicast over VPN tunnel:
FP no mcast output intrf
Debug pim group x.x.x.x shows Assert processing message wins
Troubleshooting commands:
On the Firewall:
On the Router:
Packet capture sample:

Introduction:

The purpose of this document is to provide assistance to everyone in configuring and troubleshooting multicast through the firewall.

Documentation:

This document is meant to be interpreted with the aid of the official documentation from the configuration guide located here:

ASA:http://www.cisco.com/en/US/docs/security/asa/asa82/configuration/guide/route_multicast.html

FWSM:http://www.cisco.com/en/US/docs/security/fwsm/fwsm32/configuration/guide/ip_f.html#wp1041648

PIX:http://www.cisco.com/en/US/docs/security/pix/pix63/configuration/guide/bafwcfg.html#wp1170913

The Cisco TAC has created another ASA Multicast Troubleshooting and Common Problems guide and posted it to Cisco.com:

http://www.cisco.com/en/US/products/ps6120/products_tech_note09186a0080befd2a.shtml

Support:

ASA supports igmp forwarding, sparse mode, and bidirectional mode. There is no sparse-dense-mode support. ASA forwards auto-rp packets (unless configured to not to). ASA itself requires static auto-rp config.

Configuring multicast:

Pls. follow this link: http://www.cisco.com/en/US/products/ps6120/products_configuration_example09186a00807631d2.shtml

Shared tree and Source specific tree:

I read once that Multicast is like dating service. Sender starts the stream and the DR on the local LAN registers the sender with the dating service (RP). Reciever sends IGMP report and the DR on that local LAN segment sends a PIM join towards the RP- shared tree. Once the dating service connects the two - sender and receiver, then they can talk via the shortest path and do not have to go through the dating service - shared tree anymore.

Where to add these commands:

igmp join-group:

This command when applied under the interface, sends a join for the group address. Think of this as a permanent receiver for the group. This comes in handy when the receivers off of the interface do not send igmp reports in a regular interval. This command makes the firewall to accept and forward the multicast packets. Configuring the firewall to join a multicast group causes upstream routers to maintain multicast routing table information for that group and keep the paths for that group active.

Where to add this command: This command needs to be configured under the interface config mode facing the receivers.

hostname(config-if)# igmp join-group group-address

igmp static-group:

The firewall does not accept the multicast packets but rather forwards them to the specified interface. To configure the security appliance to forward the multicast traffic without being a member of the multicast group, use the igmp static-group command. When this command is added, the ASA sends out an IGMP report out the interface sourcing with its IP address on the interface.

Where to add this command: This command needs to be applied under the interface config facing the receivers.

hostname(config-if)# igmp static-group group-address

igmp forward interface:

Where to add this command: This command when applied under the interface config facing the receivers.

This command will forward all the igmp reports received on the interface towards the interface where the server is located. This command cannot be configured along with PIM.

hostname(config-if)# igmp forward interface outside

Terminology:

What is a first hop router:

A first hop router is the router that is connected to the source which is responsible for registering the source with the RP. If there are two routers connected to the same segment as the source then the one that is the DR (designated router) for that LAN segment will take care of the registering process. Now, if one of the two is the firewall and you want the firewall to take care of the registration process because the RP is on the other side of the firewall then, the firewall should be DR for that LAN segment.

What is a last hop router:

A last hop router is the one that is connected to the receiver. Again if there are two or more routers in the LAN connected to the receiver, the router which is the DR is the one that is resonsible to connect the reciver to the shared tree. If one of the devices is the firewall then, if we want the firewall to process the igmp reports then it has to be the DR with a higher priority.

OIL:

OIL stands for Outgoing Interface List. OIL is always taken from the (*,G) and copied as the OIL for the (S,G).

In coming interface and out going interface for (*,G):

Incoming interface for the (*,G) is always towards the RP. Out going interface is towards the receiver.

In coming interface and out going interface for (S,G):

Incoming interface for (S,G) is always towards the source. Out going interface is towards the receiver.

DR - How to increase the pim priority:

The following example sets the DR (Designated Router) priority for the interface to 5:

hostname(config-if)# pim dr-priority 5

PIM Chart:

(*,G)
(S,G)

When it is created	By receipt of IGMP Membership report from directly-connected Receiver (host) Dynamically created when an (S,G) entry must be created.	By receipt of (S,G) PIM join By receipt of Multicast packet
OIL info	Interface that received IGMP membership report Interface that received PIM (*,G) Join Manually configured If none of the above, "NULL"	Interface that received a PIM (S,G) join Cop of (*,G) OIL except when matching (S,G) IIF Otherwise "NULL"
RPF info	IP address of next-hop neighbor towards RP according to unicast routing table. If on the RP itself then "NULL"	IP address of next-hop neighbor towards the Multicast Source If directly connected to Multicast Source then 0.0.0.0
IIF Info	Interface that leads to RP according to unicast routing table If on the RP itself then "NULL"	Interface that leads to Multicast Source according to unicast routing table.
When this entry is deleted?	when (S,G) entry expires When notified by the IGMP process that all members for a group are gone. By receipt of PIM (*,G) Prune message from downstream neighbor.	After 210 sec. if no multicast packets or PIM register messages received from SPT. When (*,G) entry deleted.

How to interpret the "sh mroute x.x.x.x"

sample 1:

topology:
Receiver--RP---(INT_SCI/8)ASA(INT_MANDAT/0)--(10.2.112.9)source

Group address: 239.252.1.10

ASA#sh mroute 239.252.1.10

Multicast Routing Table
Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group,
       C - Connected, L - Local, I - Received Source Specific Host Report,
       P - Pruned, R - RP-bit set, F - Register flag, T - SPT-bit set,
       J - Join SPT
Timers: Uptime/Expires

(10.2.112.9, 239.252.1.10), 00:10:48/00:03:11, flags: SFT
  Incoming interface: INT-MANDAT
  RPF nbr: 10.2.112.9, Registering
  Outgoing interface list:
    INT-SCI, Forward, 00:10:48/00:03:29
    Tunnel0, Forward, 00:10:48/never

There is no *,G which indicates that the receiver hasn't requested the feed yet. Only the source has started and the source lives 
behind the interface named INT-MANDAT. "F" - is the Register Flag. Meaning the firewall hasn't seen a register stop message 
from the RP yet. The "Tunnel0" interface is a transient state and goes away once the register stop is received. 
RP lives behind the INT-SCI interface.

sample 2:

RPF neighbor all zeros means that the ASA is the first hop router connected local to the sender.

When you see the "T" flat you can jump in joy. This mean the shortest path tree has formed and multicast traffic is being received by the receiver.

topology:

receiver--(inside)ASA(outside)--sender

ASA#sh mroute 230.0.0.10

(*, 230.0.0.10), 00:08:20/never, RP 0.0.0.0, flags: SCL (Connected Local LAN)

Incoming interface: Null -----> receiver is directly connected
RPF nbr: 0.0.0.0
Immediate Outgoing interface list: ------> receiver is off of the inside interface
inside, Forward, 00:08:20/never

(10.135.152.47, 230.0.0.10), 00:08:16/00:03:13, flags: SJT (shortest Path Tree - T)
Incoming interface: outside ------> source is on the outside interface.
RPF nbr: 0.0.0.0
Inherited Outgoing interface list:
inside, Forward, 00:08:20/never ------> receiver is off the inside interface.

sample 3:

topology:

sender(172.25.1.11)--RP(172.20.50.254)--vlan50--nonpci(0)ASA(0)WLAN--vlan30--Receiver(172.20.30.133) 
same security level - 0

Group add 225.4.5.7

ASA#sh mroute 225.4.5.7
(*, 225.4.5.7), 01:08:42/never, RP 172.20.50.254, flags: SCLJ
   Incoming interface: nonpci ------> (routing table shows int nonpci towards the RP)
   RPF nbr: 172.20.50.254
   Immediate Outgoing interface list: ------> (receiver is off the interface WLAN)
     WLAN, Forward, 01:08:42/never

(172.25.1.11, 225.4.5.7), 00:04:59/00:03:00, flags: SJT
   Incoming interface: nonpci ------>(routing table shows int nonpci for the source)
   RPF nbr: 172.20.50.254
   Inherited Outgoing interface list:
     WLAN, Forward, 01:08:42/never------> (receiver is off the interface WLAN)

How to interpret the "sh mfib x.x.x.x"

sample 1:

topology:

(10.204.125.81)R1-------(10.204.125.85/OUT)***ASA***(IN/172.25.250.11)-R2(172.25.250.1)--receiver.
                             I
                       --------------
                       |             |
 Sender-10.29.95.133 RP-10.29.95.252
Group address: 233.49.81.127

Source or source is on the lower security interface and the receiver is on the higher security interface. It is very important to see the
"F" flag so indicate that the ASA is forwarding multicast traffic.

ASA5520-01#    sh mfib
Entry Flags: C - Directly Connected, S - Signal, IA - Inherit A flag,
             AR - Activity Required, K - Keepalive
Forwarding Counts: Pkt Count/Pkts per second/Avg Pkt Size/Kbits per second
Other counts: Total/RPF failed/Other drops
Interface Flags: A - Accept, F - Forward, NS - Negate Signalling 
             IC - Internal Copy, NP - Not platform switched
             SP - Signal Present
Interface Counts: FS Pkt Count/PS Pkt Count

(*,233.49.81.127) Flags: C K
   Forwarding: 0/0/0/0, Other: 846486/2/846484
   OUT Flags: A NS  ----> We are accepting packets on the outside interface
   inside Flags: F NS ---> We are forwarding packets to the inside interface.
     Pkts: 0/0
              
(10.29.95.133,233.49.81.127) Flags: K
   Forwarding: 1/0/36/0, Other: 0/0/0
   OUT Flags: A
   inside Flags: F NS
     Pkts: 0/1

sample 2:

topology:

receiver--(inside)ASA(XETRA)--router--N/W-RP-Source

source: 193.29.93.62
Group address: 224.0.46.0 

ASA5520-01# sh conn 

UDP XETRA 193.29.93.62:25100 inside 224.0.46.0:55199, idle 0:00:00, bytes 2649468, flags - 

ASA5520-01# sh mfib 224.0.46.0 193.29.93.62

Entry Flags: C - Directly Connected, S - Signal, IA - Inherit A flag,
             AR - Activity Required, K - Keepalive
Forwarding Counts: Pkt Count/Pkts per second/Avg Pkt Size/Kbits per second
Other counts: Total/RPF failed/Other drops
Interface Flags: A - Accept, F - Forward, NS - Negate Signalling 
             IC - Internal Copy, NP - Not platform switched
             SP - Signal Present
Interface Counts: FS Pkt Count/PS Pkt Count



(193.29.93.62,224.0.46.0) Flags: K   
 Forwarding: 18629/10/1295/105, Other: 4716/0/4716  
XETRA Flags: A F  ---> XETRA interface is where the source is and we Accept packets from.
     Pkts: 0/0 
inside Flags: F NS ---> inside interface is where the receiver is where we Forward the packets to.  
     Pkts: 1598/2

Common Problems:

syslog 106010:

topology:

receiver---(inside)FWSM(outside)---sender (10.11.21.10)

group add: 239.226.16.3

%FWSM-3-106010: Deny inbound udp src OUTSIDE:10.11.21.10/1450 dst DMZ-EMP90:239.226.16.3/30120

With nat-control enabled on the FWSM the above issue can be corrected with the following:

nat (inside) 0 access-list 101
access-list 101 permit ip host 239.226.16.3 host 10.11.21.10

Think of the above as the source when sends out multicast packets the destination is the multicast address 239.226.16.3 
and is destined towards the inside. Even though no traffic will be sourced from a multicast address, we just need to provide
translation for the reverse flow from high security to low security.

If this case were an ASA we would see the following in asp drop captures

1. ASA was dropping all the multicast packet for the reason below:
union-asa# sh cap capasp | i 239.0.1.2
   38: 12:07:56.782689 10.80.8.38>  239.0.1.2: icmp: echo request Drop-reason: (no-mcast-intrf)

2. nat 0 with acl was configured on this ASA and it didn't allow exemption for our group 239.0.1.2. 
Once I added that flow reached the other ASA2 and we saw *,g as well as s,g

mfib limit (5000) reached:

"sh mfib <group-address>" may not show any output. This may due to the fact that the mfib entries max limit may have been reached.  This ENH request CSCtj22365 is filed so, that when resolved, the FWSM will send a syslog message indicating that the max mfib entry limit has been reached.

fwsm# sh mfib sum
IPv4 MFIB summary:
   5000 total entries [4974 (S,G), 23 (*,G), 3 (*,G/m)]
  190440 total MFIB interfaces

Failed to locate egress interface:

Oct 23 2010 21:40:44: %ASA-6-110002: Failed to locate egress interface 
for UDP from outside:10.135.152.47/1034 to 230.0.0.10/7060

Look for contradicting lines in the config. In this case where does the sender live on the inside or outside?

static (inside,outside) 10.135.152.47 10.135.152.47 net 255.255.255.255
mroute 10.135.152.47 net 255.255.255.255 outside

In the above case we had to remove the static and "toggle" multicast-routing on the ASA.

HSRP:

Let us say that the RP is a loop back address configured on a pair of routers running HSRP. We need to make sure the mroute configured on the firewall points to one of the physical IP addresses and not the HSRP address. When we issue "sh pim int <name>" , the firewall only forms neighbor relationship with the physical IP addresses and not the HSRP address. Also, we cannot configure mroute to both the physical IP addresses. There is no redundancy when it comes to multicast.

Pls. read this link for further explanation: http://www.cisco.com/en/US/tech/tk828/technologies_tech_note09186a0080094aab.shtml

TTL (Time to Live Value) too low or set to 1:

When a router forwards a multicast packet from one interface to another, it decrements the Time To Live (TTL) value in the IP header by one. The firewall does not decrement the TTL value in the IP header but, still it will drop the packet if the TTL is set to 1. It sends the packet only if the resulting TTL value is not zero and is greater than the multicast TTL threshold value of the outgoing interface when configured. If a multicast application on the source allows the setting of a TTL value and results in not matching the mentioned criteria, the packets are dropped by the router. The multicast source needs to set the TTL value as there are number of hops between the source and the receiver.

Multicast TTL threshold:

http://www.cisco.com/en/US/customer/tech/tk828/technologies_tech_note09186a0080094b55.shtml#ttlsetting

Multicast TTL value too low or set to 1:

http://www.cisco.com/en/US/customer/tech/tk828/technologies_tech_note09186a0080094b55.shtml#ttlthreshold

Multlicast over VPN tunnel:

ASA/PIX will not pass multicast traffic over IPsec VPN tunnels. Only unicast traffic over VPN tunnel is supported. The alternative option is to use GRE between two end points on either side of the tunnel and send multicast over GRE and encrypt that traffic and send it over the tunnel.

FP no mcast output intrf

R1(receiver)--(inside)ASA(outside)---R3(rp/sender)

Group 239.1.1.1

"capture capasp typs asp-drop all" may show the following:

ASA1# sh cap capasp | i 239.1.1.1

18: 14:52:02.979731 10.7.123.3.64281 > 239.1.1.1.5000: udp 16 Drop-reason: (no-mcast-intrf) FP no mcast output intrf

Here is the command reference link to sh asp-drop:

Name: no-mcast-intrf

FP no mcast output intrf:

All output interfaces have been removed from the multicast entry.

- OR -

The multicast packet could not be forwarded.

Recommendation:

Verify that there are no longer any receivers for this group.

- OR -

Verify that a flow exists for this packet.

Syslogs:

None

In this case R1 being the receiver was also the DR for that segment so, the ASA did not do anything with the

IGMP reports that it received. In this case making the ASA as the DR resolved the issue.

Debug pim group x.x.x.x shows Assert processing message wins

Topology:

Group address 239.1.1.247

vlan13 - INSIDE

vlan300 - OUTSIDE

FWSM (10.20.213.1)----vlan 13----FWSM---vlan300--RTR----RP---Sender (172.16.41.205)

|

Receivers (10.20.213.204)----|

|

ASA(10.20.13.2)----|

IGMP reports from the receivers arrive on the FWSM but, the FWSM doesn't send PIM join up to the RP. The egress multicast packets do not show the joins due to CSCsf31515

Debug pim group 239.1.1.247 showed the following:

IPv4 PIM: (172.16.41.205,239.1.1.227)RPT J/P adding Prune on OUTSIDE

IPv4 PIM: (172.16.41.253,239.1.1.227)RPT J/P adding Prune on OUTSIDE

IGMP: Received v2 Report on INSIDE from 10.20.213.201 for 239.1.1.227

IGMP: Updating EXCLUDE group timer for 239.1.1.227

IPv4 PIM: (172.16.41.205,239.1.1.227) Received [15/110] Assert from 10.20.13.2 on INSIDE

IPv4 PIM: (172.16.41.205,239.1.1.227) Assert processing message wins

IPv4 PIM: (172.16.41.205,239.1.1.227) INSIDE Update assert timer (winner 10.20.13.2)

FWSM (backup site)

ASA (production site)

FWSM and ASA though configured to be on the same vlan, were not supposed to see each other. The problem is that the FWSM saw the presence of the ASA but the ASA didn't see the presence of the FWSM. FWSM was DR on VLAN 13 and the ASA was DR (since it did not see the FWSM) as well. Both tried to process multicast packets and ended in a race condition and the Assert messages and winner indicated that.

For more information read this link:

Troubleshooting commands:

On the Firewall:

sh pim neighbor

sh igmp interface

sh igmp traffic

sh mroute x.x.x.x

sh mfib x.x.x.x

debug pim group x.x.x.x

debug igmp group x.x.x.x

On the Router:

sh ip pim interface

sh ip mroute x.x.x.x

sh ip igmp group

debug ip pim

Packet capture sample:

http://packetlife.net/captures/category/multicast/

1. igmp report - IGMP Ver 2 - Multicast packet, destined to the multicast group address.

2. PIM join - PIM Ver 2 - Multicast packet destined to 224.0.0.13

3. Register packet - Is a Unicast packet sent from First hop router to RP address and the multicast packet is encapsualted.

4. Register Stop - Is a Unicast packet sent from the RP IP address to first hop router.

Marcin Latosiewicz · ‎09-20-2011

Excellent stuff Kureli!

Stuart Patton · ‎04-30-2014

I know this is a document regarding ASA/PIX, but if you have AutoRP configured on your routers, can you use the AutoRP address as the RP address for unsupported devices like ASA/PIX?

If that is supported, does the answer change if you use AnycastRP's peered with MSDP? Should you use a separate (non-anycast) loopback on the anycast routers as the PIM RP?

john thoren · ‎03-30-2020

Thank you for this wonderful set of explanations and details.

ASA-PIX/FWSM: Multicast tips and common problems

Introduction:

Documentation:

Support:

Configuring multicast:

Shared tree and Source specific tree:

Where to add these commands:

igmp join-group:

igmp static-group:

igmp forward interface:

Terminology:

What is a first hop router:

What is a last hop router:

OIL:

In coming interface and out going interface for (*,G):

In coming interface and out going interface for (S,G):

DR - How to increase the pim priority:

PIM Chart:

How to interpret the "sh mroute x.x.x.x"

sample 1:

sample 2:

sample 3:

How to interpret the "sh mfib x.x.x.x"

sample 1:

sample 2:

Common Problems:

syslog 106010:

mfib limit (5000) reached:

Failed to locate egress interface:

HSRP:

TTL (Time to Live Value) too low or set to 1:

Multicast TTL threshold:

Multicast TTL value too low or set to 1:

Multlicast over VPN tunnel:

FP no mcast output intrf

Debug pim group x.x.x.x shows Assert processing message wins

Troubleshooting commands:

On the Firewall:

On the Router:

Packet capture sample: