AV devices using mDNS not working between switches in an SDA Fabric

[JamesChristley2230]

Hello,

We have a Cisco SDA fabric with 9300 edge switches and a single 9500 border node, with DNAC version 2.3.3.6-70045. The fabric in question contains a VN with native ASM multicast enabled in the overlay and an IP pool with Layer 2 flooding turned off. The underlay network has been provisioned with LAN automation and has multicast enabled.

Our AV team have been working with us to test some Dante audio devices that utilise mDNS for a PC connected to the same IP pool to discover audio devices on the network. We have found that, with an IP pool with layer 2 flooding disabled, the discovery only works with devices on the same switch stack. However, with layer 2 flooding enabled, the discovery works. Packet capture suggests that the mDNS traffic does not make it between the switches. It also suggests that the audio receivers are not sending IGMP joins for the mDNS group (224.0.0.251) - also does not appear in a 'show ip igmp snooping groups' on the switches. We inititally thought it was a TTL problem in the multicast packets, causing it to be droppped at the first hop. However, packet captures suggest that mDNS is being sent out with a TTL of 255.

Does anyone have any experience with mDNS on SDA fabrics? I'm not too familiar with the protocol, so is it the same as other multicast traffic or are there specific things that need to be configured for this to function?

Thanks,

James

[Flavio Miranda]

Hi

 From protocol perspective I had been seeing mDNS as multicast protocol.  For SDA you may need L2VN . 

I'd recommend you to read this Cisco paper:

https://www.cisco.com/c/en/us/solutions/collateral/enterprise-networks/sd-access-wired-wireless-dg.html 

this is related to Bonjour which also uses mDNS solution. 

And here how to enable L2VN in DNAC

https://www.cisco.com/c/en/us/td/docs/cloud-systems-management/network-automation-and-management/dna-center/2-2-3/user_guide/b_cisco_dna_center_ug_2_2_3/b_cisco_dna_center_ug_2_2_3_chapter_01110.html 

[Rahul Kachalia]

Hello Flavio,

Implementing L2VN for mDNS use-cases is not the recommended solution. It is in-secure, unscalable and it may put adverse performance impact on WLC when implemented in large scale environment.

While mDNS is using Layer 2 Multicast, the Catalyst switches and WLCs learns the meta-data based on IT defined policies and securely implement end-to-end mDNS service-routing for any network designs SD-Access, Traditional L2/L3 networks, MPLS VPNs and BGP EVPN VXLAN based fabric networks.

You may find following blog that captures several challenges with classic "Service-on-a-Stick" model that is being now resolved using Wide Area Bonjour solution that is completely Unicast based service-routing solution:

Multicast DNS - Still Flooding?

In Reference section of this blog you may find several collaterals to guide our customers to finally end mDNS flooding across Wired and Wireless networks and instead route based on policies in complete Unicast form.

thanks,

rahul.

[Rahul Kachalia]

Hello James,

Cisco Catalyst 9000, Nexus 9000 and Catalyst 9800 series WLC - can route mDNS services in Unicast form, instead flooding over L2VNs OR using traditional Multicast routing protocols on Wired or Wireless VLANs. You may only need Multicast between WLC Mgmt VLAN + AP Mgmt VLAN to enable AP Multicast.

On each Catalyst 9300 series FE switches, you may implement following policy to enable "Local Area Bonjour" this means Wired mDNS Dante Audinate + Wireless endpoints can communicate within single Layer 3 switch: 

mdns-sd service-definition dante_apps
 service-type _netaudio-arc._udp.local
 service-type _dante-safe._udp.local
 service-type _dante-upgr._udp.local
 service-type _netaudio-chan._udp.local
 service-type _netaudio-cmc._udp.local
 service-type _netaudio-dbc._udp.local
mdns-sd service-list LOCAL-AREA-SERVICES-IN IN
 match dante_apps
mdns-sd service-list LOCAL-AREA-SERVICES-OUT OUT
 match dante_apps
!
vlan configuration <AV VLAN>, <Wired User VLAN>, <Wireless User VLAN>
 mdns-sd gateway
 service-policy LOCAL-AREA-SERVICES-IN in
 service-policy LOCAL-AREA-SERVICES-OUT out
!

You may also need to implement Wide Area Bonjour. This means if Dante AV services needs to be discovered beyond single Layer 3 switch boundary - then you need to route Dante AV services to central controller which is DNA-Center.

The Wide Area Bonjour Application in DNA-Center enables global mDNS service-routing based on policy. You may route AV, Apple TV, ChromeCast or any other types of mDNS services from one point of network to other based on their services locations OR even micro-locations (services zones).

You may want to use following guide to implement the end-to-end Local Area and Wide Area Bonjour solution for SD-Access:

Deployment Guide : PDF

Quick Configuration Guide : PDF

thanks,

rahul.

[JamesChristley2230]

Thanks for your response Rahul.

In this case, the AV devices are connected to different FE switches and a device connected to switch A needs to discover the devices connected to switch B all within the same VLAN, VN and IP pool. From your reponse, I would infer that this needs Wide Area Bonjour to be configured. Is that correct?

I have started reading up on this from the paper Flavio provided and I need to work with the AV team to work out what the service types are for the Dante devices so we can test this out in our lab fabric.

James

[Rahul Kachalia]

Hello James,

Yes - Just like IP routing, if you need mDNS services to be discovered beyond single IP gateway switch boundary = you will need Wide Area Bonjour.  The DNA-Center will need to be configured with appropriate routing policy to discover DANTE services from one FE and route it to other FE switch within same floor.

We have other DANTE customers successfully implemented mDNS service-routing solution in SD-Access with above described custom policy. You will need similar configuration done on DNA-Center side.

thanks,

rahul.

[Rahul Kachalia]

Hello James,

Apparently, I was corrected by our engineering team that in latest Cisco IOS-XE you may simply need to configure "match audinate" in service-list instead creating custom service-definition as described above. Apology if any confusion caused, but something that you may consider for ease of deployment.

thanks,

rahul.

[JamesChristley2230]

Hi Rahul,

I know it's been a little while since your reply, but I have finally had a chance to read through the documentation provided and test this on our lab fabric.

I have configured the two fabric edge nodes, directly connecting a Shure microphone and a laptop running Dante controller (doing the discovery), with the mdns policy based on your template above.

---

mdns-sd gateway
ingress-client query-suppression enable
mdns-sd service-list LOCAL-AREA-SERVICES-IN IN
match audinate
mdns-sd service-list LOCAL-AREA-SERVICES-OUT OUT
match audinate
mdns-sd service-list DNAC-CONTROLLER-SERVICES OUT
match audinate
mdns-sd service-policy LOCAL-AREA-POLICY
service-list LOCAL-AREA-SERVICES-IN IN
service-list LOCAL-AREA-SERVICES-OUT OUT
mdns-sd service-policy DNAC-CONTROLLER-POLICY
service-list DNAC-CONTROLLER-SERVICES OUT
service-export mdns-sd controller WIDE-AREA-BONJOUR-POLICY
controller-address <DNAC IP>
controller-service-policy DNAC-CONTROLLER-POLICY
controller-source-interface Loopback0

interface vlan1023

  mdns-sd gateway
  service-policy LOCAL-AREA-POLICY
  active-query timer 90
  transport ipv4

---

---

I have also set up the domains, with the two FE switches as SDG agents, and two service filter policies. One from switch A and B and the other for the return direction (example shown below):

The 'Audinate Dante' service type is one I created which contains the PTR records used by Dante.

After all this, I am still not able to succesfully discover devices between different FE switches (it does work when the mic and laptop are connected to the same switch). I believe all the configuration is correct, based on how I understood the documentation. Unfortunately, I can find little or no documentation on how mDNS works specifically from Audinate.

Any help at this point would be greatly appreciated as I'm unsure where to go from here.

Many thanks,

James

[Rahul Kachalia]

Hi James,

Thanks for the update. And we can willingly help.

If the Mic and Laptop works on same switch but not across, seems to be some service routing issue that we need to look into it.

I request you to open up the TAC case and copy me (kachalia@cisco.com) on the case. Please upload configurations and other information to analyze offline. As next week is Cisco Live, I will not be able to help much. But please suggest TAC engineer to setup Webex meeting in week of 6/12 and we can help you to resolve the issue.

thanks,

rahul.

[LC.IT]

@JamesChristley2230 Did you found a solution?

[Stefano7K]

Hello Experts!!
We are facing some issues to configure mDNS for Audinate-Dante in our SD-Access fabric, where also the Wireless network is in Fabric-mode.
These are the steps implemented so far:
- Deploy LAN-Automation with flag "Enable Multicast" and build Fabric
- Create VN "IoT" with 2 pool: "IoT_Wired", "IoT_Wireless". For both pool we tried to enable/disable also L2 Flooding. Now is disabled
- Create ssid "IoT" type fabric and associate the previous "IoT_Wireless"
- Deploy Native Multicast ASM on VN "IoT", defining internal RP the 2 Border.
- Configure in every FE the following CLI template:

mdns-sd service-list LOCAL-AREA-SERVICES-IN IN
match audinate
mdns-sd service-list LOCAL-AREA-SERVICES-OUT OUT
match audinate
mdns-sd service-list DNAC-CONTROLLER-SERVICES OUT
match audinate

mdns-sd service-policy LOCAL-AREA-POLICY
service-list LOCAL-AREA-SERVICES-IN IN
service-list LOCAL-AREA-SERVICES-OUT OUT

mdns-sd service-policy DNAC-CONTROLLER-POLICY
service-list DNAC-CONTROLLER-SERVICES OUT

mdns-sd gateway
mode sdg-agent
active-query timer 1

service-export mdns-sd controller DNAC
controller-address <DNAC-IP>
controller-source-interface Loopback 0
controller-service-policy DNAC-CONTROLLER-POLICY

interface vlan <IoT_WIRED>
mdns-sd gateway
service-policy LOCAL-AREA-POLICY
active-query timer 90
transport ipv4

interface vlan <IoT_WIRELESS>
mdns-sd gateway
service-policy LOCAL-AREA-POLICY
active-query timer 90
transport ipv4

- Configure than the Wide Area Bonjour application from DNAC GUI.
- Define a custom "Audinate Dante" service type as James suggested
- Create a service filter in a subdomain:
- network mode: traditional
- service type: Audinate Dante
- then configuring the SDG agent by adding every FE:
- type:source
- service layer: local
- interface: IoT_Wired, IoT_Wireless

The Dante devices on IoT_Wired are working fine also in different FE.
At the moment we cannot discover from IoT_Wireless the IoT_Wired device via mDNS.

1) It is not clear if we have to activate Multicast on WLC in Fabric mode, or other setting. At the moment is in default, so Multicast disable.
2) mDNS source and query and randomly distribuited through all FE...I do not understand how to configure in this case. A SDG agent seems can be source or query, but not both role.

Any help at this point would be greatly appreciated
Thanks!
Stefano

[jalejand]

Hi Stefano

For Bonjour in fabric enabled wireless, you need the following things:

Configure wireless multicast in SD Access
Configure a wireless multicast group (like 225.1.1.1 or anythinig of your choice).
Enable "ip pim passive" under the AP VLAN SVI on every FE.


WLC configuration:
wireless multicast
wireless multicast 225.1.1

Fabric Edges:

interface vlan 2045 (change it with your AP MGMT VLAN)
 ip pim passive

As your fabric underlay is already enabled for multicast, this will enable the capability of Fabric APs to pass multicast traffic over VXLAN using their access tunnels (not that multicast will be necesarily encapsulated using capwap, this just enables the capabiliy).

To verify which APs were enabled with multicast over multicast, use the following command:

WLC1#show ap multicast mom
AP Name MOM-IP TYPE MOM-STATUS
-----------------------------------------------------------------
APA453.0E5B.3EF8 IPv4 Up



In the case you do not see the APs in UP state, you will need to troubleshoot the multicast trees from the FE to the WLC which is the multicast source.

Some verifications include:

* Verify the S,G (or *,G in case there is no S,G, it means something is wrong in the multicast path between the WLC and FE) in the FEs in global routing table "show ip mroute 225.1.1.1"

* Trace down the shared tree for the multicast group from the FE to the RP
* Verify if the S,G is registered on the underlay RP
* Verify if the SPT tree is converging as it should from FE to WLC.

For your second query:
2) mDNS source and query and randomly distribuited through all FE...I do not understand how to configure in this case. A SDG agent seems can be source or query, but not both role.

In a single service filter, you can have devices acting either as query or source but not both, but  if you create a second service filter, you can reverse their roles, the logic is like an ACL.

There are some explanations about both wireless multicast in SDA and bidirectional rules in the following doc:

https://www.ciscolive.com/c/dam/r/ciscolive/global-event/docs/2023/pdf/BRKTRS-3011.pdf

Let me know if you have any more doubts or questions

[Stefano7K]

Hi Ale,

Many thanks for sharing this info.

Let me schedule this implementation and I will provide the result.

Stefano

[LC.IT]

@Stefano7K Did you found a solution?

[AndyOReilly29406]

Having issues with Dante on our SDA too, all are Wired. We've not configured Wide Area Bonjour.

Started out with the AV team deploying two rooms on two separate floors. We have Multicast configured on the AV Virtual Network, we have Layer 2 Flooding configured on the AV VN.

AV devices connected to the stack on Floor 1 can send streams to AV devices connected to the stack on Floor 2, that seems to work without issue even if the AV Team say they don't want that to happen.

However, the main issue comes with the Precision Time Protocol that the devices use to elect a Master Clock.

The AV Team found that they had two Master Clocks, one on Floor 1 and one on Floor 2. Devices on Floor 1 were using the Master Clock on Floor 1. Devices on Floor 2 were using the Master Clock on Floor 2.

Packet captures could see the devices receiving PTP multicast packets from both master clocks and other devices.

Whilst troubleshooting we got to a stage where we only had 1 Master Clock, we still don't know why, and 2 devices on Floor 1 were able to use the Master Clock on Floor 2, All Devices on Floor 2 were using the Master Clock on Floor 2 but the rest of the devices on Floor 1 wouldn't talk to the Master Clock.

Another packet capture of one of the devices on Floor 1 that wouldn't use the Master Clock and we could see it receiving PTP packets from the Master Clock so should have been working.

And from what the AV Team described, if they can't reach a Master Clock they should elect a new one themselves which also wasn't happening.

We raised it with Audinate and they said that Dante isn't supported on a Software Defined Networks

We raised it with Cisco and they pretty much said you need Professional Services.

We're looking into disabling Layer 2 Flooding so that each Stack is separate and has it's own Master Clock.

From reading the other posts configuring Wide Area Bonjour is required for the mDNS but we're seeing some of the traffic do this without Bonjour.

Anyone else got any suggestions or solutions?