3850 broadcast forwarding and multicast issues

[ROBERT THOMPSON]

Hello,

We are experiencing broadcast and multicast issues on a 3850 stack. The 3850 stack is the wireless controller to local ap's. 

Multicasts or mdns between subnets is not working. Broadcast forwarding is not working.

The switch is configured as an mdns gateway globally and is running permit any for all services on upstream and downstream. Wireless multicast and wireless broadcast enabled for all vlans.  

We are having issues discovering Sonos music devices and airplay devices even though I can see them in the mdns cache. 

Also, wireless clients stop working randomly.  Almost as if they have power saving enabled if they are "idle" for a while. 

When I put wireshark on the vlan using a wireless client, I dont see broadcast or ssdp traffic. 

Does anybody have an idea? Are we hitting ios xe arp,  multicast and broadcast bug number Cscuv89764?  We are on the 03.06.02ae code.

Any help appreciated 

Rob

[Rasika Nayanajith]

I would suggest 3.6.3E code. Go for that and then see if problem persist.

Rasika

[ROBERT THOMPSON]

Hi,

No, that did not work as we had complete wireless failure after 24 hours of running the software. All Access Points stopped working. All Radio interfaces down (see attached).

It may be that we are hitting IOS XE bug. Not great software as we have seen similar on 4500E's using 3.6. code where forwarding and arp requests fail.

Cisco have a fix but the software is not downloadable yet

Details

Last Modified:

Oct 16,2015

Status:

Fixed

Severity:

2 Severe

Product:

Cisco Catalyst 3850 Series Switches

Support Cases:

4

Known Fixed Releases:

(1)

16.2(0.151)

Known Fixed Releases:

(1)

16.2(0.151)

We will have to buy new 2504 controller and 3560x as Layer3 switch and just use 3850's as very expensive Layer2 switches.

Regards

Rob

[gkloepfer]

Rob-

Don't expect L2 functionality to be any better.  We've had a case open for the better part of a month now where multicast traffic from our security camera gateway doesn't traverse the stacking ring on the 3850 switches (we are using this product entirely as a L2 device).  It is not clear if the issue is related to the stacking ring, or inter-ASIC because the equipment (in addition to the security gateway devices) we need to place on these switches is in production.  I know of at least one of our users who's application requires multicast to work.

If you can go back to IOS-XE 3.3.5SE, that version does not appear to have the problem (tested on our C3850-48T switches).  Since we're setting up a stack that includes three C3850-12X48U switches (which only works on 3.7.x and later), we are not able to go to the earlier release.

At the moment, we have very expensive paper weights.  There are a few servers that don't do any multicast traffic on the switches only because we don't want to interrupt service to move them again.

I've become really frustrated with Cisco as of late.  This kind of "release it than let the customer test it" looks to be becoming the norm with Cisco, and it saddens me.

On the plus side, I'm glad we're not the only one experiencing the problem.

Best of luck in resolving your issue as well.

-- Gil.

[ROBERT THOMPSON]

Hi Gil,

I rolled them back to 3.7.2 code and did the following:

Turn off dynamic arp inspection and dhcp snooping and all works as planned. (I had this problem in IOS XE 3.6 on the Cisco 4500-E, so we ditched the SUP8's and exchanged for SUP7's and the problem disappeared! We had to do this: no macro auto monitor

https://supportforums.cisco.com/discussion/12162221/catalyst-4506-e-sup-8e-arp-issues-any-vlan )

However, the issue with using the WLAN Controller still persisted so we installed a 2504 controller with a 3560X as layer 3 core switch and the 3850's as very expensive Layer2 switches. The controller was directly patched into the 3560X and the AP's onto the 3850's. The 3560X then L2 trunked onto 3850's using L2 Portchannels. FYI - This was for our customer, so we had to absorb the cost in the end. 

So I guess the "problem" followed the XE code through all releases. It may not be a problem in code but actually understanding why they turn things on that really should be our job to turn on or off. 

Its all working now using the above additional hardware.

Not the best solution but seen as though . There is some very basic explanations of what the arp inspection with dhcp snooping does so it makes sense. 

Have a look at this after digesting no Macro Auto Monitor

http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3850/software/release/3-2_0_se/autosmartports/configuration_guide/iosaspcg/configure.html 

Cheers

Rob

[gkloepfer]

Hi Rob-

It sounds like your issue is a bit different from the one we're experiencing (which appears to be related to the bug ID that has been discussed).  Our intent has been to use the 3850s as very fast L2 switches (obviously, we're using a fair amount of Cisco-specific management features as well, which is why this switch would be useful to us).

The problem we're experiencing is that multicast traffic (at the least, the specific traffic related to one application we have), with switching only (no routing), does not actually get from one stack member to another in a 3850 stack on IOS-XE 3.7.2E (which you have to be running on the WS-C3850-12X48U as the earlier versions don't support this hardware).  The thinking right now is that that bug ID CSCuv89764 (mentioned earlier) is related to this problem, although Cisco's engineering team is still not sure yet (they haven't made it clear to us, anyway).  On the WS-C3850-48T hardware, the earlier IOS-XE revision does not exhibit the behavior, while the later IOS-XE does (same configuration).

I did check out to see if there was any difference between IP ARP inspection or DHCP snooping on the two switch stacks, just in case that was the silver bullet that could have resolved this, but unfortunately they are the same in both configurations (both disabled).  That was actually a good thing to check, though, so thank you for pointing me in that direction.

For your customer, just be aware that there does look like a switching problem if you stack 3850s together and are trying to switch multicast traffic from one stack member to another on the 3.7 train of IOS-XE.  If you do encounter the problem. and your hardware and configuration allow you to revert back to 3.3.5SE, then that could be a good workaround.  Otherwise, you'll join the group of us who have stumbled on this bug.

Just for further amusement, we've had issues with the 4500-X on certain IOS-XE releases when the unit is in a VSS configuration where it would start dropping packets due to internal corruption in certain tables (again, a bug that was finally fixed in the next IOS-XE release).

Unfortunately, these problems don't bode well for us when the people we're supporting start having connectivity issues following what we communicate as "network upgrades."

Again, it sounds like you have resolved the problems you encountered, so while it was frustrating it does sound like you have a suitable workaround for the moment.

-- Gil.

[petwang]

We run into this bug on 3 sites with 3650 switches that has Wireless Controller enabled and APs attached.

The more severe cases were related to using VLAN 1 for user traffic. So we rolled back to 03.06.02ae code, and moved user traffic to a different VLAN. So far so good.

In another case, DHCP request got blocked/stopped by 3650 switches with 03.07.01E code. Shutdown/unshut the uplink interface fixed that issue.

[petwang]

BTW, the issue started immediately after WLAN/Mobile controller configured && AP turned on on those switches (L2 function only).

[ROBERT THOMPSON]

Hi,

See my post below to fix the DHCP and ARP issues

Cheers

Rob