WLC 9800 restart and CRC and input errors on Cisco switches

[syedhaq]

Hi, 

We are facing a major issue with new installed WLC 9800, we are seeing CRC and input errors on switch 2960 and 3850 when connected to WLC, TAC could not find the issue and throwing ball to each other from Wireless team to Switching team and vise versa. 

Every weak as the CRC and input error increase the active WLC restarts and the passive becomes active and cannot ping the gateway and all APs get disconnected. once the other WLC become online from there we can ping gateway but as it is passive.

We changed cables, we changed switch ports, we also change the location from 3850 to 2960  switch but nothing helped.

As per TAC this is happening because switch CPU is more than 90% but i dont this is the issue, LAN users and other on cisco WLC working perfectly. 

We need this issue to be escalated to higher level of support to resolve ASAP as we have invested a huge amount in this solution and management is too upset.

Appreciate if anyone help on this. thanks in advance

Regards

Syed

[Leo Laohoo]

Can I see the interface output that shows the CRC errors? 
NOTE:  DO NOT POST screenshots.

[Arshad Safrulla]

Which model of WLC do you have?

How the WLC is connected to the Core Switch (Fiber, Copper cable & SFP type and part number used at both switch and WLC)?

I would recommend hardcoding the speed and duplex settings from the WLC side if you are connecting to any mGig or SFP ports. I see investigating why switch is hitting 90% CPU will be a good place to start before the WLC issue as this can have a major impact than the WLC. 

[Scott Fella]

If you need this escalated, then you need to do that with TAC.  On the forum, we "normal folks" can only help suggest things.  

You never mentioned what model and firmware your 9800, ap's or switches are on?  Have you tried to break HA and use N+1 in the meantime?  You mentioned, "management is upset", so its time to figure out a way you can get this stable and that is to break SSO.  Once you get the network stable, then lab it out and validate that everything is configured right and see if in the lab, you see errors or not.  Might be the code you are running, who knows.  I'm not a fan of SSO from my experience and saying that we need SSO for clients is not really a good use case, since SSO can also ruin the user experience.  This is my opinion, and do I test and use SSO... I do, in the lab, but have never deployed it yet with the 9800's in production.  I even removed SSO from our old AireOS due to some issues we were seeing.

[Rich R]

So the interface errors and SSO failover may be connected.

Like others have said you need to provide full details of hardware models and software versions but generally speaking:

- We use SSO and it's been working fine on 9800-80's currently running 17.5.1

- Depending on SSO setup it needs to ping default gateway to confirm availability.  So interface errors could affect that and if you're using your switch as router (default gateway) then high CPU could definitely affect ping replies causing failover (you should have a failover reason in the logs).

- high CPU on a switch is unusual - what is causing that high CPU?  If you're using it as a router that could be the cause.

- interface errors are generally port, SFP or cable. So swap and test each to eliminate.

- if you're using copper cables you need to test them with a good quality tester.  Cheaper testers will just test pin to pin continuity but not pick up incorrect pairs which cause crosstalk which results in errors when you try to run traffic over the cable at high frequencies.  Obviously check cable length.  What does switch port TDR test show for the cable as a start?

[mgonzalez15]

Hi Syed.

Did you find a solution for this problem? I have a similar issue with the same model of WLC's 9800. I have CRC and input errors on the switch, it would be helpful if you found a solution.

Thank you.

[CiscoWiFiGuy]

Hi,

Did you also find a solution? i am also facing the same issue with 9800-40 wlc on 17.9.3 version.

[Rich R]

Did you read my reply below @CiscoWiFiGuy ?

Open a new thread with full and complete details of your problem and all the troubleshooting steps you have completed with relevant outputs.

[Rich R]

@mgonzalez15 the original poster @syedhaq never even bothered to respond to the many helpful replies here!

If you had read any of those replies you'd have seen that we can't offer meaningful help without all the relevant details.

I'd recommend opening a new post with a FULL description of your problem, what steps have been taken, troubleshooting done etc and the full details of all hardware and software versions involved.   Take note of all the feedback already given above to make sure you include all the required info.  Make sure you're running up to date software and firmware on all your equipment for a start.