topic Re: Prime 3.10.2 Coral to 9800 WLC version 17.8/9/10.x reports down in Wireless

Prime 3.10.2 Coral to 9800 WLC version 17.8/9/10.x reports down

Ethan Grinnell — Thu, 29 Dec 2022 22:37:07 GMT

After upgrading our 9800 WLC to 17.9.2, Prime has started indicating that the Coral connection is down. It was fine in 17.6.x. Prime is 3.10.2 with all of the latest patches. As far as I have been able to determine the actual telemetry stream is fine though. AP discovery completes with no issues, I have client details, etc.

This document and this bug both mention the command:

show telemetry internal protocol cntp-tcp manager x.x.x.x 20828

https://www.cisco.com/c/en/us/support/docs/wireless/catalyst-9800-series-wireless-controllers/214286-managing-catalyst-9800-wireless-controll.html

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvs40684

It appears that Prime uses that command in the "Catalyst 9800 Telemetry Coral Health" service check. However, as of 17.8.1+ that command requires additional arguments (It wants the vrf-id and mgmt IP). Prime doesn't use the VRF-aware command syntax though. I've tested with both 9800-80 and 9800-L-F models, I can't speak for any other 9800 model

wc9800-06#show telemetry internal protocol cntp-tcp manager x.x.x.x 20830 ? source-vrf Source VRF wc9800-06# wc9800-06#show telemetry internal protocol cntp-tcp manager PrimeInfrastructure_IP 20830 source-vrf 0 WLC_MGMT_IP Telemetry protocol manager stats: Con str : PrimeInfrastructure_IP:20830:0:WLC_MGMT_IP Sockfd : 109 Protocol : cntp-tcp State : CNDP_STATE_CONNECTED Table id : 0 Wait Mask : Connection Retries : 0 Send Retries : 0 Pending events : 0 Source ip : WLC_MGMT_IP Bytes Sent : 22288781 Msgs Sent : 20578 Msgs Received : 0 Creation time: : Sun Dec 25 16:32:27:746 Last connected time: : Sun Dec 25 16:32:27:747 Last disconnect time: : Last error: : Connection flaps: : 0 Last flap Reason: : Keep Alive Timeouts: : 0 Last Transport Error : No Error

As a test to see if this was the only thing that is affecting that service, I made an EEM script that translates from the original command to the new syntax. With that script in place Prime's "Catalyst 9800 Telemetry Coral Health" service is happy. So it seems it was that single command's syntax changing that broke it. To be clear, this doesn't affect telemetry actually functioning, it only impacts Prime's reporting of the telemetry status.

event manager applet telemManagerCmd authorization bypass event cli pattern "^show telemetry internal protocol cntp-tcp manager [[:digit:]\.]+ [[:digit:]]+$" enter action 000 set wlc_mgmt_ip" "Unknown" action 001 set wlc_mgmt_vrf_id "0" action 002 set pi_ip "Unknown" action 003 set pi_port "Unknown" action 004 regexp "([[:digit:]\.]+) ([[:digit:]]+)" "$_cli_msg" ignore pi_ip pi_port action 005 cli command "enable" action 006 cli command "terminal length 0" action 007 cli command "terminal width 0" action 008 cli command "show wireless interface summary" action 009 regexp "[[:alnum:]\/]+[[:space:]]+Management[[:space:]]+[[:digit:]]+[[:space:]]+([[:digit:]\.]+)" "$_cli_result" ignore wlc_mgmt_ip action 010 cli command "show telemetry internal protocol cntp-tcp manager $pi_ip $pi_port source-vrf $wlc_mgmt_vrf_id $wlc_mgmt_ip" action 011 puts "$_cli_result"

Is anyone else seeing this with 9800 WLC versions 17.8.1 and higher? I'm curious if there is something particular to our setup where it makes the older CLI command disappear or if Prime uses the VRF-aware command syntax somehow.

Re: Prime 3.10.2 Coral to 9800 WLC version 17.8/9/10.x reports down

Mark Elsen — Mon, 26 Dec 2022 07:24:02 GMT

- FYI : https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvt65754 , check if that could help for your current predicament too ,

Re: Prime 3.10.2 Coral to 9800 WLC version 17.8/9/10.x reports down

Ethan Grinnell — Mon, 26 Dec 2022 08:43:46 GMT

Thanks for the link.

I'm not sure how the Coral version ties into WLC version, it doesn't need to be a 1-to-1 correspondence though. I think that Coral is 17.9.1 in Prime 3.10.2.

Re: Prime 3.10.2 Coral to 9800 WLC version 17.8/9/10.x reports down

balaji.bandi — Mon, 26 Dec 2022 10:10:10 GMT

Do you have any firewall between ? check these ports are allowed :

Cisco Prime Infrastructure to controller: TCP port 830 is used by Cisco Prime Infrastructure to push the telemetry configuration to the controller (using NETCONF).
Controller to Cisco Prime Infrastructure: TCP port 20828 is used for Cisco IOS-XE 16.10.x and 16.11.x, and TCP port 20830 is used for Cisco IOS-XE 16.12x, 17.1.x and later releases.

Re: Prime 3.10.2 Coral to 9800 WLC version 17.8/9/10.x reports down

Ethan Grinnell — Mon, 26 Dec 2022 11:35:35 GMT

There is a firewall in production, no interesting ports blocked.

In the lab I have them in the same network/VLAN and no firewall. Production Prime and WLC aren't in the same network.

Re: Prime 3.10.2 Coral to 9800 WLC version 17.8/9/10.x reports down

balaji.bandi — Mon, 26 Dec 2022 12:21:01 GMT

that just a suggestion, not necessary to be that is a problem. since you mentioned it was working before the upgrade and having errors after the upgrade. this required some TAC involvement.

Personally, since you went to the latest version of Code - cisco moving towards DNAC deployment - rather focusing on Prime Infra. (just thinking some features may fading - may be ?)

Re: Prime 3.10.2 Coral to 9800 WLC version 17.8/9/10.x reports down

eglinsky2012 — Tue, 27 Dec 2022 03:11:28 GMT

Prime 3.10.3 is out. Not sure if this will be resolved, but it might be worth checking out if only for the long list of other bugs resolved.

Re: Prime 3.10.2 Coral to 9800 WLC version 17.8/9/10.x reports down

Ethan Grinnell — Tue, 27 Dec 2022 03:49:06 GMT

Thanks, I'll give that a shot. Not sure how I missed that it came out last Thursday. I guess I need to confirm my notification settings.

Re: Prime 3.10.2 Coral to 9800 WLC version 17.8/9/10.x reports down

Ethan Grinnell — Tue, 27 Dec 2022 06:15:05 GMT

No change. Oh well, it was worth doing anyway.

I opened a TAC case earlier this afternoon

Re: Prime 3.10.2 Coral to 9800 WLC version 17.8/9/10.x reports down

balaji.bandi — Tue, 27 Dec 2022 13:11:29 GMT

Sure new PI released only bug fix, which was reported earlier, but TAC can assist you to fix the issue.

its worth posting back what TAC suggested for the benefit of wide community members.

Re: Prime 3.10.2 Coral to 9800 WLC version 17.8/9/10.x reports down

Rich R — Sat, 31 Dec 2022 00:36:04 GMT

Nice analysis and workaround @Ethan Grinnell

Make sure TAC open a bug for it to get it fixed.

Re: Prime 3.10.2 Coral to 9800 WLC version 17.8/9/10.x reports down

jvodeb — Tue, 21 Feb 2023 13:57:01 GMT

@Ethan Grinnell thanks for workaround.

What did TAC say?

Re: Prime 3.10.2 Coral to 9800 WLC version 17.8/9/10.x reports down

Ethan Grinnell — Fri, 24 Feb 2023 10:22:50 GMT

Still investigating internally

Re: Prime 3.10.2 Coral to 9800 WLC version 17.8/9/10.x reports down

brian.smith — Mon, 20 Mar 2023 18:06:01 GMT

PI = Prime Infrastructure; 9800 is the Cisco 9800-CL (WLC switch)

I too have been running into issues with the Coral Services since upgrading to PI 3.10.x. TAC has always just had me stop and restart services to resolve.

Re: Prime 3.10.2 Coral to 9800 WLC version 17.8/9/10.x reports down

Ethan Grinnell — Fri, 09 Jun 2023 07:06:45 GMT

This will probably be the fix:

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwc30033

TAC says: "There is not workaround (as of now), however, it will be fixed in 3.10.4. ETA for that version has been changing, it is expected to be released late June, early July"

Re: Prime 3.10.2 Coral to 9800 WLC version 17.8/9/10.x reports down

Ethan Grinnell — Tue, 11 Jul 2023 21:11:00 GMT

3.10.4 was released last week. I confirmed that it fixes the issue. They changed to the new VRF aware syntax:

show telemetry internal protocol cntp-tcp manager PrimeInfrastructure_IP 20830 source-vrf 0 WLC_MGMT_IP

Re: Prime 3.10.2 Coral to 9800 WLC version 17.8/9/10.x reports down

spolito — Fri, 07 Jun 2024 18:25:40 GMT

I am running into the same issue. Looks like the bug CSCwc30033 is qualifying. I have updated prime to 3.10.4 on patch update 3 which is the latest release.

Currently have a TAC case open but wanted to replay to see if anyone has found a workaround.

Re: Prime 3.10.2 Coral to 9800 WLC version 17.8/9/10.x reports down

spolito — Fri, 07 Jun 2024 18:57:46 GMT

Ive updated Prime on 3.10.4 on update 3. The 9800 WLC is on 17.19.4 and im still running into this issue. Looks like I may still be hitting CSCwc30033

Re: Prime 3.10.2 Coral to 9800 WLC version 17.8/9/10.x reports down

Ethan Grinnell — Sat, 08 Jun 2024 02:15:50 GMT

Well that's strange, 3.10.4 fixed my issue. I'm also fully patched and not seeing a problem. Since they changed the syntax of the command they're using to "VRF aware", is there any chance you are using VRF on the 9800?

Re: Prime 3.10.2 Coral to 9800 WLC version 17.8/9/10.x reports down

perrymcgrew — Tue, 27 Aug 2024 12:53:52 GMT

Our 9800-L-F is 17.12.2 and Prime Infrastructure is 3.10.5 and we are still seeing the Coral error. Since it appears that it is cosmetic, not going to worry about it.