C9800-L-C-K9 giving a CRITICAL Physical Port Link Down

[CourtCopex]

Hello, we've got a C9800-L-C-K9 unit giving us this error.  Any experience with

WLC#show facility-alarm status
System Totals  Critical: 6  Major: 0  Minor: 0
Source                     Time                   Severity      Description [Index]
------                     ------                 --------      -------------------
TwoGigabitEthernet0/0/0    Dec 15 2023 05:48:54   CRITICAL      Physical Port Link Down [1]
TwoGigabitEthernet0/0/1    Dec 15 2023 05:48:54   CRITICAL      Physical Port Link Down [1]
TwoGigabitEthernet0/0/2    Dec 15 2023 05:48:54   CRITICAL      Physical Port Link Down [1]
TwoGigabitEthernet0/0/3    Dec 15 2023 05:48:54   CRITICAL      Physical Port Link Down [1]
TenGigabitEthernet0/1/0    Dec 15 2023 05:48:54   CRITICAL      Physical Port Link Down [1]
TenGigabitEthernet0/1/1    Dec 15 2023 05:48:54   CRITICAL      Physical Port Link Down [1]

WLC#show log

Syslog logging: enabled (0 messages dropped, 3 messages rate-limited, 0 flushes, 0 overruns, xml disabled, filtering disabled)

No Active Message Discriminator.

No Inactive Message Discriminator.

    Console logging: level debugging, 21 messages logged, xml disabled,

                     filtering disabled

    Monitor logging: level debugging, 0 messages logged, xml disabled,

                     filtering disabled

    Buffer logging:  level debugging, 159 messages logged, xml disabled,

                    filtering disabled

    Exception Logging: size (4096 bytes)

    Count and timestamp logging messages: disabled

    Persistent logging: disabled

No active filter modules.

    Trap logging: level informational, 153 message lines logged

        Logging Source-Interface:       VRF Name:

    TLS Profiles:

Log Buffer (4096 bytes):

TPOINT_CREATE: Trustpoint: CISCO_IDEVID_SUDI_LEGACY0 created succesfully

*Dec 15 05:49:30.509: %PKI-6-TRUSTPOINT_CREATE: Trustpoint: CISCO_IDEVID_SUDI created succesfully

*Dec 15 05:49:30.515: %CRYPTO_ENGINE-5-KEY_ADDITION: A key named CISCO_IDEVID_SUDI has been generated or imported by pki-sudi

*Dec 15 05:49:30.545: %PKI-6-TRUSTPOINT_CREATE: Trustpoint: CISCO_IDEVID_SUDI0 created succesfully

*Dec 15 05:49:32.067: %CRYPTO_ENGINE-5-KEY_ADDITION: A key named TP-self-signed-2281856104 has been generated or imported by crypto-engine

*Dec 15 05:49:32.068: %SSH-5-ENABLED: SSH 1.99 has been enabled

*Dec 15 05:49:32.107: %PKI-4-NOCONFIGAUTOSAVE: Configuration was modified.  Issue "write memory" to save new IOS PKI configuration

*Dec 15 05:49:33.927: %CRYPTO_ENGINE-5-KEY_ADDITION: A key named TP-self-signed-2281856104.server has been generated or imported by crypto-engine

*Dec 15 05:49:40.353: %PKI-6-TRUSTPOINT_CREATE: Trustpoint: SLA-TrustPoint created succesfully

*Dec 15 05:49:40.355: %PKI-4-NOCONFIGAUTOSAVE: Configuration was modified.  Issue "write memory" to save new IOS PKI configuration

*Dec 15 05:49:40.359: %CALL_HOME-6-CALL_HOME_ENABLED: Call-home is enabled by Smart Agent for Licensing.

*Dec 15 05:49:42.161: %CRYPTO_ENGINE-4-CSDL_COMPLIANCE_RSA_WEAK_KEYS: RSA keypair CISCO_IDEVID_SUDI_LEGACY is in violation of Cisco security compliance guidelines and will be rejected by future releases.

*Dec 15 05:50:14.414: %IM-6-IOX_ENABLEMENT: Chassis 1 R0/0: ioxman: IOX is ready.

*Dec 15 05:50:48.664: %SYS-5-CONFIG_P: Configured programmatically by process DHCP Autoinstall from console as vty0

*Dec 15 06:01:07.730: %SYS-5-USERLOG_NOTICE: Message from tty0(user id: Device in day0 workflow, some non user-configured options may be enabled by default

*Dec 15 06:01:07.733: %SYS-5-CONFIG_P: Configured programmatically by process Setup from console as vty0

*Dec 15 06:01:09.698: %LINK-5-CHANGED: Interface GigabitEthernet0, changed state to administratively down

*Dec 15 06:01:14.417: %IOXN_APP-6-PRE_INIT_DAY0_GS_INFO: Day0 Guestshell pre-initilization API is being invoked

*Dec 15 06:01:26.421: %SYS-5-CONFIG_P: Configured programmatically by process PnP Agent Discovery from console as vty0

*Dec 15 06:01:26.424: %SYS-5-CONFIG_P: Configured programmatically by process PnP Agent Discovery from console as vty0

*Dec 15 06:01:26.424: %PNP-6-PNP_SAVING_TECH_SUMMARY: Saving PnP tech summary (/pnp-tech/pnp-tech-discovery-summary)... Please wait. Do not interrupt.

*Dec 15 06:01:26.553: %SYS-5-CONFIG_P: Configured programmatically by process PnP Agent Discovery from console as vty0

*Dec 15 06:01:26.742: %SYS-5-CONFIG_P: Configured programmatically by process PnP Agent Discovery from console as vty0

*Dec 15 06:01:26.826: %SYS-5-CONFIG_P: Configured programmatically by process PnP Agent Discovery from console as vty0

*Dec 15 06:01:26.950: %SYS-5-CONFIG_P: Configured programmatically by process PnP Agent Discovery from console as vty0

*Dec 15 06:01:27.060: %PNP-6-PNP_SUDI_UPDATE: Device SUDI [PID:C9800-L-C-K9,SN:FCL25250080] identified

*Dec 15 06:01:27.109: %SYS-5-CONFIG_P: Configured programmatically by process PnP Agent Discovery from console as vty0

*Dec 15 06:01:27.214: %PNP-6-PNP_TECH_SUMMARY_SAVED_OK: PnP tech summary (/pnp-tech/pnp-tech-discovery-summary) saved successfully (elapsed time: 1 seconds).

*Dec 15 06:01:27.214: %PNP-6-PNP_DISCOVERY_STOPPED: PnP Discovery stopped (Config Wizard)

*Dec 15 06:01:44.418: %IOXN_APP-6-PRE_INIT_DAY0_GS_INFO: Day0 Guestshell de-initilization API is being invoked

*Dec 15 06:01:46.523: %UICFGEXP-6-SERVER_NOTIFIED_STOP: Chassis 1 R0/0: psd: Server iox has been notified to stop

*Dec 15 06:01:47.448: %IOXN_APP-6-IOX_START_STOP_REQ: Got IOX DOWN COMPLETE event, invoking registered callback(s)

*Dec 15 06:01:48.436: %PMAN-3-PROC_EMPTY_EXEC_FILE: Chassis 1 R0/0: pvp: Empty executable used for process service_mgr

*Dec 15 06:01:49.621: %PMAN-3-PROC_EMPTY_EXEC_FILE: Chassis 1 R0/0: pvp: Empty executable used for process bt_logger

*Dec 15 06:11:45.425: %SYS-6-TTY_EXPIRE_TIMER: (exec timer expired, tty 0 (0.0.0.0)), user

WLC#

WLC#

WLC#show version

Cisco IOS XE Software, Version 17.06.03

Cisco IOS Software [Bengaluru], C9800 Software (C9800_IOSXE-K9), Version 17.6.3, RELEASE SOFTWARE (fc4)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2022 by Cisco Systems, Inc.

Compiled Wed 30-Mar-22 23:12 by mcpre

Cisco IOS-XE software, Copyright (c) 2005-2022 by cisco Systems, Inc.

All rights reserved.  Certain components of Cisco IOS-XE software are

licensed under the GNU General Public License ("GPL") Version 2.0.  The

software code licensed under GPL Version 2.0 is free software that comes

with ABSOLUTELY NO WARRANTY.  You can redistribute and/or modify such

GPL code under the terms of GPL Version 2.0.  For more details, see the

documentation or "License Notice" file accompanying the IOS-XE software,

or the applicable URL provided on the flyer accompanying the IOS-XE

software.

ROM: 16.12(3r)

WLC uptime is 28 minutes

Uptime for this control processor is 29 minutes

System returned to ROM by LocalSoft

System image file is "bootflash:/packages.conf"

Last reload reason: LocalSoft

[M02@rt37]

Hello @CourtCopex 

Physically inspect the cables and connections on the affected interfaces. Ensure they are securely connected, and there are no obvious issues.

Please, use the command show interfaces status to check the status of the interfaces. Look for any indications of errors, and verify if the interfaces are in an administratively down state.

[marce1000]

 - You may also want to look into the outputs of :
                show platform resources
                  show platform
                 show inventory
                 show environment

 M.
                 

[Leo Laohoo]

---

@CourtCopex wrote:

TwoGigabitEthernet0/0/0    Dec 15 2023 05:48:54   CRITICAL      Physical Port Link Down [1]
TwoGigabitEthernet0/0/1    Dec 15 2023 05:48:54   CRITICAL      Physical Port Link Down [1]
TwoGigabitEthernet0/0/2    Dec 15 2023 05:48:54   CRITICAL      Physical Port Link Down [1]
TwoGigabitEthernet0/0/3    Dec 15 2023 05:48:54   CRITICAL      Physical Port Link Down [1]
TenGigabitEthernet0/1/0    Dec 15 2023 05:48:54   CRITICAL      Physical Port Link Down [1]
TenGigabitEthernet0/1/1    Dec 15 2023 05:48:54   CRITICAL      Physical Port Link Down [1]

---

Compare the time and date with the uptime of the controller.  Do they match?

[Scott Fella]

What interface do you have connected to the network?  Interfaces that are not connected to a switch that are in a down state, will show up in that command.  Here is a screenshot of mine using the 10gig in LAG:

[Rich R]

Have you updated ethernet PHY to 17.11.01 as per https://www.cisco.com/c/en/us/td/docs/wireless/controller/9800/config-guide/b_upgrade_fpga_c9800.html#id_131526
And upgrade IOS-XE as per https://www.cisco.com/c/en/us/support/docs/wireless/catalyst-9800-series-wireless-controllers/214749-tac-recommended-ios-xe-builds-for-wirele.html#toc-hId-2052757410

Then check the port configurations and other items as mentioned by the others above.  Also use the Config Analyzer (link below) to check your config.

Are you expecting all those ports to be up?