%TRANSCEIVER-3-RPC_FAILED: Chassis 1 R0/0: Where do I start??

[brianw360]

I have distribution C9500 switch that has this re-occuring error  %TRANSCEIVER-3-RPC_FAILED: Chassis 2 R0/0: iomd: Application error rc = 6: -Traceback= 1#251b3a6af82efeb430dadc10a45c4c2d errmsg:7FF93C672000+A80 iomd_spa_infra:7FF949085000+5969A iomd_spa_infra:7FF949085000+420CF iomd_spa_infra:7FF949085000+42E4D spa_ipc_shim:7FF92BB81000+1360DC if_combined_ngmod:7FF91EBAB000+15757D if_combined_ngmod:7FF91EBAB000+157357 :55EDA3769000+66C1B evutil:7FF938F55000+7966 evutil:7FF938F55000+7745 evlib:7FF93D148000+8F37 evlib:7FF93D148000+997C :55EDA3769000+65E54 :55EDA3769000+65C24 :55EDA3769000+DC

OR227-DXA1#sho ver
Cisco IOS XE Software, Version 16.12.04
Cisco IOS Software [Gibraltar], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 16.12.4, RELEASE SOFTWARE (fc5)

The chassis number will cycle between 1 and 2 since there are two stack members.  

I digress....where and what tools (aside from Google) can I use to troubleshoot this issue. I figured this would be best place to ask.  I've tried the Bug Search Tool, but it gives me the error in the screenshot.

TIA

[marce1000]

              - FYI : https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvj80183

        Adding additional note , for bug search , avoid using Chrome -> use other browser

 M.

[brianw360]

Switched to MS Edge.   Still getting the same error as in the screenshot I posted with the original post. "The Bug Id seems to be an invalid format......".   What is the preferred format?  What is best way to tackle the error messages on a switch?  I came into a organization that did a network upgrade, and no one here has had time to fine tune it.  

[marce1000 · Accepted solution]

 - Can you 'read' the link about the bug as I posted it directly (pressing on it)  ? There are no known fixed release mentioned in the bug report , try current advisory release : https://software.cisco.com/download/home/286315863/type/282046477/release/Amsterdam-17.3.5 . if that does not help , contact Cisco TAC.

 M.

[Leo Laohoo]

Please post the complete output to the following commands: 

1. sh version
2. dir flash-2:
3. dir flash-2:core
4. dir crashinfo-2:
5. sh log on switch 2 uptime detail

NOTE:  16.12.X is "toxic" to 9300, 9400, 9500 and 9600.  I strongly recommend upgrading to the latest 17.3.X.

[brianw360]

I agree with the upgrade from what I've seen in the community, and I've alerted my management that the IOS version we are running will not be supported very much longer.  But considering I work for a major company, I do not have the ability to enact that kind of change.

OR227-DXA1#sho ver
Cisco IOS XE Software, Version 16.12.04
Cisco IOS Software [Gibraltar], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 16.12.4, RELEASE SOFTWARE (fc5)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2020 by Cisco Systems, Inc.
Compiled Thu 09-Jul-20 21:49 by mcpre

Cisco IOS-XE software, Copyright (c) 2005-2020 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0. For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.

ROM: IOS-XE ROMMON
BOOTLDR: System Bootstrap, Version 17.1.1[FC2], RELEASE SOFTWARE (P)

OR227-DXA1 uptime is 1 year, 35 weeks, 4 days, 23 hours, 8 minutes
Uptime for this control processor is 1 year, 35 weeks, 4 days, 23 hours, 12 minutes
System returned to ROM by Image Install at 15:49:07 PST Tue Sep 1 2020
System restarted at 09:13:13 PST Sat Sep 26 2020
System image file is "flash:packages.conf"
Last reload reason: Image Install

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

Technology Package License Information:

------------------------------------------------------------------------------
Technology-package Technology-package
Current Type Next reboot
------------------------------------------------------------------------------
network-advantage Smart License network-advantage
dna-advantage Subscription Smart License dna-advantage
AIR License Level: AIR DNA Advantage
Next reload AIR license Level: AIR DNA Advantage

Smart Licensing Status: UNREGISTERED/EVAL EXPIRED

cisco C9500-24Y4C (X86) processor with 1867864K/6147K bytes of memory.
Processor board ID CAT2413L27W
7 Virtual Ethernet interfaces
48 TwentyFive Gigabit Ethernet interfaces
8 Hundred Gigabit Ethernet interfaces
32768K bytes of non-volatile configuration memory.
16010152K bytes of physical memory.
11161600K bytes of Bootflash at bootflash:.
11161600K bytes of Bootflash at bootflash-2:.
1638400K bytes of Crash Files at crashinfo:.
1638400K bytes of Crash Files at crashinfo-2:.
0K bytes of WebUI ODM Files at webui:.

Base Ethernet MAC Address : ac:3a:67:e4:10:80
Motherboard Assembly Number : 4874
Motherboard Serial Number : CAT2413L27W
Model Revision Number : V02
Motherboard Revision Number : 2
Model Number : C9500-24Y4C
System Serial Number : CAT2413L27W

Switch 02
---------
Base Ethernet MAC Address : ac:3a:67:e4:15:80
Motherboard Assembly Number : 4874
Motherboard Serial Number : CAT2412L58K
Model Revision Number : V02
Motherboard Revision Number : 2
Model Number : C9500-24Y4C
System Serial Number : CAT2412L58K

OR227-DXA1#

OR227-DXA1#dir flash:
Directory of bootflash:/

275810 drwx 4096 May 8 2020 23:04:39 -07:00 .dbpersist
275815 -rw- 805827585 Sep 26 2020 08:35:54 -07:00 cat9k_iosxe.16.12.04.SPA.bin
275816 drwx 4096 Sep 26 2020 09:10:28 -07:00 .installer
275817 -rw- 555 Sep 26 2020 09:10:30 -07:00 bootloader_evt_handle.log
275819 drwx 4096 May 8 2020 23:13:42 -07:00 .ssh
275826 drwx 4096 Aug 23 2021 22:42:00 -07:00 core
324481 drwx 4096 Sep 26 2020 09:13:18 -07:00 .prst_sync
340705 drwx 4096 Sep 26 2020 09:00:26 -07:00 .rollback_timer
348817 drwx 4096 May 8 2020 23:13:55 -07:00 gs_script
356929 drwx 4096 Sep 26 2020 09:07:22 -07:00 tech_support
356943 drwx 4096 Sep 26 2020 09:10:46 -07:00 dc_profile_dir
275813 -rw- 396 Sep 26 2020 09:10:46 -07:00 boothelper.log
275829 -rw- 132108 Sep 26 2020 09:11:06 -07:00 memleak.tcl
275830 drwx 4096 May 8 2020 23:14:33 -07:00 onep
275835 -rw- 2896 Jul 12 2021 17:08:04 -07:00 vlan.dat
275831 -rwx 1953 Sep 26 2020 09:11:06 -07:00 svl_ipc.tcl
324484 -rw- 14464008 Sep 26 2020 08:55:13 -07:00 cat9k-cc_srdriver.16.12.04.SPA.pkg
324485 -rw- 89134084 Sep 26 2020 08:55:13 -07:00 cat9k-espbase.16.12.04.SPA.pkg
324486 -rw- 1684484 Sep 26 2020 08:55:13 -07:00 cat9k-guestshell.16.12.04.SPA.pkg
324487 -rw- 531063808 Sep 26 2020 08:55:13 -07:00 cat9k-rpbase.16.12.04.SPA.pkg
324493 -rw- 41799673 Sep 26 2020 08:55:32 -07:00 cat9k-rpboot.16.12.04.SPA.pkg
324488 -rw- 31478788 Sep 26 2020 08:55:13 -07:00 cat9k-sipbase.16.12.04.SPA.pkg
324489 -rw- 60392448 Sep 26 2020 08:55:13 -07:00 cat9k-sipspa.16.12.04.SPA.pkg
324490 -rw- 23217156 Sep 26 2020 08:55:13 -07:00 cat9k-srdriver.16.12.04.SPA.pkg
324491 -rw- 12907520 Sep 26 2020 08:55:13 -07:00 cat9k-webui.16.12.04.SPA.pkg
381270 drwx 4096 Sep 26 2020 09:12:05 -07:00 callhome
275842 -rw- 7553 Sep 26 2020 09:00:04 -07:00 packages.conf
275818 -rw- 396 Sep 26 2020 09:07:23 -07:00 boothelper.log.old
324492 -rw- 9216 Sep 26 2020 08:55:13 -07:00 cat9k-wlc.16.12.04.SPA.pkg
405603 -rw- 27231232 Jun 16 2020 15:05:28 -07:00 cat9k-cc_srdriver.16.09.04.SPA.pkg
405604 -rw- 81241084 Jun 16 2020 15:05:28 -07:00 cat9k-espbase.16.09.04.SPA.pkg
616513 -rw- 1647612 Jun 16 2020 15:05:28 -07:00 cat9k-guestshell.16.09.04.SPA.pkg
616514 -rw- 425411576 Jun 16 2020 15:05:28 -07:00 cat9k-rpbase.16.09.04.SPA.pkg
616520 -rw- 34612458 Jun 16 2020 15:05:44 -07:00 cat9k-rpboot.16.09.04.SPA.pkg
616515 -rw- 28931068 Jun 16 2020 15:05:28 -07:00 cat9k-sipbase.16.09.04.SPA.pkg
616516 -rw- 55047160 Jun 16 2020 15:05:28 -07:00 cat9k-sipspa.16.09.04.SPA.pkg
616517 -rw- 35243004 Jun 16 2020 15:05:28 -07:00 cat9k-srdriver.16.09.04.SPA.pkg
616518 -rw- 15856632 Jun 16 2020 15:05:28 -07:00 cat9k-webui.16.09.04.SPA.pkg
616519 -rw- 9208 Jun 16 2020 15:05:28 -07:00 cat9k-wlc.16.09.04.SPA.pkg
324483 -rw- 7560 Sep 26 2020 08:55:32 -07:00 cat9k_iosxe.16.12.04.SPA.conf
275839 -rw- 0 Sep 26 2020 09:13:18 -07:00 rdope_out.txt
275841 -rw- 91 Sep 26 2020 09:13:33 -07:00 rdope.log

11250098176 bytes total (8261337088 bytes free)
OR227-DXA1#

OR227-DXA1#dir flash:core
Directory of bootflash:/core/

332593 drwx 4096 May 8 2020 23:13:53 -07:00 modules
275843 -rw- 670893 Aug 23 2021 11:31:15 -07:00 OR227-DXA1_1_RP_0_bash_6732_20210823-113115-PST.core.gz
275844 -rw- 670845 Aug 23 2021 22:42:00 -07:00 OR227-DXA1_1_RP_0_bash_20741_20210823-224200-PST.core.gz

OR227-DXA1#dir crashinfo-2:
Directory of crashinfo-2:/

86770 drwx 32768 Jun 3 2022 08:23:32 -07:00 tracelogs
87230 drwx 4096 Sep 26 2020 09:16:32 -07:00 license_evlog
87472 -rw- 5722300 Sep 3 2020 10:22:17 -07:00 OR227-DXA1_2_RP_0_trace_archive_0-20200903-102216.tar.gz

[p11l]

We had same error on our C9500. Our Solution is to change SFP+ Module (we used copper direct attached cable) from 3rd manufacturer to original cisco. Voilá - error no longer exists.

ios version: 17.9.4a