Re: 10G Interface Fault on ASR1001-X after recent upgrade from 3.16 to

CarsonDavis56998 · ‎11-20-2022

I recently upgraded a pair of ASR-1001-X's that were in our DC Storage for a while to repurpose them elsewhere. I brought them from running code 3.16 to 17.3. As a prerequesite, based on the upgrade matrix I also needed to upgrade the rommon as well, so I upgraded this to (173-1r). After upgrading both rommon and the ios code, I am beginning to see issues with using 10G Interfaces on the router. When enabling my 10g interfaces, I see the following msg:

*Nov 20 15:38:12.413: %LINK-5-CHANGED: Interface TenGigabitEthernet0/0/0, changed state to administratively down
*Nov 20 15:38:15.254: %IOSXE_SPA-6-UPDOWN: Interface TenGigabitEthernet0/0/0, link down due to local fault
*Nov 20 15:38:15.314: %IOSXE_SPA-6-UPDOWN: Interface TenGigabitEthernet0/0/0, link down due to local fault

I am currently running eval for my 10g Licenses on the box until I receive my permanent licenses from my var, and still have 8 wks left so I assume it's not a licensing issue. I have also read through Cisco Bug Documentation and saw that this was a "bug" that existed on later 15.X code, however, I am running 17.3 code now, so I do not think this is the root of the issue either. I reached out to an engineer that started before me and had used these routers in the past and he states that the 10g interfaces worked fine before me so, I can assume that at one point they were working just fine before my code upgrade. My 1gig interfaces work just fine with no issues, this weird issue only affects my 10g interfaces at the moment. Any idea what may have gone wrong here? Thanks for any comments.

show dir flash:

ALPH-ASR-CE1-IDF2#dir flash:
Directory of flash:/

11 drwx 16384 Aug 25 2016 18:24:34 +00:00 lost+found
196609 drwx 4096 Nov 20 2022 15:35:37 +00:00 .prst_sync
475137 drwx 4096 Nov 15 2022 14:09:45 +00:00 pnp-info
524289 drwx 4096 Nov 15 2022 14:09:54 +00:00 guest-share
622593 drwx 4096 Nov 15 2022 14:10:14 +00:00 iox_data_share
147457 drwx 4096 Nov 15 2022 14:14:44 +00:00 .installer
81921 drwx 4096 Aug 25 2016 18:32:43 +00:00 core
671745 drwx 4096 Aug 25 2016 18:31:49 +00:00 .rollback_timer
13 -rw- 0 Aug 25 2016 18:31:55 +00:00 tracelogs.695
294913 drwx 36864 Nov 20 2022 15:41:13 +00:00 tracelogs
311297 drwx 4096 Aug 25 2016 18:31:57 +00:00 virtual-instance
14 -rw- 30 Nov 20 2022 15:35:41 +00:00 throughput_monitor_params
557057 drwx 4096 Nov 20 2022 15:36:10 +00:00 syslog
15 -rw- 526240224 Sep 6 2017 08:24:29 +00:00 asr1001x-universalk9.03.16.06.S.155-3.S6-ext.SPA.bin
16 -rw- 1802 Sep 6 2017 08:45:24 +00:00 octeon_pci_console_dev0_20170906_084522.log
17 -rw- 1802 Sep 6 2017 08:52:07 +00:00 octeon_pci_console_dev0_20170906_085205.log
18 -rw- 34 Sep 6 2017 08:54:14 +00:00 pnp-tech-time
19 -rw- 13793 Sep 6 2017 08:54:20 +00:00 pnp-tech-discovery-summary
20 -rw- 1444 Jul 11 2021 13:11:41 +00:00 octeon_pci_console_dev0_20210711_131139.log
21 -rw- 1444 Jul 12 2021 11:32:54 +00:00 octeon_pci_console_dev0_20210712_113252.log
22 -rw- 1444 May 4 2022 19:30:50 +00:00 octeon_pci_console_dev0_20220504_193048.log
163841 drwx 4096 May 15 2022 18:25:05 +00:00 221subs
23 -rw- 41059376 May 16 2022 06:46:32 +00:00 asr1000-rommon.173-1r.SPA.pkg
24 -rw- 905521973 May 16 2022 06:58:20 +00:00 asr1001x-universalk9.17.03.05.SPA.bin
25 -rw- 1802 May 16 2022 07:26:45 +00:00 octeon_pci_console_dev0_20220516_072643.log
26 -rw- 1000 Nov 15 2022 14:08:41 +00:00 mode_event_log
704513 drwx 4096 May 16 2022 07:35:00 +00:00 bootlog_history
327681 drwx 4096 May 16 2022 07:35:08 +00:00 gs_script
425985 drwx 4096 May 16 2022 07:35:36 +00:00 .USWAP
491521 drwx 4096 May 16 2022 07:35:20 +00:00 .CRFT
27 -rw- 134458 Nov 15 2022 14:09:10 +00:00 memleak.tcl
28 -rw- 20109 May 16 2022 07:35:21 +00:00 ios_core.p7b
29 -rw- 1923 May 16 2022 07:35:21 +00:00 trustidrootx3_ca_092024.ca
573441 drwx 4096 Nov 15 2022 14:04:30 +00:00 .dbpersist
770049 drwx 4096 Nov 15 2022 14:09:00 +00:00 .inv
262145 drwx 4096 May 16 2022 07:36:16 +00:00 onep
753665 drwx 4096 Nov 15 2022 14:13:42 +00:00 license_evlog
344065 drwx 4096 Nov 15 2022 14:10:14 +00:00 iox_host_data_share
65537 drwx 4096 Nov 15 2022 14:13:53 +00:00 iox
540673 drwx 4096 Nov 15 2022 14:13:48 +00:00 pnp-tech
12 -rw- 1444 Nov 15 2022 14:19:08 +00:00 octeon_pci_console_dev0_20221115_141906.log
30 -rw- 1444 Nov 15 2022 14:29:02 +00:00 octeon_pci_console_dev0_20221115_142900.log
31 -rw- 1444 Nov 18 2022 06:34:52 +00:00 octeon_pci_console_dev0_20221118_063450.log
32 -rw- 1444 Nov 18 2022 07:22:11 +00:00 octeon_pci_console_dev0_20221118_072209.log
33 -rw- 1444 Nov 18 2022 12:39:07 +00:00 octeon_pci_console_dev0_20221118_123905.log
34 -rw- 1444 Nov 18 2022 13:09:14 +00:00 octeon_pci_console_dev0_20221118_130912.log
35 -rw- 1444 Nov 20 2022 15:35:41 +00:00 octeon_pci_console_dev0_20221120_153539.log

6737653760 bytes total (4904751104 bytes free)
ALPH-ASR-CE1-IDF2#show run | inc boot
boot-start-marker
boot system flash:asr1001x-universalk9.17.03.05.SPA.bin
boot-end-marker
no ip bootp server
ip dhcp bootp ignore
license boot suite FoundationSuiteK9
license boot level advipservices

show logging:

Log Buffer (4096 bytes):

*Nov 20 15:38:12.413: %LINK-5-CHANGED: Interface TenGigabitEthernet0/0/0, changed state to administratively down
*Nov 20 15:38:15.254: %IOSXE_SPA-6-UPDOWN: Interface TenGigabitEthernet0/0/0, link down due to local fault
*Nov 20 15:38:15.314: %IOSXE_SPA-6-UPDOWN: Interface TenGigabitEthernet0/0/0, link down due to local fault
*Nov 20 15:38:15.710: %SYS-5-CONFIG_I: Configured from console by first.last on vty0 (10.10.240.21)
*Nov 20 15:38:17.218: %LINK-3-UPDOWN: Interface TenGigabitEthernet0/0/0, changed state to down
*Nov 20 15:38:17.219: %LINK-3-UPDOWN: SIP0/0: Interface TenGigabitEthernet0/0/0, changed state to down
ALPH-ASR-CE1-IDF2#

show license

ALPH-ASR-CE1-IDF2#show license | begin interface

Index 33 Feature: interface_10g

Period left: 8 weeks 1 day

Period Used: 2 days 8 hours

License Type: EvalRightToUse

License State: Active, In Use

License Count: 2/0 (In-use/Violation)

License Priority: Low

Index 34 Feature: FoundationSuiteK9

Period left: 6 weeks 1 day

Period Used: 2 weeks 2 days

License Type: EvalRightToUse

License State: Active, In Use

License Count: Non-Counted

License Priority: Low

Index 35 Feature: AdvUCSuiteK9

Period left: Not Activated

Period Used: 0 minute 0 second

License Type: EvalRightToUse

License State: Active, Not in Use, EULA not accepted

License Count: Non-Counted

License Priority: None

show platform

ALPH-ASR-CE1-IDF2#show plat

ALPH-ASR-CE1-IDF2#show platform

Chassis type: ASR1001-X

Slot Type State Insert time (ago)

--------- ------------------- --------------------- -----------------

0 ASR1001-X ok 00:51:50

0/0 BUILT-IN-2T+6X1GE ok 00:51:01

R0 ASR1001-X ok, active 00:51:50

F0 ASR1001-X ok, active 00:51:50

P0 ASR1001-X-PWR-AC ok 00:51:35

P1 ASR1001-X-PWR-AC ok 00:51:33

P2 ASR1001-X-FANTRAY ok 00:51:37

Slot CPLD Version Firmware Version

--------- ------------------- ---------------------------------------

0 14041015 17.3(1r)

R0 14041015 17.3(1r)

F0 14041015 17.3(1r)

show inventory

ALPH-ASR-CE1-IDF2#show inventory

NAME: "Chassis", DESCR: "Cisco ASR1001-X Chassis"

PID: ASR1001-X , VID: V02, SN: XXXXXXXXXXX

NAME: "Power Supply Module 0", DESCR: "Cisco ASR1001-X AC Power Supply"

PID: ASR1001-X-PWR-AC , VID: V01, SN: XXXXXXXXXXX

NAME: "Power Supply Module 1", DESCR: "Cisco ASR1001-X AC Power Supply"

PID: ASR1001-X-PWR-AC , VID: V01, SN: XXXXXXXXXXX

NAME: "Fan Tray", DESCR: "Cisco ASR1001-X Fan Tray"

PID: ASR1001-X-FANTRAY , VID: , SN:

NAME: "module 0", DESCR: "Cisco ASR1001-X SPA Interface Processor"

PID: ASR1001-X , VID: , SN:

NAME: "SPA subslot 0/0", DESCR: "Built-in GE SPA with 2 port 10G and 6 port 1G Interfaces"

PID: BUILT-IN-2T+6X1GE , VID: , SN:

NAME: "subslot 0/0 transceiver 0", DESCR: "SFP+ 10GBASE-LR"

PID: SFP-10G-LR-S , VID: A , SN: XXXXXXXXXXX

NAME: "subslot 0/0 transceiver 2", DESCR: "GE SX"

PID: GLC-SX-MMD , VID: 01 , SN: XXXXXXXXXXX

NAME: "module R0", DESCR: "Cisco ASR1001-X Route Processor"

PID: ASR1001-X , VID: V02, SN: XXXXXXXXXXX

NAME: "module F0", DESCR: "Cisco ASR1001-X Embedded Services Processor"

PID: ASR1001-X , VID: , SN:

balaji.bandi · ‎11-20-2022

is this worked before you upgrade the code ? how about shut and no shut - is this fix the issue?

can you post the below information :

show Interface TenGigabitEthernet0/0/0

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

CarsonDavis56998 · ‎11-20-2022

Yes, engineer who worked on this router before me years ago stated that it worked w/o any issues before my upgrade. If I preform a "shut / no shut" on the 2x ten gig interfaces the same issue appears. Where it states there is a fault and the interface shows in a Down/Down State.

ALPH-ASR-CE1-IDF2#show int ten 0/0/0
TenGigabitEthernet0/0/0 is down, line protocol is down
Hardware is BUILT-IN-2T+6X1GE, address is 0081.c4c9.ac00 (bia 0081.c4c9.ac00)
Description: AT&T-LINK-TO-QTS-CORE1-NUA#
Internet address is 10.251.0.137/31
MTU 1500 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not supported
Full Duplex, 10000Mbps, link type is force-up, media type is SFP-LR
output flow-control is on, input flow-control is on
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
0 packets input, 0 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 packets output, 0 bytes, 0 underruns
0 output errors, 0 collisions, 8 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

Leo Laohoo · ‎11-20-2022

What was the reason for the need to upgrade to 17.X.X?

CarsonDavis56998 · ‎11-20-2022

It was originally running on old code (3.X) and before I removed it from storage to place back into production I wanted to upgrade the code to gold star,

Leo Laohoo · ‎11-20-2022

Unless there is an operational need, I would strongly recommend downgrading back to the original working version.

CarsonDavis56998 · ‎11-20-2022

Understood. I will downgrade back to 3.16 to see if that fixes my issue. The only thing is that TAC typically will not work on a case issue in an appliance unless you are running supported code. Thus the reason why I wanted to go ahead and upgrade the box prior to adding it into production, in case support from TAC is needed in the future until we can have the router refreshed. I will let you know how it works out after downgrading back to 3.16.

CarsonDavis56998 · ‎11-20-2022

I just downgraded the running code to 3.16, however, I am still receiving the same errors, unfortunately.

ALPH-ASR-CE1-IDF2(config-if-range)#do show version
Cisco IOS XE Software, Version 03.16.06.S - Extended Support Release
Cisco IOS Software, ASR1000 Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 15.5(3)S6, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2017 by Cisco Systems, Inc.
Compiled Mon 24-Jul-17 20:24 by mcpre

Cisco IOS-XE software, Copyright (c) 2005-2017 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0. For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.

ROM: IOS-XE ROMMON

ALPH-ASR-CE1-IDF2 uptime is 3 minutes
Uptime for this control processor is 4 minutes
System returned to ROM by reload at 17:25:37 UTC Sun Nov 20 2022
System image file is "bootflash:/asr1001x-universalk9.03.16.06.S.155-3.S6-ext.SPA.bin"
Last reload reason: Reload Command

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

License Level: adventerprise
License Type: EvalRightToUse
Next reload license Level: adventerprise

ALPH-ASR-CE1-IDF2(config-if-range)#do show logging
Syslog logging: enabled (0 messages dropped, 6 messages rate-limited, 0 flushes, 0 overruns, xml disabled, filtering disabled)

No Active Message Discriminator.

No Inactive Message Discriminator.

Console logging: level debugging, 54 messages logged, xml disabled,
filtering disabled
Monitor logging: level debugging, 0 messages logged, xml disabled,
filtering disabled
Buffer logging: level informational, 56 messages logged, xml disabled,
filtering disabled
Exception Logging: size (4096 bytes)
Count and timestamp logging messages: disabled
Persistent logging: enabled, url bootflash:/syslog, disk space 673765376 bytes, file size 262144 bytes, batch size 4096 bytes

No active filter modules.

Trap logging: level informational, 60 message lines logged
Logging Source-Interface: VRF Name:

Log Buffer (4096 bytes):

*Nov 20 17:31:57.473: %LINK-5-CHANGED: Interface TenGigabitEthernet0/0/0, changed state to administratively down
*Nov 20 17:31:57.530: %LINK-5-CHANGED: Interface TenGigabitEthernet0/0/1, changed state to administratively down
*Nov 20 17:31:58.006: %IOSXE_SPA-6-UPDOWN: Interface TenGigabitEthernet0/0/0, link down due to local fault
*Nov 20 17:31:58.058: %IOSXE_SPA-6-UPDOWN: Interface TenGigabitEthernet0/0/1, link down due to local fault
*Nov 20 17:31:58.095: %IOSXE_SPA-6-UPDOWN: Interface TenGigabitEthernet0/0/0, link down due to local fault
*Nov 20 17:31:59.960: %LINK-3-UPDOWN: Interface TenGigabitEthernet0/0/0, changed state to down
*Nov 20 17:31:59.970: %LINK-3-UPDOWN: Interface TenGigabitEthernet0/0/1, changed state to down
*Nov 20 17:31:59.973: %LINK-3-UPDOWN: SIP0/0: Interface TenGigabitEthernet0/0/0, changed state to down
ALPH-ASR-CE1-IDF2(config-if-range)#

Leo Laohoo · ‎11-20-2022

@CarsonDavis56998 wrote:
*Nov 20 17:31:58.006: %IOSXE_SPA-6-UPDOWN: Interface TenGigabitEthernet0/0/0, link down due to local fault
*Nov 20 17:31:58.058: %IOSXE_SPA-6-UPDOWN: Interface TenGigabitEthernet0/0/1, link down due to local fault
*Nov 20 17:31:58.095: %IOSXE_SPA-6-UPDOWN: Interface TenGigabitEthernet0/0/0, link down due to local fault

"local fault" to all two 10 Gbps ports?

Please post the complete output to the command "sh license feature".

CarsonDavis56998 · ‎11-20-2022

Correct, only the 2x 10G Ports are reporting this issue. 1Gig is just fine.

ALPH-ASR-CE1-IDF2#show license feature
Feature name Enforcement Evaluation Subscription Enabled RightToUse
adventerprise yes yes no yes yes
advipservices yes yes no no yes
ipbase no no no no no
avc yes yes no no yes
broadband no no no no no
broadband_4k no no no no no
cube_250 no no no no no
cube_250_red no no no no no
cube_ent_100 no no no no no
cube_ent_100_red no no no no no
cube_lab no no no no no
cube_video_b2btp no no no no no
cube_video_b2btp_red no no no no no
firewall no no no no no
fpi no no no no no
fwnat_red yes yes no no yes
gtp_addon_aic no no no no no
internal_service yes no no no no
ipsec yes yes no no yes
lawful_intr yes yes no no yes
lisp yes yes no no yes
nat64_stateful_2m no no no no no
otv yes yes no no yes
sgt_fw no no no no no
sw_redundancy yes yes no no yes
throughput_5g yes yes no no yes
throughput_5g_to_10g yes no no no no
throughput_5g_to_20g yes no no no no
throughput_10g yes yes no no yes
throughput_10g_to_20g yes no no no no
throughput_20g yes yes no yes yes
vpls yes yes no no yes
interface_10g yes yes no yes yes
FoundationSuiteK9 yes yes no yes yes
AdvUCSuiteK9 yes yes no no yes

Leo Laohoo · ‎11-20-2022

@CarsonDavis56998 wrote:

interface_10g yes yes no yes yes

That tells me 10 Gbps is enabled.

@CarsonDavis56998 wrote:

throughput_10g yes yes no no yes

I think this is what is also required.

CarsonDavis56998 · ‎11-20-2022

I see. I had thought that the 20g throughput license would suffice, but perhaps not?

throughput_20g yes yes no yes yes

Leo Laohoo · ‎11-20-2022

Ooops. I did not notice the 20G license.

If there are SFP+ installed, remove it, wait for about 5 seconds and insert them back in.

10G Interface Fault on ASR1001-X after recent upgrade from 3.16 to17.3