Solved: Cisco ISR 1111-8P reboots randomly

MarkusD · ‎03-01-2022

Hi there,

about 6 month ago I bought a Cisco C1111-8P ISR router with IOS XE Version 16.12.04.
It is working as an edge router for DSL dialup. So no really special configuration. (Ok, source based routing for direct IPv6 and tunneled HE-Uplink; both static IPs).

So far it is working fine. But last month I experienced 2 unexpected reboots of the router.

The syslog (remote logging server) stays empty. No SSH login that time or any error message.

In

bootflash:/tracelogs/ I found 3 files named shutdown_cc0.log, shutdown_fp0.log and shutdown_rp0.log

I attached them to this post.

In every file I could find a line like see

pvp pvp_0-0.17986_0.20220217042043.bin

log for how shutdown got started on this FRU

.
I could find that file gzip-compressed and decompressed it. Unfortunately it is not a text file.

Do you have idea, how I can investigate those reloads?

Is it save to attach that bin log or are there any sensitive information included?

Thanks a lot for any help.

Leo Laohoo · ‎03-02-2022

@MarkusD wrote:

Sorry, there is a lot of sensitive information in this file. I have to delete it again here.

No, it does not. It was a CORE DUMP from the QFP. Only TAC would have access to read it.

~~Need sleep.~~

~~I'll have a look at this tomorrow.~~

In regards to unable to download software because there is no maintenance contract, keep an eye out on the Cisco Security Bulletin.

In the next few weeks, there will be PSIRT/software vulnerability announcement and will affects all IOS-XE from 17.3.4 and earlier. The fix will be in 16.12.7 and 17.3.5 (not sure about the rest).

When this happens, read the Security Bulletin very carefully. Each Security Bulletin should have a section called "Customers Without Service Contracts" and it should read:

Customers Without Service Contracts

Customers who purchase directly from Cisco but do not hold a Cisco service contract and customers who make purchases through third-party vendors but are unsuccessful in obtaining fixed software through their point of sale should obtain upgrades by contacting the Cisco TAC.

Customers should have the product serial number available and be prepared to provide the URL of this advisory as evidence of entitlement to a free upgrade.

Read it. Understand it.

Hope this helps.

View solution in original post

balaji.bandi · ‎03-01-2022

Looks for me some bug, how long this version running on this Router, If you get chance upgrade to latest stable and check or contact TAC.

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

MarkusD · ‎03-01-2022

The router was shipped with this firmware.
I do not have a Smartnet Contract for this router. Therefore I can't contact TAC, I suppose.

Is there any way to display the pvp binlog in a human readable way?

marce1000 · ‎03-01-2022

>...Do you have idea, how I can investigate those reloads?

- Connect to the ISR (SSH) with https://cway.cisco.com/cli , at the top right , press or run

Crashdump Analyzer

, as stated already , consider upgrade to latest advisory release : https://software.cisco.com/download/home/286321996/type/282046477/release/Amsterdam-17.3.4a

M.

-- Each morning when I wake up and look into the mirror I always say ' Why am I so brilliant ? '
When the mirror will then always repond to me with ' The only thing that exceeds your brilliance is your beauty! '

Leo Laohoo · ‎03-02-2022

Post the complete output to the following commands:

```
sh version
```
```
dir
```
```
dir bootflash:core
```

MarkusD · ‎03-02-2022

Thanks a lot for your replies.

Here are the outputs of the mentioned commands:

sh version

Cisco IOS XE Software, Version 16.12.04
Cisco IOS Software [Gibraltar], ISR Software (ARMV8EL_LINUX_IOSD-UNIVERSALK9-M), Version 16.12.4, RELEASE SOFTWARE (fc5)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2020 by Cisco Systems, Inc.
Compiled Thu 09-Jul-20 20:16 by mcpre

Cisco IOS-XE software, Copyright (c) 2005-2020 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0. For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.

ROM: 16.12(2r)

Router uptime is 1 day, 4 hours, 38 minutes
Uptime for this control processor is 1 day, 4 hours, 40 minutes
System returned to ROM by LocalSoft at 20:58:12 CET Sun Nov 7 2021
System restarted at 07:02:28 CET Tue Mar 1 2022
System image file is "bootflash:c1100-universalk9.16.12.04.SPA.bin"
Last reload reason: LocalSoft

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

Suite License Information for Module:'esg'

--------------------------------------------------------------------------------
Suite Suite Current Type Suite Next reboot
--------------------------------------------------------------------------------
FoundationSuiteK9 None Smart License None
securityk9
appxk9

Technology Package License Information:

-----------------------------------------------------------------
Technology Technology-package Technology-package
Current Type Next reboot
------------------------------------------------------------------
appxk9 None Smart License None
uck9 None Smart License None
securityk9 None Smart License None
ipbase ipbasek9 Smart License ipbasek9

The current throughput level is unthrottled

Smart Licensing Status: UNREGISTERED/No Licenses in Use

cisco C1111-8P (1RU) processor with 1398091K/6147K bytes of memory.
Processor board ID FCZ2530R27V
2 Virtual Ethernet interfaces
10 Gigabit Ethernet interfaces
32768K bytes of non-volatile configuration memory.
4194304K bytes of physical memory.
2863103K bytes of flash memory at bootflash:.
0K bytes of WebUI ODM Files at webui:.

Configuration register is 0x2102

dir


Directory of bootflash:/

   11  drwx            16384  Jul 20 2021 08:49:31 +01:00  lost+found
138721  drwx             4096   Nov 7 2021 21:01:10 +01:00  .prst_sync
   12  -rw-               35  Aug 14 2021 11:58:04 +01:00  pnp-tech-time
   13  -rw-            79731  Aug 14 2021 11:58:13 +01:00  pnp-tech-discovery-summary
   14  -rw-        542525852  Jul 20 2021 08:52:04 +01:00  c1100-universalk9.16.12.04.SPA.bin
97921  drwx             4096   Mar 1 2022 07:01:49 +01:00  .installer
89761  drwx             4096   Mar 1 2022 06:59:51 +01:00  core
122401  drwx             4096  Jul 20 2021 08:57:12 +01:00  .rollback_timer
114241  drwx             4096  Jul 20 2021 08:57:13 +01:00  gs_script
   15  -rw-           132108   Mar 1 2022 07:02:26 +01:00  memleak.tcl
40801  drwx             4096  Jul 20 2021 08:57:23 +01:00  virtual-instance
48961  drwx             4096  Aug 14 2021 12:06:57 +01:00  .dbpersist
57121  drwx            53248   Mar 2 2022 11:32:25 +01:00  tracelogs
65281  drwx             4096  Jul 20 2021 08:57:45 +01:00  .attrib
   16  -rw-               30   Mar 1 2022 07:02:42 +01:00  throughput_monitor_params
73441  drwx             4096  Jul 20 2021 08:57:50 +01:00  onep
146881  drwx             4096   Mar 1 2022 07:02:49 +01:00  license_evlog
   17  -rw-              656  Aug 14 2021 12:35:52 +01:00  vlan.dat
   18  -rw-            11318  Aug 14 2021 21:18:38 +01:00  running_210814.txt
   21  -rw-              274  Oct 30 2021 21:11:41 +01:00  btdecode_err
   19  -rw-             8325  Aug 22 2021 14:28:06 +01:00  startup_210822.txt
   20  -rw-            12429  Aug 22 2021 15:13:03 +01:00  shaping-test_210822.txt
   22  -rw-          2672640  Oct 16 2021 12:13:52 +01:00  nbar2_http_default.tar
171361  drwx             4096  Oct 16 2021 12:13:54 +01:00  nbar2
106081  drwx             4096  Oct 16 2021 12:20:50 +01:00  guest-share
   36  -rw-             9757   Nov 9 2021 16:12:04 +01:00  startup_211109.txt

2908606464 bytes total (2112970752 bytes free)

dir bootflash:core

Directory of bootflash:/core/

97923  drwx             4096  Jul 20 2021 08:57:11 +01:00  modules
89762  -rw-                1   Mar 2 2022 11:32:56 +01:00  .callhome
89763  -rw-         15232975  Aug 14 2021 20:54:21 +01:00  Router_RP_0_qfp-ucode-tsn-le_17679_20210814-205402-CET.core.gz
89764  -rw-         19512780   Nov 1 2021 16:26:20 +01:00  Router_RP_0_qfp-ucode-tsn-le_17657_20211101-162559-CET.core.gz
89765  -rw-         19655984  Feb 17 2022 01:19:36 +01:00  Router_RP_0_qfp-ucode-tsn-le_17692_20220217-011914-CET.core.gz
89766  -rw-         18477992  Feb 17 2022 05:18:10 +01:00  Router_RP_0_qfp-ucode-tsn-le_17891_20220217-051749-CET.core.gz
89767  -rw-         16080432   Mar 1 2022 06:59:51 +01:00  Router_RP_0_qfp-ucode-tsn-le_17734_20220301-065931-CET.core.gz

2908606464 bytes total (2112905216 bytes free)

MarkusD · ‎03-02-2022

And another reload again. This time the PPPoE tunnel of the Dialer interface got reconnected first.

The following happened according to my logs:

Mar  2 10:57:22.987: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access2, changed state to down
Mar  2 10:58:18.832: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access2, changed state to up

And at 11:04 the router rebooted again.

Leo Laohoo · ‎03-02-2022

@MarkusD wrote:

Router_RP_0_qfp-ucode-tsn-le_17734_20220301-065931-CET.core.gz

Upload this file.

Alternatively, raise a TAC Case and get them to verify if this is CSCvy57684.

MarkusD · ‎03-02-2022

~~Requested file is attached.~~

~~I had to zip it again, since gz extension is not allowed here.~~

Sorry, there is a lot of sensitive information in this file. I have to delete it again here.

Nevertheless, thanks a lot for your help.

Leo Laohoo · ‎03-02-2022