已解决: 【求助】关于ISR4451-X/K9 (2RU)使用不定时重启问题疑问

zhihonghan42550 · ‎12-22-2019

各路大神好，最近遇到一个问题，超出自己知识范围，特来求助。
Parameter map cws-tunnel global not configuredcisco ISR4451-X/K9 (2RU) processor with 1647778K/6147K bytes of memory.
Processor board ID FT***
4 Gigabit Ethernet interfaces
32768K bytes of non-volatile configuration memory.
4194304K bytes of physical memory.
7341807K bytes of flash memory at bootflash:.
这么一台设备，从11月开始，一接办公网络，就不定时重启，查看深信服，大概带机量在400-500之间，
但日常查看CPU 内存也没啥异常，但就是不定时重启。
这台设备19年已经修两次了，联系供应商，也说没遇到类似情况，现在还拖着
然后一直用着备用路由，备用路由CPU平均值80%左右，好怕出问题。:'(
请各路大神协助解答一下？小弟不胜感激，仅有的22金币虽说不多，但也表达个心意，如数奉上
大概拓扑如图：

ASR4451#sh log
Syslog logging: enabled (0 messages dropped, 10 messages rate-limited, 0 flushes, 0 overruns, xml disabled, filtering disabled)
No Active Message Discriminator.
No Inactive Message Discriminator.
    Console logging: level warnings, 14 messages logged, xml disabled,
                     filtering disabled
    Monitor logging: level debugging, 0 messages logged, xml disabled,
                     filtering disabled
    Buffer logging:  level warnings, 14 messages logged, xml disabled,
                    filtering disabled
    Exception Logging: size (4096 bytes)
    Count and timestamp logging messages: disabled
    Persistent logging: disabled
No active filter modules.
    Trap logging: disabled
          
Log Buffer (4096 bytes):
*Nov 11 08:14:30.421: %LINK-3-UPDOWN: Interface Lsmpi0, changed state to up
*Nov 11 08:14:30.421: %LINK-3-UPDOWN: Interface EOBC0, changed state to up
*Nov 11 08:14:30.421: %LINK-3-UPDOWN: Interface GigabitEthernet0, changed state to down
*Nov 11 08:14:30.425: %LINK-3-UPDOWN: Interface LIIN0, changed state to up
*Nov 11 16:14:35.296 Beijing: %SPA-3-ENVMON_NOT_MONITORED:iomd:  Environmental monitoring is not enabled for ISR4451-X-4x1GE[0/0]
*Nov 11 16:14:35.430 Beijing: %LINK-3-UPDOWN: Interface Virtual-Access1, changed state to up
*Nov 11 16:14:41.870 Beijing: %LINK-3-UPDOWN: Interface GigabitEthernet0/0/0, changed state to down
*Nov 11 16:14:41.877 Beijing: %LINK-3-UPDOWN: Interface GigabitEthernet0/0/1, changed state to down
*Nov 11 16:14:41.880 Beijing: %LINK-3-UPDOWN: Interface GigabitEthernet0/0/2, changed state to down
*Nov 11 16:14:41.880 Beijing: %LINK-3-UPDOWN: Interface GigabitEthernet0/0/3, changed state to down
*Nov 11 16:14:55.393 Beijing: %LINK-3-UPDOWN: Interface GigabitEthernet0/0/1, changed state to up
*Nov 11 16:14:57.394 Beijing: %LINK-3-UPDOWN: Interface GigabitEthernet0/0/3, changed state to up
*Nov 11 16:15:05.821 Beijing: %LINK-3-UPDOWN: Interface Virtual-Access2, changed state to up
*Nov 11 16:15:05.825 Beijing: %LINK-3-UPDOWN: Interface Virtual-Access3, changed state to up

ASR4451#sh run        
Building configuration...
Current configuration : 5215 bytes
!
version 15.5
service timestamps debug datetime msec localtime show-timezone
service timestamps log datetime msec localtime show-timezone
service password-encryption
no platform punt-keepalive disable-kernel-core
!
hostname ASR4451
!
boot-start-marker
boot system flash:isr4400-universalk9.03.16.08.S.155-3.S8-ext.SPA.bin
boot system flash isr4400-universalk9.0
boot system flash bootflash:/isr4400-universalk9.03.16.08.S.155-3.S8-ext.SPA.bin
boot-end-marker
!
!
vrf definition Mgmt-intf
 !
 address-family ipv4
 exit-address-family
 !
 address-family ipv6
 exit-address-family
!
logging buffered warnings
logging console warnings
!
no aaa new-model
clock timezone Beijing 8 0
!
!
!
!
!
login block-for 600 attempts 4 within 60
!
!
!
!         
!
!
!
subscriber templating
!
multilink bundle-name authenticated
!
license udi pid ISR4451-X/K9 sn 
!
spanning-tree extend system-id
!
!
redundancy
 mode none
!
no cdp run
!
track 10 interface GigabitEthernet0/0/2 line-protocol
!
track 20 interface GigabitEthernet0/0/1 line-protocol
!
track 30 interface GigabitEthernet0/0/3 line-protocol
!
interface Loopback0
 ip address 192.168.*.1 255.255.255.0
!
interface Loopback1
 no ip address
!
interface GigabitEthernet0/0/0
 ip address 192.168.*.1 255.255.255.0
 negotiation auto
!
interface GigabitEthernet0/0/1
 no ip address
 ip nat outside
 ip access-group port_445_5938 in
 negotiation auto
 pppoe enable group 1
 pppoe-client dial-pool-number 1
 ip virtual-reassembly
!
interface GigabitEthernet0/0/2
 no ip address
 ip nat outside
 ip access-group port_445_5938 in
 negotiation auto
 pppoe enable group 2
 pppoe-client dial-pool-number 2
 ip virtual-reassembly
!
interface GigabitEthernet0/0/3
 no ip address
 ip nat outside
 ip access-group port_445_5938 in
 negotiation auto
 pppoe enable group 3
 pppoe-client dial-pool-number 3
 ip virtual-reassembly
!
interface GigabitEthernet0
 vrf forwarding Mgmt-intf
 no ip address
 shutdown
 negotiation auto
!
interface Dialer0
 description adsl-dianxin200
 ip address negotiated
 ip nat outside
 encapsulation ppp
 ip tcp adjust-mss 1440
 dialer pool 1
 dialer idle-timeout 0
 dialer-group 1
 ppp authentication pap callin
 ppp pap sent-username *
 ppp ipcp dns request
!
interface Dialer1
 description adsl-dianxin500
 ip address negotiated
 ip nat outside
 encapsulation ppp
 ip tcp adjust-mss 1440
 dialer pool 2
 dialer idle-timeout 0
 dialer-group 2
 ppp authentication pap callin
 ppp pap sent-username *
!
interface Dialer2
 description adsl-liantong200
 ip address negotiated
 ip nat outside
 encapsulation ppp
 ip tcp adjust-mss 1440
 dialer pool 3
 dialer idle-timeout 0
 dialer-group 3
 ppp authentication pap callin
 ppp pap sent-username *
 ppp ipcp dns request
 ip virtual-reassembly max-reassemblies 1024
!
ip nat inside source route-map to_D0 interface Dialer0 overload
ip nat inside source route-map to_D1 interface Dialer1 overload
ip nat inside source route-map to_D2 interface Dialer2 overload
ip forward-protocol nd
no ip http server
no ip http secure-server
ip tftp source-interface GigabitEthernet0
ip route 0.0.0.0 0.0.0.0 Dialer1 track 10
ip route 0.0.0.0 0.0.0.0 Dialer0 10
ip route 0.0.0.0 0.0.0.0 Dialer2 20
ip route 192.168.0.0 255.255.0.0 192.168.20.2
ip ssh authentication-retries 5
ip ssh version 2
!
!
ip access-list extended NAT_D0
 permit ip 192.168.0.0 0.0.255.255 any
ip access-list extended NAT_D1
 permit ip 192.168.0.0 0.0.255.255 any
ip access-list extended NAT_D2
 permit ip 192.168.0.0 0.0.255.255 any
ip access-list extended out_D0
 permit ip 192.168.100.0 0.0.0.255 any
 permit ip 192.168.*.0 0.0.0.255 any
 permit ip 192.168.*.0 0.0.0.255 any
 permit ip 192.168.*.0 0.0.0.255 any
 permit ip 192.168.*.0 0.0.0.255 any
ip access-list extended out_D2
 permit ip 192.168.*.0 0.0.0.255 any
 permit ip 192.168.*.0 0.0.0.255 any
ip access-list extended port_445_5938
 deny   tcp any any eq 445
 deny   tcp any any eq 5938
 permit ip any any
!
no logging trap
logging source-interface Loopback0
logging host 192.168*
access-list 90 permit 192.168*
access-list 90 permit 192.168*
!
route-map to_D0 permit 10
 match ip address NAT_D0
 match interface Dialer0
!
route-map to_D1 permit 10
 match ip address NAT_D1
 match interface Dialer1
!
route-map to_D2 permit 10
 match ip address NAT_D2
 match interface Dialer2
!
route-map out_D0 permit 10
 match ip address out_D0
 match track  20
 set default interface Dialer0
!
route-map out_D0 permit 20
 match ip address out_D2
 match track  30
 set default interface Dialer2
!
!
control-plane
!
!
line con 0
 exec-timeout 3 0
 logging synchronous
 stopbits 1
line aux 0
 stopbits 1
line vty 0 4
 exec-timeout 20 0
 privilege level 15
 login local
 transport input ssh
line vty 5 15
 login local
 transport input ssh
!
end

ASR4451# sh processes cpu
CPU utilization for five seconds: 0%/0%; one minute: 0%; five minutes: 0%
 PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
 PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process 
   1           3          33         90  0.00%  0.00%  0.00%   0 Chunk Manager    
   2          18          81        222  0.00%  0.00%  0.00%   0 Load Meter       
   3           0           2          0  0.00%  0.00%  0.00%   0 SSH Event handle 
   4           0           1          0  0.00%  0.00%  0.00%   0 Retransmission o 
   5           0           1          0  0.00%  0.00%  0.00%   0 IPC ISSU Dispatc 
   6           5          12        416  0.00%  0.00%  0.00%   0 RF Slave Main Th 
   7           0           1          0  0.00%  0.00%  0.00%   0 EDDRI_MAIN       
   8           0           1          0  0.00%  0.00%  0.00%   0 RO Notify Timers 
   9          87          54       1611  0.00%  0.01%  0.00%   0 Check heaps      
  10           0           9          0  0.00%  0.00%  0.00%   0 Pool Manager     
  11           0           1          0  0.00%  0.00%  0.00%   0 DiscardQ Backgro 
 PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
  12           0           2          0  0.00%  0.00%  0.00%   0 Timers           
  13           0          20          0  0.00%  0.00%  0.00%   0 WATCH_AFS        
  14           0           1          0  0.00%  0.00%  0.00%   0 MEMLEAK PROCESS  
  15           2           5        400  0.00%  0.00%  0.00%   0 ARP Input        
  16           8         440         18  0.00%  0.00%  0.00%   0 ARP Background   
  17           0           2          0  0.00%  0.00%  0.00%   0 ATM Idle Timer   
  18           0           1          0  0.00%  0.00%  0.00%   0 ATM ASYNC PROC   
  19           0           1          0  0.00%  0.00%  0.00%   0 CEF MIB API      
  20           0           1          0  0.00%  0.00%  0.00%   0 AAA_SERVER_DEADT 
  21           0           1          0  0.00%  0.00%  0.00%   0 Policy Manager   
  22           1          28         35  0.00%  0.00%  0.00%   0 DDR Timers

ASR4451#sh processes memory 
Processor Pool Total: 1687325248 Used:  284327624 Free: 1402997624
 lsmpi_io Pool Total:    6295128 Used:    6294296 Free:        832
 PID TTY  Allocated      Freed    Holding    Getbufs    Retbufs Process
   0   0  423479288  168386280  237274936        686        799 *Init*          
   0   0        360     257952        360          0          0 *Sched*         
   0   0     844776     337976     446416          1          1 *Dead*          
   0   0          0          0     395600          0          0 *MallocLite*    
   1   0    2089280          0    2119144          0          0 Chunk Manager   
   2   0        448        448      17864          0          0 Load Meter      
   3   0       8696          0      38464          2          2 SSH Event handle
   4   0          0          0      29864          0          0 Retransmission o
   5   0          0          0      29864          0          0 IPC ISSU Dispatc
   6   0     528368      22648     463560          0          0 RF Slave Main Th
   7   0      65632          0     167496          0          0 EDDRI_MAIN      
   8   0          0          0      29864          0          0 RO Notify Timers
   9   0       6240        448      35656          0          0 Check heaps     
  10   0     562688     548864     112296         51          0 Pool Manager    
  11   0          0          0      29864          0          0 DiscardQ Backgro
  12   0        448        448      29864          0          0 Timers          
  13   0          0          0      17864          0          0 WATCH_AFS       
  14   0          0          0      29864          0          0 MEMLEAK PROCESS 
  15   0          0          0      29864          9          9 ARP Input

ASR4451#sh log uptime 
 Slot              Reset reason  Power On                  
---------------------------------------------------------                     
    0      reset local software  07/19/19 11:36:52         
    0      reset local software  07/19/19 12:04:47         
    0      reset local software  10/23/19 09:39:28         
    0      reset local software  10/24/19 09:29:06         
    0      reset local software  11/02/19 17:59:08         
    0      reset local software  11/02/19 19:01:01         
    0      reset local software  11/03/19 04:08:53         
    0      reset local software  11/03/19 11:05:09         
    0      reset local software  11/03/19 16:23:54         
    0      reset local software  11/04/19 02:57:56         
    0      reset local software  11/04/19 10:38:26         
    0      reset local software  11/04/19 14:11:46         
    0      reset local software  11/05/19 12:15:55         
    0      reset local software  11/05/19 12:24:53         
    0      reset local software  11/05/19 12:31:36         
    0      reset local software  11/05/19 14:12:36         
    0      reset local software  11/05/19 14:36:20         
    0      reset local software  11/05/19 14:54:20         
    0      reset local software  11/09/19 08:28:54         
    0      reset local software  11/11/19 09:24:28         
    0      reset local software  11/11/19 13:19:15         
    0      reset local software  11/11/19 16:13:50         
    0      reset local software  11/12/19 09:21:09         
    0      reset local software  11/14/19 18:18:27         
    0      reset local software  11/29/19 10:08:06         
    0      reset local software  01/01/10 08:02:34         
    0      reset local software  01/01/10 08:15:53         
    0      reset local software  01/01/10 08:45:06         
    0      reset local software  01/01/10 08:58:54         
    0            reset power on  01/04/10 07:23:51         
    0            reset power on  01/06/10 04:13:23         
    0            reset power on  01/06/10 06:42:11         
    0            reset power on  01/11/10 07:56:48         
    0      reset local software  12/19/19 13:46:54         
    0      reset local software  12/19/19 15:18:48         
    0      reset local software  12/19/19 15:39:55         
    0      reset local software  12/19/19 16:01:15         
    0      reset local software  12/19/19 17:13:57         
    0      reset local software  12/19/19 19:13:16         
    0      reset local software  12/20/19 08:44:12         
    0      reset local software  12/20/19 09:57:44         
    0      reset local software  12/20/19 10:42:10

YilinChen · ‎12-22-2019

1、show ver 信息是什么？
2、dir 检查 flash目录，看是不是有CoreDump 文件存在

在原帖中查看解决方案

YilinChen · ‎12-22-2019

1、show ver 信息是什么？
2、dir 检查 flash目录，看是不是有CoreDump 文件存在

YilinChen · ‎12-22-2019

另外，为什么一定要把路由器顶在最外面，用于连接互联网呢？
防火墙直连不行么？
多线路负载均衡可以考虑直接上个负截均衡设备

zhihonghan42550 · ‎12-22-2019

YilinChen 发表于 2019-12-23 13:23
1、show ver 信息是什么？
2、dir 检查 flash目录，看是不是有CoreDump 文件存在

感谢大佬提醒，请帮忙查看，好像是有生成dump
ASR4451#sh ver
Cisco IOS XE Software, Version 03.16.08.S - Extended Support Release
Cisco IOS Software, ISR Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 15.5(3)S8, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2018 by Cisco Systems, Inc.
Compiled Wed 08-Aug-18 10:48 by mcpre
Cisco IOS-XE software, Copyright (c) 2005-2018 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0. For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.
ROM: IOS-XE ROMMON
ASR4451 uptime is 3 days, 2 hours, 51 minutes
Uptime for this control processor is 3 days, 2 hours, 52 minutes
System returned to ROM by reload
System image file is "bootflash:/isr4400-universalk9.03.16.08.S.155-3.S8-ext.SPA.bin"
Last reload reason: Critical software exception, check bootflash:crashinfo_RP_00_00_20191220-103738-Beijing
This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.
A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html
If you require further assistance please contact us by sending email to
export@cisco.com.

Suite License Information for Module:'esg'
--------------------------------------------------------------------------------
Suite Suite Current Type Suite Next reboot
--------------------------------------------------------------------------------
FoundationSuiteK9 None None None
securityk9
appxk9
AdvUCSuiteK9 None None None
uck9
cme-srst
cube
Technology Package License Information:
-----------------------------------------------------------------
Technology Technology-package Technology-package
Current Type Next reboot
------------------------------------------------------------------
appxk9 None None None
uck9 None None None
securityk9 securityk9 Permanent securityk9
ipbase ipbasek9 Permanent ipbasek9
cisco ISR4451-X/K9 (2RU) processor with 1647778K/6147K bytes of memory.
Processor board ID FT**
4 Gigabit Ethernet interfaces
32768K bytes of non-volatile configuration memory.
4194304K bytes of physical memory.
7341807K bytes of flash memory at bootflash:.
Configuration register is 0x2102
ASR4451#dir fl
ASR4451#dir flash:
Directory of bootflash:/
574561 drwx 4096 Apr 22 2019 15:50:36 +08:00 .rollback_timer
771553 drwx 4096 Dec 20 2019 10:38:54 +08:00 core
12 -rw- 491568196 Nov 26 2018 14:46:58 +08:00 isr4400-universalk9.03.16.05.S.155-3.S5-ext.SPA.bin
853633 drwx 4096 Dec 20 2019 10:42:47 +08:00 .prst_sync
11 -rw- 0 Apr 22 2019 15:53:03 +08:00 tracelogs.547
804385 drwx 65536 Dec 21 2019 23:11:48 +08:00 tracelogs
426817 drwx 4096 Apr 22 2019 15:53:04 +08:00 .installer
131329 drwx 4096 Apr 22 2019 15:53:05 +08:00 virtual-instance
13 -rw- 30 Dec 20 2019 10:42:53 +08:00 throughput_monitor_params
14 drwx 4096 Apr 25 2019 10:58:46 +08:00 lost+found
15 -rw- 494675108 Nov 5 2019 12:09:20 +08:00 isr4400-universalk9.03.16.08.S.155-3.S8-ext.SPA.bin
16 -rw- 241859 Nov 11 2019 16:09:38 +08:00 crashinfo_RP_00_00_20191111-160858-Beijing
17 -rw- 35 Dec 19 2019 19:15:41 +08:00 pnp-tech-time
18 -rw- 17613 Dec 19 2019 19:15:49 +08:00 pnp-tech-discovery-summary
19 -rw- 247943 Dec 20 2019 09:53:47 +08:00 crashinfo_RP_00_00_20191220-095308-Beijing
20 -rw- 241497 Dec 20 2019 10:38:17 +08:00 crashinfo_RP_00_00_20191220-103738-Beijing
7397990400 bytes total (5552988160 bytes free)
ASR4451#

zhihonghan42550 · ‎12-22-2019

YilinChen 发表于 2019-12-23 13:25
另外，为什么一定要把路由器顶在最外面，用于连接互联网呢？
防火墙直连不行么？
多线路负载均衡可以考虑 ...

这个架构在我来的时候就已经是这样了，
后期再考虑升级，因为都是拨号的宽带。
按照大佬的意思，这个拓扑还有很大的改善空间哦？

wuhao0015 · ‎12-22-2019

more flash:/crashinfo_RP_00_00_20191220-103738-Beijing
看下里面的信息。
估计是bug。。。

YilinChen · ‎12-22-2019

zhihonghan42550 发表于 2019-12-23 13:36
感谢大佬提醒，请帮忙查看，好像是有生成dump
ASR4451#sh ver

Last reload reason: Critical software exception,
check bootflash:crashinfo_RP_00_00_20191220-103738-Beijing

考虑升级版本吧

zhihonghan42550 · ‎12-23-2019

wuhao0015 发表于 2019-12-23 14:41
more flash:/crashinfo_RP_00_00_20191220-103738-Beijing
看下里面的信息。
估计是bug。。。

是的，4000+行BUG，看不懂，无法分析什么原因导致:'(

zhihonghan42550 · ‎12-23-2019

YilinChen 发表于 2019-12-23 15:36
Last reload reason: Critical software exception,

11.2无故出现不定时重启，当时的版本是这个S5的，
System image file is "bootflash:/isr4400-universalk9.03.16.05.S.155-3.S5-ext.SPA.bin"
11.5升级后版本
System image file is "bootflash:/isr4400-universalk9.03.16.08.S.155-3.S8-ext.SPA.bin"
但一切换回办公网就会重启，与升级前一样的情况额、
然后 11.15返厂检修，说是一个配件问题，然后更换了。
12.9收到返修的路由器，恢复配置，未接回办公网，正常。
一接回办公网，又不定时重启。
然后就是现在这情况了。:'(

YilinChen · ‎12-23-2019

zhihonghan42550 发表于 2019-12-23 16:55
11.2无故出现不定时重启，当时的版本是这个S5的，
System image file is "bootflash:/isr4400-universal ...

不行就改架构吧，防火墙直接顶外面:P

zhihonghan42550 · ‎12-23-2019

YilinChen 发表于 2019-12-23 17:20
不行就改架构吧，防火墙直接顶外面

现有设备的防火墙是一台思科的5515，这个不用配置什么，直接改架构就行了？

YilinChen · ‎12-23-2019

zhihonghan42550 发表于 2019-12-23 17:37
现有设备的防火墙是一台思科的5515，这个不用配置什么，直接改架构就行了？

需要改配置的

liyanazure · ‎12-25-2019

可以检查下crashinfo里面的内容，里面应该有rp重启的原因，如果看不过来可以发出来，大家一起看下

zhihonghan42550 · ‎12-26-2019

liyanazure 发表于 2019-12-25 22:40
可以检查下crashinfo里面的内容，里面应该有rp重启的原因，如果看不过来可以发出来，大家一起看下

感谢感谢，我看了里面信息蛮多的，我用附件的形式吧，我把敏感的用*代替，请大佬容易下载，这可能也对故障收集解决有个思路。
原机器已寄回厂家处理，希望也能收到结论，看到底是什么原因导致。

zhihonghan42550 · ‎04-11-2020

这台路由器后面寄回厂家处理，最好是更换主板了，目前已返回使用，已正常运行7week+
希望持续正常，感谢各位大佬解答。