ISR C1111-4PLTE Stopped working after several months

DeanHarris27378 · ‎07-19-2020

I have been using an ISR C1111-4PLTE for several months with no major issues other than needing to reboot now and then. The last time I had to reboot it though it didn't come back as usual. If I watch it closely while it is coming up I can catch it while internet access is working, but that only lasts for a minute or two. Unfortunately I don't know a ton about how these work and my attempts at debugging hasn't figure out the issue. Internet access seems to go in and out a few times, then it shuts down service completely until I reboot again. It's clearly getting a good LTE signal though. Below is some logs I grabbed that shows a SPA_OIR-3-SPA_POWERED_OFF message and then the LTE slot going offline. Any ideas or things I should try?

Syslog logging: enabled (0 messages dropped, 2 messages rate-limited, 0 flushes, 0 overruns, xml disabled, filtering disabled)
No Active Message Discriminator.
No Inactive Message Discriminator.
    Console logging: level informational, 305 messages logged, xml disabled,
                     filtering disabled
    Monitor logging: level informational, 0 messages logged, xml disabled,
                     filtering disabled
    Buffer logging:  level informational, 311 messages logged, xml disabled,
                    filtering disabled
    Exception Logging: size (4096 bytes)
    Count and timestamp logging messages: disabled
    Persistent logging: disabled
No active filter modules.
    Trap logging: level informational, 315 message lines logged
        Logging to 10.5.16.11  (udp port 514, audit disabled,
              link up),
              314 message lines logged, 
              0 message lines rate-limited, 
              0 message lines dropped-by-MD, 
              xml disabled, sequence number disabled
              filtering disabled
        Logging Source-Interface:       VRF Name:
Log Buffer (4096 bytes):
OSXE-3-PLATFORM: R0/0: ngiolite: /tmp/sw/cc/0/2/cc_spa/mount/usr/binos/lib64/libprelib.so(backtrace+0x60) [0x7faa312738]
*Jul 18 23:25:36.064: %IOSXE-3-PLATFORM: R0/0: ngiolite: /tmp/sw/cc/0/2/cc_spa/mount/usr/binos/lib64/libC1111-LTE.so(cwan_print_backtrace+0x24) [0x7fa8c572b0]
*Jul 18 23:25:36.065: %IOSXE-3-PLATFORM: R0/0: ngiolite: /tmp/sw/cc/0/2/cc_spa/mount/usr/binos/lib64/libC1111-LTE.so(gobi_sig_handler+0xcc) [0x7fa8c47f30]
*Jul 18 23:25:36.065: %IOSXE-3-PLATFORM: R0/0: ngiolite: linux-vdso.so.1(__kernel_rt_sigreturn+0) [0x7faa3a6510]
*Jul 18 23:25:36.066: %IOSXE-3-PLATFORM: R0/0: ngiolite: /tmp/sw/cc/0/2/cc_spa/mount/lib64/libc.so.6(gsignal+0x38) [0x7fa90556f0]
*Jul 18 23:25:36.066: %IOSXE-3-PLATFORM: R0/0: ngiolite: /tmp/sw/cc/0/2/cc_spa/mount/lib64/libc.so.6(abort+0x1b0) [0x7fa9056bd0]
*Jul 18 23:25:36.067: %IOSXE-3-PLATFORM: R0/0: ngiolite: /tmp/sw/cc/0/2/cc_spa/mount/lib64/libc.so.6(+0x6903c) [0x7fa908d03c]
*Jul 18 23:25:36.067: %IOSXE-3-PLATFORM: R0/0: ngiolite: /tmp/sw/cc/0/2/cc_spa/mount/lib64/libc.so.6(+0x6ee08) [0x7fa9092e08]
*Jul 18 23:25:36.068: %IOSXE-3-PLATFORM: R0/0: ngiolite: /tmp/sw/cc/0/2/cc_spa/mount/lib64/libc.so.6(+0x70e00) [0x7fa9094e00]
*Jul 18 23:25:36.068: %IOSXE-3-PLATFORM: R0/0: ngiolite: /tmp/sw/cc/0/2/cc_spa/mount/lib64/libc.so.6(__libc_malloc+0x54) [0x7fa9096b1c]
*Jul 18 23:25:36.068: %IOSXE-3-PLATFORM: R0/0: ngiolite: /tmp/sw/cc/0/2/cc_spa/mount/usr/lib64/libmaroon.so(__maroon_malloc+0x40) [0x7fa94a1a90]
*Jul 18 23:25:36.068: %IOSXE-3-PLATFORM: R0/0: ngiolite: /tmp/sw/cc/0/2/cc_spa/mount/usr/binos/lib64/libtdllib.so(tdllib_malloc+0x58) [0x7fa9f70ba4]
*Jul 18 23:25:36.068: %IOSXE-3-PLATFORM: R0/0: ngiolite: /tmp/sw/cc/0/2/cc_spa/mount/usr/binos/lib64/libcmcc_ngio.so(+0x1c46c) [0x7fa9b6846c]
*Jul 18 23:25:36.068: %IOSXE-3-PLATFORM: R0/0: ngiolite: /tmp/sw/cc/0/2/cc_spa/mount/usr/binos/bin/ngiolite(ngiolite_cmcc_fpga_led_en_send+0x68) [0x40bf6c]
*Jul 18 23:25:36.068: %IOSXE-3-PLATFORM: R0/0: ngiolite: /tmp/sw/cc/0/2/cc_spa/mount/usr/binos/bin/ngiolite(ngiolite_lte_led_set+0x54) [0x40b4bc]
*Jul 18 23:25:36.068: %IOSXE-3-PLATFORM: R0/0: ngiolite: /tmp/sw/cc/0/2/cc_spa/mount/usr/binos/bin/ngiolite() [0x408524]
*Jul 18 23:25:36.068: %IOSXE-3-PLATFORM: R0/0: ngiolite: /tmp/sw/cc/0/2/cc_spa/mount/usr/binos/lib64/libC1111-LTE.so(ExtractSignalStrengthRSSI+0x2bc) [0x7fa8c355cc]
*Jul 18 23:25:36.068: %IOSXE-3-PLATFORM: R0/0: ngiolite: /tmp/sw/cc/0/2/cc_spa/mount/usr/binos/lib64/libC1111-LTE.so(GetSignalStrengthRespHdlr+0x1bc) [0x7fa8c1e448]
*Jul 18 23:25:36.068: %IOSXE-3-PLATFORM: R0/0: ngiolite: /tmp/sw/cc/0/2/cc_spa/mount/usr/binos/lib64/libC1111-LTE.so(dispatchQmiResponse+0x70) [0x7fa8c2cfcc]
*Jul 18 23:25:36.068: %IOSXE-3-PLATFORM: R0/0: ngiolite: /tmp/sw/cc/0/2/cc_spa/mount/usr/binos/lib64/libC1111-LTE.so(start_send_recv+0xa88) [0x7fa8c4d244]
*Jul 18 23:25:36.068: %IOSXE-3-PLATFORM: R0/0: ngiolite: /tmp/sw/cc/0/2/cc_spa/mount/usr/binos/lib64/libC1111-LTE.so(modem_comm_init+0x1a0) [0x7fa8c5020c]
*Jul 18 23:25:36.068: %IOSXE-3-PLATFORM: R0/0: ngiolite: /tmp/sw/cc/0/2/cc_spa/mount/usr/binos/lib64/libC1111-LTE.so(+0xa2414) [0x7fa8c45414]
*Jul 18 23:25:36.068: %IOSXE-3-PLATFORM: R0/0: ngiolite: /tmp/sw/cc/0/2/cc_spa/mount/usr/binos/lib64/libprelib.so(+0x3654) [0x7faa30f654]
*Jul 18 23:25:36.068: %IOSXE-3-PLATFORM: R0/0: ngiolite: /tmp/sw/cc/0/2/cc_spa/mount/lib64/libpthread.so.0(+0x6fd8) [0x7fa918efd8]
*Jul 18 23:25:36.068: %IOSXE-3-PLATFORM: R0/0: ngiolite: /tmp/sw/cc/0/2/cc_spa/mount/lib64/libc.so.6(+0xc71c0) [0x7fa90eb1c0]
*Jul 18 23:25:38.512: %PMAN-3-PROCHOLDDOWN: R0/0: pman: The process ngiolite has been helddown (rc 134)
*Jul 18 23:25:42.746: %IOMD-3-MODULE_MESSAGE: R0/0: iomd: C1111-LTE[0/2] NGIO control packet loss detected: module reloading
*Jul 18 23:25:48.754: %CELLWAN-2-MODEM_DOWN: Modem in slot 0/2 is DOWN
*Jul 18 23:25:48.754: %CELLWAN-2-MODEM_DOWN: Modem in slot 0/2 is DOWN
*Jul 18 23:25:48.760: %SPA_OIR-3-SPA_POWERED_OFF: subslot 0/2: SPA C1111-LTE powered off after 5 failures within 1200 seconds
*Jul 18 23:25:48.762: %SPA_OIR-6-OFFLINECARD: SPA (C1111-LTE) offline in subslot 0/2

Thanks,

Dean

Georg Pauwen · ‎07-19-2020

Hello,

does the output of:

ISR1111#sh cel 0/2/0 network

render:

Current Service Status = Normal

Try to configure auto-sim:

ISR1111#conf t
ISR1111(config)#controller cellular 0/2/0
ISR1111(config-controller)#lte firmware auto-sim

Wait for 5 minutes, then power cycle the modem:

ISR1111#test cellular 0/2/0 modem-power-cycle

DeanHarris27378 · ‎07-19-2020

Thanks for the super fast reply!

I don't really know what I'm doing, but here are the results from the commands. I didn't see anything coming through the sys log related to these commands though.

#sh cel 0/2/0 network
======================================================
Module is in OFFLINE state
======================================================
Current System Time = None None 0 0:0:0 0
Current Service Status = Normal
Current Service = Circuit Switched
Current Roaming Status = Home
Network Selection Mode = Automatic
Network = 
Mobile Country Code (MCC) = 0
Mobile Network Code (MNC) = 0
Packet switch domain(PS) state = Unknown/Not Applicable
LTE Carrier Aggregation state = Deconfigured
Cell ID = 0
Network MTU is not Available

# #conf t
Command Executed Successfully

# #controller cellular 0/2/0
Command Executed Successfully

# #lte firmware auto-sim
Command Executed Successfully

# #test cellular 0/2/0 modem-power-cycle
Command Executed Successfully

Georg Pauwen · ‎07-19-2020

Hello,

auto sim is supposed to select the right firmware for the respective ISP. It requires a modem recycle, that is what you did. Obviously, the idea is to get the line stable and working...

Can you post the full running configuration of the 1111 ?

DeanHarris27378 · ‎07-19-2020

I hope this is what you mean. I also redacted things I "thought" might be unwise to share like keys and IPs.

#show configuration
Using 3997 out of 33554432 bytes
!
! Last configuration change at 20:30:17 UTC Thu Apr 23 2020 by *****************
!
version 16.9
service timestamps debug datetime msec
service timestamps log datetime msec
platform qfp utilization monitor load 80
no platform punt-keepalive disable-kernel-core
!
hostname NETLTE001-1
!
boot-start-marker
boot-end-marker
!
!
logging buffered informational
logging console informational
logging monitor informational
enable secret 5 *************************
!
aaa new-model
!
!
aaa group server tacacs+ ISE
 server 10.#.#.#
 server 10.#.#.#
 server 10.#.#.#
 server 10.#.#.#
 ip tacacs source-interface Vlan1
!
aaa authentication login default group ISE local
!
!
!
!
!
!
aaa session-id common
!
!
!
!
login on-success log
!
!
!
!
!
!
!
subscriber templating
! 
! 
! 
! 
!
multilink bundle-name authenticated
!
chat-script lte "" "AT!CALL" TIMEOUT 60 "OK"
!
!
crypto pki trustpoint TP-self-signed-4149831175
 enrollment selfsigned
 subject-name cn=IOS-Self-Signed-Certificate-4149831175
 revocation-check none
 rsakeypair TP-self-signed-4149831175
!
!
crypto pki certificate chain TP-self-signed-4149831175
 certificate self-signed 01 nvram:IOS-Self-Sig#1.cer
!
license udi pid C1111-4PLTEEA sn ******************
license accept end user agreement
license boot level securityk9
no license smart enable
!
diagnostic bootup level minimal
!
spanning-tree extend system-id
!
!
username ***************************************************
username **************************************************
!
redundancy
 mode none
!
!
!
!
controller Cellular 0/2/0
 lte sim data-profile 3 attach-profile 1 slot 0
 lte sim data-profile 3 attach-profile 3 slot 1
 lte modem link-recovery rssi onset-threshold -110
 lte modem link-recovery monitor-timer 20
 lte modem link-recovery wait-timer 10
 lte modem link-recovery debounce-count 6
!
!
vlan internal allocation policy ascending
!
! 
!
!
!
!
crypto isakmp policy 10
 encr aes 256
 authentication pre-share
 group 2
crypto isakmp key ******** address #.#.#.# 
!
!
crypto ipsec transform-set TS1 esp-aes 256 esp-sha-hmac 
 mode tunnel
!
!
!
crypto map Miami 10 ipsec-isakmp 
 set peer #.#.#.#
 set security-association lifetime seconds 28800
 set transform-set TS1 
 match address Miami_traffic
!
!
!
!
! 
! 
!
!
interface GigabitEthernet0/0/0
 no ip address
 negotiation auto
!
interface GigabitEthernet0/0/1
 no ip address
 negotiation auto
!
interface GigabitEthernet0/1/0
!
interface GigabitEthernet0/1/1
!
interface GigabitEthernet0/1/2
!
interface GigabitEthernet0/1/3
!
interface Cellular0/2/0
 ip address negotiated
 ip nat outside
 dialer in-band
 dialer idle-timeout 3600 either
 dialer enable-timeout 8
 dialer-group 1
 ipv6 enable
 pulse-time 1
 crypto map Miami
 ip virtual-reassembly
!
interface Cellular0/2/1
 no ip address
 shutdown
!
interface Vlan1
 ip address 10.#.#.# 255.255.255.0
 ip helper-address 10.#.#.#
 ip nat inside
!
ip forward-protocol nd
ip http server
ip http authentication local
ip http secure-server
ip nat inside source list NO_NAT interface Cellular0/2/0 overload
ip route 0.0.0.0 0.0.0.0 Cellular0/2/0
!
!
ip access-list extended Miami_traffic
 permit ip 10.#.#.0 0.0.0.255 10.0.0.0 0.255.255.255
 permit ip 10.#.#.0 0.0.0.255 172.18.0.0 0.0.255.255
ip access-list extended NO_NAT
 deny   ip 10.#.#.0 0.0.0.255 172.18.0.0 0.0.255.255
 deny   ip 10.#.#.0 0.0.0.255 10.0.0.0 0.255.255.255
 permit ip 10.#.#.0 0.0.0.255 any
!
logging host 10.#.#.#
access-list 1 permit 100.111.150.213
access-list 1 permit any
dialer-list 1 protocol ip permit
!
!
!
tacacs-server host 10.#.#.# key ************
tacacs-server host 10.#.#.# key ************
tacacs-server host 10.#.#.# key ************
tacacs-server host 10.#.#.# key ************
!
!
!
control-plane
!
!
line con 0
 exec-timeout 0 0
 transport input none
 stopbits 1
line vty 0 4
 password rnspa55,,,
 length 0
!
!
!
!
!
!
!
pnp profile pnp_cco_profile
 transport https ipv4 #.#.#.# port 443
end

Georg Pauwen · ‎07-19-2020

Hello,

the configuration looks by the book.

The recommended IOS-XE release is Gibraltar-16.12.4 MD. You are currently running 16.9. Upgrading to 16.12 might solve the issue...is that an option ?

DeanHarris27378 · ‎07-19-2020

It's possible. I'll have to work with the networking folks at my employer. The whole covid19 work from home situation makes that difficult if they need physical access to the router to get it upgraded since it's not connecting (for long anyway). I'm wondering if it's something on the employer's side like with the crypto map and access list stuff (none of which I understand). But so far they haven't seen anything to cause this on their end. I guess we keep looking or maybe try a different router, though not sure it shows any sign of being a possible hardware issue.

I appreciate you taking a look though. If nothing else I learned just a tiny bit. :)

- Dean

Georg Pauwen · ‎07-19-2020

Actually, if possible, try to find out if anything has changed on the ISP side as well. It looks like you are losing the physical connection, so the most likely suspects are the router and/or modem themselves, or something the ISP has done.

DeanHarris27378 · ‎07-21-2020

Finally managed to drive the router several miles away where I was sure to get different towers. Worked liked a charm. So evidently the problems are with the tower(s) I'm hitting from here. Now to figure out how to inform the ISP and "hope" they do something to correct the problem.

Thanks for the attempted assistance!

george1214324523 · ‎04-29-2023

this worked C1117-4PLTEEAWE

Leo Laohoo · ‎07-20-2020

%SPA_OIR-3-SPA_POWERED_OFF: subslot 0/2: SPA C1111-LTE powered off after 5 failures within 1200 seconds

Contact Cisco TAC and organize for an RMA. The router is toast.