03-29-2020 09:31 AM
Recently, within the last week we've begun having an issue at one of our customer sites with an ASR 1001-X running IOS-XE Everest 16.6.4 (See below for exact IOS). The symptom is that when using the 10G port Te0/0/1 which connects to the ISP's fiber internet service, when attempting to SSH remotely it just gets a connection timed out. The same issue with SNMP also occurs where the requests time out and the NMS reports no response.
The puzzling part is that when the WAN is failed over from Te0/0/1 to the SPA card which has an uplink to an HFC cable modem we're able to SSH in to the device remotely. When the 10G port is up we're also able to SSH in to the switches behind the router that the router has static NAT translations built to port forward to and then we can SSH from those switches back into the router (currently our round about method of remote access for the time being).
When SSH debugging was enabled and we attempted to remotely SSH to the device it acted as though it never received the packet and we got no output at all. When we failed the WAN over to the HFC connection we got plenty of output indicating normal SSH operation.
We have attempted swapping the 10G SFP+ with a known working one, but still seeing the same result. Reloading the router also didn't help. Another side note - periodically the 10G interface will stop responding to pings for about 4 hour periods, although the devices behind such as switches and access points are still accessible. Last night we completed a factory reset and reconfigured the router to no avail as well. If anyone has any ideas I'm open to suggestions.
Output Section:
Cisco IOS XE Software, Version 16.06.04
Cisco IOS Software [Everest], ASR1000 Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 16.6.4, RELEASE SOFTWARE (fc3)
System image file is "bootflash:/asr1001x-universalk9.16.06.04.SPA.bin"
PEOSAZNA1AW-PEORAGGC02#sh ip ssh
Load for five secs: 1%/0%; one minute: 1%; five minutes: 1%
Time source is NTP, *07:41:50.558 Arizona Sun Mar 29 2020
SSH Enabled - version 2.0
Authentication methods:publickey,keyboard-interactive,password
Authentication Publickey Algorithms:x509v3-ssh-rsa,ssh-rsa
Hostkey Algorithms:x509v3-ssh-rsa,ssh-rsa
Encryption Algorithms:aes128-ctr,aes192-ctr,aes256-ctr
MAC Algorithms:hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-sha1-96
KEX Algorithms:diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1
Authentication timeout: 120 secs; Authentication retries: 5
Minimum expected Diffie Hellman key size : 2048 bits
IOS Keys in SECSH format(ssh-rsa, base64 encoded): XXXXXXX.XXXXXXX.XXXXXXX
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQCqJNyhjDAW7rTf26E2McOClY1c4VMDIn1v/8EVvKN1
xjVQSwlc8bc6zunj4qIEHPF96Sg3Kncsc7VUljZhT+mJ0s5gOq1BkVmfWvxOH1J6tn19iddhzGGw9L4v
XQy1+J1WdKFNM1HBfMRorpqdanGJJ7bswHi2FSHFkXEljVUf2w==
For SNMP we see matches on the ACL, but get no response back to our NMS. I have removed the community string and IP for security.
PEOSAZNA1AW-PEORAGGC02#sh snmp host
Load for five secs: 1%/0%; one minute: 1%; five minutes: 1%
Time source is NTP, *07:46:00.032 Arizona Sun Mar 29 2020
Notification host: AAA.BBB.CCC.DDD udp-port: 162 type: trap
user: XXXXXXXXXXXX security model: v2c
PEOSAZNA1AW-PEORAGGC02#show snmp community
Load for five secs: 1%/0%; one minute: 1%; five minutes: 1%
Time source is NTP, *07:47:34.658 Arizona Sun Mar 29 2020
Community name: XXXXXXXXXXXX
Community Index: XXXXXXXXXXXX
Community SecurityName: XXXXXXXXXXXX
storage-type: nonvolatile active access-list: 96
Community name: ILMI
Community Index: ILMI
Community SecurityName: ILMI
storage-type: read-only active
Community name: XXXXXXXXXXXX
Community Index: XXXXXXXXXXXX
Community SecurityName: XXXXXXXXXXXX
storage-type: nonvolatile active access-list: 96
PEOSAZNA1AW-PEORAGGC02#show access-list 96
Load for five secs: 1%/0%; one minute: 1%; five minutes: 1%
Time source is NTP, *07:48:22.325 Arizona Sun Mar 29 2020
Standard IP access list 96
10 permit AAA.BBB.CCC.DDD (5568 matches)
On the VTY we also see matches in the ACL
PEOSAZNA1AW-PEORAGGC02#show access-list 97 | i 30 permit
30 permit AAA.BBB.CCC.DDD (37 matches)
Interface configuration:
interface TenGigabitEthernet0/0/1
description XXXX
ip address III.JJJ.KKK.LLL 255.255.255.248 secondary
ip address EEE.FFF.GGG.HHH 255.255.255.248
no ip redirects
no ip unreachables
ip nat outside
ip policy route-map IPSLA
load-interval 30
PEOSAZNA1AW-PEORAGGC02#sh run | b line vty
line vty 0 4
session-timeout 30
access-class 97 in
exec-timeout 30 0
logging synchronous
login local
exec prompt timestamp
length 0
transport preferred ssh
transport input ssh
line vty 5 15
session-timeout 30
access-class 97 in
exec-timeout 30 0
logging synchronous
login local
exec prompt timestamp
length 0
transport preferred ssh
transport input ssh
PEOSAZNA1AW-PEORAGGC02#show int te0/0/1 controller
Load for five secs: 1%/0%; one minute: 1%; five minutes: 1%
Time source is NTP, *07:53:38.994 Arizona Sun Mar 29 2020
TenGigabitEthernet0/0/1 is up, line protocol is up
Hardware is BUILT-IN-2T+6X1GE, address is 78bc.1aa4.7b01 (bia 78bc.1aa4.7b01)
Description: XXXX
Internet address is EEE.FFF.GGG.HHH/29
MTU 1500 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not supported
Full Duplex, 10000Mbps, link type is force-up, media type is SFP-SR
output flow-control is on, input flow-control is on
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:02, output 00:06:36, output hang never
Last clearing of "show interface" counters never
Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
30 second input rate 30000 bits/sec, 21 packets/sec
30 second output rate 56000 bits/sec, 29 packets/sec
620689 packets input, 103252768 bytes, 0 no buffer
Received 14438 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
919569 packets output, 202869872 bytes, 0 underruns
2 output errors, 0 collisions, 7 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
TenGigabitEthernet0/0/1
0 input vlan errors
0 Number of sub-interface configured
PEOSAZNA1AW-PEORAGGC02#show hw-module subslot 0/0 transceiver 1 idprom
Load for five secs: 2%/0%; one minute: 1%; five minutes: 1%
Time source is NTP, *07:56:15.315 Arizona Sun Mar 29 2020
IDPROM for transceiver TenGigabitEthernet0/0/1:
Description = SFP+ optics (type 130)
Transceiver Type: = SFP+ 10GBASE-SR (273)
Product Identifier (PID) = SFP-10G-SR
Vendor Revision = 02
Serial Number (SN) = CSSSRJ72466
Vendor Name = OEM
Vendor OUI (IEEE company ID) = 00.90.65 (36965)
CLEI code = COUIA8NCAA
Cisco part number = 10-2415-03
Device State = Enabled.
Date code (yy/mm/dd) = 19/07/01
Connector type = LC.
Encoding = 64B66B
Nominal bitrate = 10GE (10300 Mbits/s)
Minimum bit rate as % of nominal bit rate = not specified
Maximum bit rate as % of nominal bit rate = not specified
PEOSAZNA1AW-PEORAGGC02#show stacks | i SSH|SNMP
20560/24000 SSH Event handler
21448/24000 SNMP MIB Platform Sync
22616/24000 SNMP Platform per-sec Job
15168/24000 SNMP IfTable Copy
22616/24000 SNMP Filesys per-sec Job
22616/24000 SNMP Bulk per-sec Job
20600/24000 SNMP MIB Bulk Sync
35256/48000 SSH Process
PEOSAZNA1AW-PEORAGGC02#sh process cpu | i SSH|SNMP
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
268 0 2 0 0.00% 0.00% 0.00% 0 SNMP DBAL WRAPPE
269 0 1 0 0.00% 0.00% 0.00% 0 SNMP DBAL Cache
270 0 1 0 0.00% 0.00% 0.00% 0 SNMP DBAL Cache
293 0 3 0 0.00% 0.00% 0.00% 0 SNMP Timers
543 0 4 0 0.00% 0.00% 0.00% 0 EEM ED SNMP
544 0 4 0 0.00% 0.00% 0.00% 0 EEM ED SNMP Obje
545 0 4 0 0.00% 0.00% 0.00% 0 EEM ED SNMP Noti
555 169 1197 141 0.07% 0.03% 0.00% 2 SSH Process
635 0 12 0 0.00% 0.00% 0.00% 0 SSH Event handle
637 219 5439 40 0.00% 0.00% 0.00% 0 IP SNMP
639 375 2791 134 0.00% 0.00% 0.00% 0 SNMP ENGINE
640 0 2 0 0.00% 0.00% 0.00% 0 IP SNMPV6
641 0 1 0 0.00% 0.00% 0.00% 0 SNMP ConfCopyPro
642 16 67 238 0.00% 0.00% 0.00% 0 SNMP Traps
----------------------- New Test -----------------------
Paessler SNMP Tester - 20.1.58 Computername: TEIMBLS01 Interface: AA.AA.AA.AA
3/29/2020 8:27:52 AM (13 ms) : Device: XX.XX.XX.XX
3/29/2020 8:27:52 AM (25 ms) : SNMP v2c
3/29/2020 8:27:52 AM (32 ms) : Uptime
3/29/2020 8:27:57 AM (5053 ms) : SNMP Datatype: ASN_UNIVERSAL
3/29/2020 8:27:57 AM (5071 ms) : -------
3/29/2020 8:27:57 AM (5077 ms) : DISMAN-EVENT-MIB::sysUpTimeInstance = No response (check: firewalls, routing, snmp settings of device, IPs, SNMP version, community, passwords etc) (SNMP error # -2003) ( 0 seconds )
3/29/2020 8:28:02 AM (10100 ms) : SNMP Datatype: ASN_UNIVERSAL
3/29/2020 8:28:02 AM (10101 ms) : HOST-RESOURCES-MIB::hrSystemUptime.0 = No response (check: firewalls, routing, snmp settings of device, IPs, SNMP version, community, passwords etc) (SNMP error # -2003) ( 0 seconds )
3/29/2020 8:28:02 AM (10101 ms) : Done
----------------------- New Test -----------------------
Paessler SNMP Tester - 20.1.58 Computername: TEIMBLS01 Interface: AA.AA.AA.AA
3/29/2020 8:28:47 AM (6 ms) : Device: XX.XX.XX.XX
3/29/2020 8:28:47 AM (11 ms) : SNMP v2c
3/29/2020 8:28:47 AM (19 ms) : Scaninterfaces
Scanning Standard Interfaces...
3/29/2020 8:28:47 AM Port Scan: Start Scan
3/29/2020 8:28:47 AM Host: XX.XX.XX.XX
3/29/2020 8:28:47 AM Community: XXXX
3/29/2020 8:28:47 AM SNMP Version: V2
3/29/2020 8:28:47 AM Delay: 0
3/29/2020 8:28:47 AM Force32: No
3/29/2020 8:28:47 AM Single Get: No
3/29/2020 8:28:47 AM GET[OIDVal]: 1.3.6.1.2.1.1.3.0
3/29/2020 8:28:47 AM GET[OIDVal]: Try 1
3/29/2020 8:28:47 AM DoRequest[ 0] Init Session
3/29/2020 8:28:47 AM DoRequest[ 0] Init; Host: XX.XX.XX.XX:161, Timeout: 5000000, Version: 2
3/29/2020 8:28:47 AM DoRequest[ 0] Session: Open
3/29/2020 8:28:47 AM DoRequest[ 0] Session: Send Request
3/29/2020 8:28:47 AM DoRequest[ 0] Session: Send Done
3/29/2020 8:28:47 AM DoRequest[ 0] SynchResponse[14546312] Start
3/29/2020 8:28:52 AM DoRequest[ 0] SynchResponse[14546312] Done: Status=2
3/29/2020 8:28:52 AM DoRequest[ 0] Timeout
3/29/2020 8:28:52 AM GET[OIDVal]: Try 2
3/29/2020 8:28:52 AM DoRequest[ 0] Init Session
3/29/2020 8:28:52 AM DoRequest[ 0] Init; Host: XX.XX.XX.XX:161, Timeout: 5000000, Version: 2
3/29/2020 8:28:52 AM DoRequest[ 0] Session: Open
3/29/2020 8:28:52 AM DoRequest[ 0] Session: Send Request
3/29/2020 8:28:52 AM DoRequest[ 0] Session: Send Done
3/29/2020 8:28:52 AM DoRequest[ 0] SynchResponse[14546312] Start
3/29/2020 8:28:57 AM DoRequest[ 0] SynchResponse[14546312] Done: Status=2
3/29/2020 8:28:57 AM DoRequest[ 0] Timeout
3/29/2020 8:28:57 AM GET[OIDVal]: Try 3
3/29/2020 8:28:57 AM DoRequest[ 0] Init Session
3/29/2020 8:28:57 AM DoRequest[ 0] Init; Host: XX.XX.XX.XX:161, Timeout: 5000000, Version: 2
3/29/2020 8:28:57 AM DoRequest[ 0] Session: Open
3/29/2020 8:28:57 AM DoRequest[ 0] Session: Send Request
3/29/2020 8:28:57 AM DoRequest[ 0] Session: Send Done
3/29/2020 8:28:57 AM DoRequest[ 0] SynchResponse[14546312] Start
3/29/2020 8:29:02 AM DoRequest[ 0] SynchResponse[14546312] Done: Status=2
3/29/2020 8:29:02 AM DoRequest[ 0] Timeout
3/29/2020 8:29:02 AM GET[OIDVal]: Try 4
3/29/2020 8:29:02 AM DoRequest[ 0] Init Session
3/29/2020 8:29:02 AM DoRequest[ 0] Init; Host: XX.XX.XX.XX:161, Timeout: 5000000, Version: 2
3/29/2020 8:29:02 AM DoRequest[ 0] Session: Open
3/29/2020 8:29:02 AM DoRequest[ 0] Session: Send Request
3/29/2020 8:29:02 AM DoRequest[ 0] Session: Send Done
3/29/2020 8:29:02 AM DoRequest[ 0] SynchResponse[14546312] Start
3/29/2020 8:29:07 AM DoRequest[ 0] SynchResponse[14546312] Done: Status=2
3/29/2020 8:29:07 AM DoRequest[ 0] Timeout
3/29/2020 8:29:07 AM GET[OIDVal]: Try 5
3/29/2020 8:29:07 AM DoRequest[ 0] Init Session
3/29/2020 8:29:07 AM DoRequest[ 0] Init; Host: XX.XX.XX.XX:161, Timeout: 5000000, Version: 2
3/29/2020 8:29:07 AM DoRequest[ 0] Session: Open
3/29/2020 8:29:08 AM DoRequest[ 0] Session: Send Request
3/29/2020 8:29:08 AM DoRequest[ 0] Session: Send Done
3/29/2020 8:29:08 AM DoRequest[ 0] SynchResponse[14546312] Start
3/29/2020 8:29:13 AM DoRequest[ 0] SynchResponse[14546312] Done: Status=2
3/29/2020 8:29:13 AM DoRequest[ 0] Timeout
Result: #O244
Found standard interfaces:
No standard interfaces found
PEOSAZNA1AW-PEORAGGC02#debug snmp detail
SNMP Detail Debugs debugging is on
PEOSAZNA1AW-PEORAGGC02#debug ip udp port 161
UDP packet debugging is on
PEOSAZNA1AW-PEORAGGC02#
000685: *Mar 29 08:28:01.245 Arizona: UDP: rcvd src=AAA.BBB.CCC.DDD(50115), dst=XX.XX.XX.XX(161), length=63
SrDoSnmp: received get pdu
CheckClassMIBView: all included
CheckMIBView: OID is in MIB view.
000686: *Mar 29 08:28:01.245 Arizona: UDP: sent src=XX.XX.XX.XX(161), dst=AAA.BBB.CCC.DDD(50115), length=66
PEOSAZNA1AW-PEORAGGC02#
000687: *Mar 29 08:28:06.294 Arizona: UDP: rcvd src=AAA.BBB.CCC.DDD(57423), dst=XX.XX.XX.XX(161), length=64
SrDoSnmp: received get pdu
FindObjectClassCore: findObject failed
No such object.
000688: *Mar 29 08:28:06.294 Arizona: UDP: sent src=XX.XX.XX.XX(161), dst=AAA.BBB.CCC.DDD(57423), length=64
PEOSAZNA1AW-PEORAGGC02#
000689: *Mar 29 08:28:56.645 Arizona: UDP: rcvd src=AAA.BBB.CCC.DDD(58361), dst=XX.XX.XX.XX(161), length=63
SrDoSnmp: received get pdu
CheckClassMIBView: all included
CheckMIBView: OID is in MIB view.
000690: *Mar 29 08:28:56.646 Arizona: UDP: sent src=XX.XX.XX.XX(161), dst=AAA.BBB.CCC.DDD(58361), length=66
PEOSAZNA1AW-PEORAGGC02#
000691: *Mar 29 08:29:01.749 Arizona: UDP: rcvd src=AAA.BBB.CCC.DDD(1200), dst=XX.XX.XX.XX(161), length=63
SrDoSnmp: received get pdu
CheckClassMIBView: all included
CheckMIBView: OID is in MIB view.
000692: *Mar 29 08:29:01.749 Arizona: UDP: sent src=XX.XX.XX.XX(161), dst=AAA.BBB.CCC.DDD(1200), length=66
PEOSAZNA1AW-PEORAGGC02#
SrCheckNotificationFilter: OID is included.
SrCheckNotificationFilter: OID is included.
Sr_send_trap: trap sent to AAA.BBB.CCC.DDD:162
SrV2GenerateNotification:Function has reached clean up routine.
When scanning the interfaces it appears to be sending the results back, but the NMS never receives them. However, when the device sends a trap on udp port 162 the NMS does receive that and I see it in the trap receiver. When attempting to read the system uptime I'm seeing a response of find object failed:
PEOSAZNA1AW-PEORAGGC02#
000714: *Mar 29 08:58:31.605 Arizona: UDP: rcvd src=AAA.BBB.CCC.DDD(52586), dst=XX.XX.XX.XX(161), length=64SrDoSnmp: received get pdu
FindObjectClassCore: findObject failed
No such object.
000715: *Mar 29 08:58:31.606 Arizona: UDP: sent src=XX.XX.XX.XX(161), dst=AAA.BBB.CCC.DDD(52586), length=64
Any help would be greatly appreciated!
03-30-2020 12:52 AM
- Activate syslog server and snmp trap receiver for the ASR to a local syslog server and trap receiver (include all possible snmp traps). Followup on messages received on both protocols during normal and also during problem conditions. Look for errors and informational info.
M.
03-31-2020 12:58 PM
04-02-2020 09:19 AM
Anyone have any suggestions?
10-05-2022 02:47 PM
Did you solve it? I'm having the same issue.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide