ifindex_server crash

[schwarzenbachcyb]

Does anyone also see DUMPER crashes related to the ifindex_server process on his ASR9k? It looks like it's something with the snmp process...

I have an open TAC but it's not a known bug. Maybe anyone else has also this problem...?

[philclemens1835]

What hardware platform, RSP, LC, and XR version?

Also, can you paste a few lines of the log file?

We had an issue with 5.1.3 where we would get process dumps periodically.  There's an update that fixes this, but we're migrating to 6.4.2, and have not seen this issue on the upgraded routers.

Fortunately, with XR, the process can restart itself and carry on.  You probably want to do a "show filesystem" to ensure you don't have a drive getting full of dumper files.

[schwarzenbachcyb]

It's 6.2.3 with A9K-RSP880 and A9K-48X10GE-1G. ASR9006.

I tried to restart the process manually, but since then I can't query some SNMP values anymore... and it's still crashing all the time.

log messages:

Aug 20 01:50:53 cybl-ch-gtg-pcr-001.cyberlink.ch 249686: RP/0/RSP0/CPU0:Aug 20 01:50:53.769 : dumper[59]: %OS-DUMPER-4-SIGNAL_NUMBER : Thread 1 received SIGNA
Aug 20 01:50:54 cybl-ch-gtg-pcr-001.cyberlink.ch 249710: RP/0/RSP0/CPU0:Aug 20 01:50:53.780 : dumper[59]: %OS-DUMPER-6-TRACE_BACK : #0  0xba1c29b
Aug 20 01:50:54 cybl-ch-gtg-pcr-001.cyberlink.ch 249712: RP/0/RSP0/CPU0:Aug 20 01:50:53.784 : dumper[59]: %OS-DUMPER-6-TRACE_BACK : #1  0x4206dfc
Aug 20 01:50:54 cybl-ch-gtg-pcr-001.cyberlink.ch 249714: RP/0/RSP0/CPU0:Aug 20 01:50:53.784 : dumper[59]: %OS-DUMPER-6-TRACE_BACK : #2  0x42055f2
Aug 20 01:50:54 cybl-ch-gtg-pcr-001.cyberlink.ch 249716: RP/0/RSP0/CPU0:Aug 20 01:50:53.784 : dumper[59]: %OS-DUMPER-6-TRACE_BACK : #3  0xd83cc69
Aug 20 01:50:54 cybl-ch-gtg-pcr-001.cyberlink.ch 249718: RP/0/RSP0/CPU0:Aug 20 01:50:53.784 : dumper[59]: %OS-DUMPER-6-TRACE_BACK : #4  0xd847962
Aug 20 01:50:54 cybl-ch-gtg-pcr-001.cyberlink.ch 249720: RP/0/RSP0/CPU0:Aug 20 01:50:53.784 : dumper[59]: %OS-DUMPER-6-TRACE_BACK : #5  0xd847f29
Aug 20 01:50:54 cybl-ch-gtg-pcr-001.cyberlink.ch 249722: RP/0/RSP0/CPU0:Aug 20 01:50:53.784 : dumper[59]: %OS-DUMPER-6-TRACE_BACK : #6  0xd848b2d
Aug 20 01:50:54 cybl-ch-gtg-pcr-001.cyberlink.ch 249724: RP/0/RSP0/CPU0:Aug 20 01:50:53.784 : dumper[59]: %OS-DUMPER-6-TRACE_BACK : #7  0xd848c01
Aug 20 01:50:54 cybl-ch-gtg-pcr-001.cyberlink.ch 249726: RP/0/RSP0/CPU0:Aug 20 01:50:53.784 : dumper[59]: %OS-DUMPER-6-TRACE_BACK : #8  0xd848d63
Aug 20 01:50:54 cybl-ch-gtg-pcr-001.cyberlink.ch 249728: RP/0/RSP0/CPU0:Aug 20 01:50:53.784 : dumper[59]: %OS-DUMPER-6-TRACE_BACK : #9  0xb98bc63
Aug 20 01:50:54 cybl-ch-gtg-pcr-001.cyberlink.ch 249730: RP/0/RSP0/CPU0:Aug 20 01:50:53.784 : dumper[59]: %OS-DUMPER-6-TRACE_BACK : #10  0xb989c8c
Aug 20 01:50:54 cybl-ch-gtg-pcr-001.cyberlink.ch 249732: RP/0/RSP0/CPU0:Aug 20 01:50:53.784 : dumper[59]: %OS-DUMPER-6-TRACE_BACK : #11  0xb98b1f9
Aug 20 01:50:55 cybl-ch-gtg-pcr-001.cyberlink.ch 249874: RP/0/RSP0/CPU0:Aug 20 01:50:53.983 : dumper[59]: %OS-DUMPER-5-CORE_FILE_NAME : Core for process pkg/bin/ifindex_server at harddisk:/dumper/ifindex_server_252.by.ifindex_server.20190820-015053.node0_RSP0_CPU0.x86.Z on local_node
Aug 20 01:51:24 cybl-ch-gtg-pcr-001.cyberlink.ch 249882: RP/0/RSP0/CPU0:Aug 20 01:51:24.099 : dumper[59]: %OS-DUMPER-5-DUMP_SUCCESS : Core dump success
Aug 20 01:51:44 cybl-ch-gtg-pcr-001.cyberlink.ch 249884: RP/0/RSP0/CPU0:Aug 20 01:51:44.343 : dumper[59]: %OS-DUMPER-6-CORE_MD5 : harddisk:/dumper/ifindex_server_252.by.ifindex_server.20190820-015053.node0_RSP0_CPU0.x86.Z has just been written. size:129477463 md5:cf73673a682788c06aa30571ab24fd88

Do you know the BUG ID?

[philclemens1835]

This is different than what we were experiencing.

However, TAC should be able to figure this out.

Also, if you have direct ssh access, you could try the Cisco CLI Analyzer on this box, and see what it comes up with.  Very good tool, however last time I checked, it doesn't work through a jump host.

[Elvin Arias]

This log is the key, but seems to be truncated:

"Aug 20 01:50:53 cybl-ch-gtg-pcr-001.cyberlink.ch 249686: RP/0/RSP0/CPU0:Aug 20 01:50:53.769 : dumper[59]: %OS-DUMPER-4-SIGNAL_NUMBER : Thread 1 received SIGNA"

The signal will indicate what specifically happened. Generally, procedure goes this way:

1. Upload the dump generated in the box.
2. Take show version / show platform / show install active summary outputs.
3. XR TAC should proceed with the decoding of the core dump and provide it to the corresponding team and provide an update in a few days.

Regards,

Elvin