Solved: Re: Cisco ISE SNMP Traps - Port unreachable

Netmart · ‎02-21-2025

Hello,

We operate a ISE cluster where two of them do show some issue with snmp traps / polling:

Cisco Identity Services Engine

---------------------------------------------

Version : 3.2.0.542

Build Date : Wed Oct 19 16:27:24 2022

Install Date : Wed Jul 26 16:49:04 2023

Cisco Identity Services Engine Patch

---------------------------------------------

Version : 2

Config Example:

snmp-server enable
snmp-server trap dskThresholdLimit 25
snmp-server engineid <engine ID>
snmp-server host <NMS IP> version 2c <Community String>
snmp-server host <NMS IP> version 2c <Community String>

test with snmp walk

$ snmpwalk -v 2c -c <Community String> <NMStationIP1> 1.3.6.1.2.1.1

Based on Wireshark capture:

Seq SRC DST Info

1 NMS IP ISE Node get-next-request 1.3.6.1.2.1.1

2 ISE Node NMS snmpv2-trap 1.3.6.1.2.1.1.3.0 1.3.6.1.6.3.1.1.4.1.0 1.3.6.1.6.3.1.1.4.3.0

3 NMS IP ISE Node Destination unreachable (Port unreachable)

Syslog:

2025-02-20T01:22:46.623491+00:00 ISE-Node-u5 ADE-SERVICE[1790]: [6078]:[info] snmp: cars_snmpcfg.c[71] [system]: Config Success

2025-02-20T01:22:46.627579+00:00 ISE-Node-u5 ADE-SERVICE[1790]: [6078]:[info] snmp: cars_snmpd_conf.c[713] [system]: Configured SysContact Unknown

2025-02-20T01:22:46.627775+00:00 ISE-Node-u5 ADE-SERVICE[1790]: [6078]:[info] snmp: cars_snmpd_conf.c[729] [system]: Configured Syslocation Unknown

2025-02-20T01:22:46.627814+00:00 ISE-Node-u5 ADE-SERVICE[1790]: [6078]:[info] snmp: cars_snmpd_conf.c[899] [system]: DiskThresholdPercentage set to 25

2025-02-20T01:22:46.627847+00:00 ISE-Node-u5 ADE-SERVICE[1790]: [6078]:[info] snmp: cars_snmpd_conf.c[819] [system]: Configured trapcommunity set to default

2025-02-20T01:22:46.627879+00:00 ISE-Node-u5 ADE-SERVICE[1790]: [6078]:[info] snmp: cars_snmpd_conf.c[496] [system]: Version: v2c

2025-02-20T01:22:46.627911+00:00 ISE-Node-u5 ADE-SERVICE[1790]: [6078]:[info] snmp: cars_snmpd_conf.c[514] [system]: TrapCommunity: <CommunityString>

2025-02-20T01:22:46.627939+00:00 ISE-Node-u5 ADE-SERVICE[1790]: [6078]:[info] snmp: cars_snmpd_conf.c[550] [system]: Inserted first host/ip <NMStationIP1>

2025-02-20T01:22:46.627970+00:00 ISE-Node-u5 ADE-SERVICE[1790]: [6078]:[info] snmp: cars_snmpd_conf.c[835] [system]: Adding Trap clients

2025-02-20T01:22:46.628002+00:00 ISE-Node-u5 ADE-SERVICE[1790]: [6078]:[info] snmp: cars_snmpd_conf.c[853] [system]: snmp server started

2025-02-20T01:22:46.628033+00:00 ISE-Node-u5 ADE-SERVICE[1790]: [6078]:[info] snmp: cars_snmpcfg.c[97] [system]: Config Success

:

/opt/CSCOcpm/appsrv/apache-tomcat-8.5.13/lib/snmp4j-2.3.1.jar:

/opt/CSCOcpm/appsrv/apache-tomcat-8.5.13/lib/snmptrap-2.4.0-357.jar:

/opt/CSCOcpm/appsrv/apache-tomcat-8.5.13/lib/snmpquery-2.4.0-357.jar:

/opt/CSCOcpm/appsrv/apache-tomcat-8.5.13/lib/snmpinfrastructure-2.4.0-357.jar:

Attribute:ProbeName value:snmpquery

Attribute:probeclass value:com.cisco.profiler.probes.snmpquery.SNMPQuery

Attribute:ProbeName value:snmptrap

Attribute:probeclass value:com.cisco.profiler.probes.snmptrap.SNMPTrapListener

/opt/CSCOcpm/appsrv/apache-tomcat-9.0.16/lib/snmptrap-2.7.0-356.jar:

/opt/CSCOcpm/appsrv/apache-tomcat-9.0.16/lib/snmpquery-2.7.0-356.jar:

/opt/CSCOcpm/appsrv/apache-tomcat-9.0.16/lib/snmpinfrastructure-2.7.0-356.jar:

/opt/CSCOcpm/appsrv/apache-tomcat-9.0.16/lib/snmp4j-2.6.2.jar:

Attribute:ProbeName value:snmpquery

Attribute:probeclass value:com.cisco.profiler.probes.snmpquery.SNMPQuery

Attribute:ProbeName value:snmptrap

Attribute:probeclass value:com.cisco.profiler.probes.snmptrap.SNMPTrapListener

/opt/CSCOcpm/appsrv/apache-tomcat-9.0.54/lib/snmpquery-3.2.0-542.jar:

/opt/CSCOcpm/appsrv/apache-tomcat-9.0.54/lib/snmptrap-3.2.0-542.jar:

/opt/CSCOcpm/appsrv/apache-tomcat-9.0.54/lib/snmpinfrastructure-3.2.0-542.jar:

/opt/CSCOcpm/appsrv/apache-tomcat-9.0.54/lib/snmp4j-2.6.2.jar:

Attribute:ProbeName value:snmpquery

Attribute:probeclass value:com.cisco.profiler.probes.snmpquery.SNMPQuery

Attribute:ProbeName value:snmptrap

Attribute:probeclass value:com.cisco.profiler.probes.snmptrap.SNMPTrapListener

drop table UPSWizrdCnfgrtn_snmpIpRngs

create table UPSWizrdCnfgrtn_snmpIpRngs

/opt/CSCOcpm/appsrv/apache-tomcat-9.0.54/lib/snmpinfrastructure-3.2.0-542.jar: FAILED

/opt/CSCOcpm/appsrv/apache-tomcat-9.0.54/lib/snmpquery-3.2.0-542.jar: FAILED

/opt/CSCOcpm/appsrv/apache-tomcat-9.0.54/lib/snmptrap-3.2.0-542.jar: FAILED

verified: /opt/CSCOcpm/appsrv/apache-tomcat-9.0.54/lib/snmpinfrastructure-3.2.0-542.jar from: patch 2

verified: /opt/CSCOcpm/appsrv/apache-tomcat-9.0.54/lib/snmpquery-3.2.0-542.jar from: patch 2

verified: /opt/CSCOcpm/appsrv/apache-tomcat-9.0.54/lib/snmptrap-3.2.0-542.jar from: patch 2

Netmart · ‎02-26-2025

Ok, I may have found the answer.

I agree, this ISE is loaded with bugs, at least it keeps the Cisco engineers busy/employed:

ISE 3.2 SNMP is not working after node restart

Last Modified

Feb 17, 2025

Products (1)

Cisco Identity Services Engine

Known Affected Release

003.002(000.902)

Description (partial)

Symptom: SNMP is not working after every node reload. CLI command "show process | include snmp" gives empty output. Conditions: ISE 3.2 until Patch 3 ISE 3.3 without any patches

View solution in original post

Arne Bier · ‎02-23-2025

SNMP in ISE has been quite buggy historically. In SE 3.3p4 it's finally stable. I had issues with older versions - you might want to patch your system. The only thing you can do as a user (apart from patching and upgrading) is to toggle the SNMP server off, and on again. Or I also had success in the past with disabling SNMP, rebooting the node, and then enabling it again. If it's still broken, then engage TAC.

Also check that there are no firewall rules that might be blocking UDP/161 and UDP/162

Netmart · ‎02-26-2025

Hi Arne,

Thank you again. With the help of snmpwalk I cycled through all three existing ISE clusters [all three running different release] and it turned out that all nodes in one cluster do not receive snmp responses; firewall issue can be ruled out. Therefore, I am wondering, if there is a known issue with release Version: 3.2.0.542.

Result:

Version: 3.2.0.542, patch 2, ADE-OS Version: 3.2.0.401 ==> no snmp response

Version: 2.7.0.356, patch 2,3,6, ADE-OS Version: 2.7.0.356 ==> working

Version: 3.1.0.518, patch 3, ADE-OS Version: 3.1.0.518 ==> working

Thank you again.

I appreciate any help/feedback

Netmart · ‎02-26-2025

Ok, I may have found the answer.

I agree, this ISE is loaded with bugs, at least it keeps the Cisco engineers busy/employed:

ISE 3.2 SNMP is not working after node restart

Last Modified

Feb 17, 2025

Products (1)

Cisco Identity Services Engine

Known Affected Release

003.002(000.902)

Description (partial)

Symptom: SNMP is not working after every node reload. CLI command "show process | include snmp" gives empty output. Conditions: ISE 3.2 until Patch 3 ISE 3.3 without any patches

Netmart · ‎02-26-2025

Working one:
Version : 2.7.0.356
# show process | include snmp
root 27611 pts/4 27607 Wed Feb 26 22:09:50 2025 grep -A 0 -B 0 snmp 00:00:00
--
root 29317 ? 1 Thu Jan 5 19:53:19 2023 /usr/sbin/snmpd -LS0-6d -Lf 09:26:20

Non-working one:
Version : 3.2.0.542
n#show process | include snmp
root 3662629 ? 1 Thu Feb 20 01:34:33 2025 /usr/sbin/snmpd -LS0-6d -Lf 00:06:3

Netmart · ‎02-26-2025

The fix is either to upgrade to a less bug-full version or remove and add snmp config; the latter may have to be repeated a number of times. Reloading the node might be another option, but without Cisco coverage there is always a risk to run into another surprise. Thank all for your comments and help.