topic Re: SNMPv3 on ISE 3.2 patch-2 stopped working after every reboot in Network Access Control

SNMPv3 on ISE 3.2 patch-2 stopped working after every reboot

adamscottmaster2013 — Sun, 11 Jun 2023 11:24:28 GMT

I have an ISE cluster 3.2 patch-2 with 4 nodes: Primary Admin/MNT, Secondary Admin/MNT, PSN, and PSN. I have this snmp configuration:

no snmp-server enable
snmp-server enable
snmp-server contact "test@test.com"
snmp-server location "TEST"
no snmp-server user test-v3 v3
snmp-server user test-v3 v3 sha1 plain XXXXXXXXXX YYYYYYYYYYY

Everything is working fine until I reboot the ISE and it stops working after that. From the CLI whenever I do a "show run | include snmp", I see this:

ISEAMP/admin#show running-config | include snmp
snmp-server enable
snmp-server contact test@test.com
snmp-server location TEST
snmp-server user test-v3 v3 sha1 hash ********** **********
ISEAMP/admin#

But it is not working. I had to perform the followings to get it working again:

no snmp-server enable
snmp-server enable
snmp-server contact "test@test.com"
snmp-server location "TEST"
no snmp-server user test-v3 v3
snmp-server user test-v3 v3 sha1 plain XXXXXXXXXX YYYYYYYYYYY

If I reboot the appliance again, it stops working. I can reproduce on multiple ISE 3.2 patch-2 appliances.

Is this another bug? Thoughts?

Re: SNMPv3 on ISE 3.2 patch-2 stopped working after every reboot

Mark Elsen — Sun, 11 Jun 2023 16:18:43 GMT

- FYI : https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvt15998
- Whilst this issue is reported for ISE 2.6 in the bug report the Known Fixed Releases section is 0. In such cases , contact TAC ,

Re: SNMPv3 on ISE 3.2 patch-2 stopped working after every reboot

adamscottmaster2013 — Mon, 12 Jun 2023 15:43:29 GMT

Your reply does nothing to help me :-). This is NOT an issue with either ISE 3.0 or ISE 3.1, I see it in ISE 3.2.

Re: SNMPv3 on ISE 3.2 patch-2 stopped working after every reboot

Mark Elsen — Mon, 12 Jun 2023 16:43:02 GMT

- Could you please start reading replies too ?

Re: SNMPv3 on ISE 3.2 patch-2 stopped working after every reboot

adamscottmaster2013 — Mon, 12 Jun 2023 18:12:41 GMT

There is a bug in ISE 3.2: CSCwe95624

Re: SNMPv3 on ISE 3.2 patch-2 stopped working after every reboot

Feds — Sat, 16 Sep 2023 03:21:37 GMT

Hi, we have the same issue with 3.2 patch 2, SNMP v2c and not v3, post-patching all nodes stopped responding to SNMP.
[edit: issue started when nodes were upgraded to 3.2 and not after patch 2 - this was also confirmed in lab]
Workaround implemented but no success, what worked for us was removing "snmp-server host ..." lines and applying workaround again.
TAC also pointed to CSCwe95624 however there's no mention to the trap config lines being the issue or part of the issue.
Seems like patch 4 (ETA Dec 2023) will contain the fix.

Re: SNMPv3 on ISE 3.2 patch-2 stopped working after every reboot

adamscottmaster2013 — Fri, 21 Jul 2023 15:23:46 GMT

When will Cisco release patch 3?

Re: SNMPv3 on ISE 3.2 patch-2 stopped working after every reboot

Feds — Sun, 23 Jul 2023 23:58:52 GMT

You have to ask TAC.. anyway patch 3 won't have the fix for that bug.

Re: SNMPv3 on ISE 3.2 patch-2 stopped working after every reboot

adamscottmaster2013 — Thu, 27 Jul 2023 14:17:44 GMT

I know patch 3 will not fix the snmp issue but it will be better than patch-2. I am looking at rolling ISE 3.2 at the end of August so I am hopeful that I can use patch 3 and avoid patching ISE for the next two years, unless there are critical security vulnerabilities.

Re: SNMPv3 on ISE 3.2 patch-2 stopped working after every reboot

Feds — Tue, 09 Jan 2024 03:47:10 GMT

For our issue which affected v2c, TAC stated bug is CSCwf32255 and confirmed it's fixed in patch 3 that is available already.

Re: SNMPv3 on ISE 3.2 patch-2 stopped working after every reboot

stayd — Tue, 05 Sep 2023 08:22:43 GMT

You do not need to wait for patch 4 until the end of year. Open the TAC and request the HOT PATCH:

*ise-apply-CSCwe95624_3.2.0.542_patch3-SPA.tar.gz

*ise-rollback-CSCwe95624_3.2.0.542_patch3-SPA.tar.gz

After applying the HP on top of patch 3, snmp works like charm again.

Re: SNMPv3 on ISE 3.2 patch-2 stopped working after every reboot

Feds — Tue, 09 Jan 2024 03:47:31 GMT

Hi stayd,

Thanks for suggesting that, TAC sent the hotpatch and I can confirm it works, but it has to be applied when SNMP is fully configured and operational (that is, after installing patch 3 as you wrote which fixes CSCwf32255 and after applying the workaround for CSCwe95624).

FYI, it's showing as "3.1".
ise32/admin#show logging application hotpatch.log
Fri Sep 15 12:42:26 ACST 2023 => CSCwe95624_3.1.x_patchall

Re: SNMPv3 on ISE 3.2 patch-2 stopped working after every reboot

Minnesotakid — Mon, 18 Dec 2023 21:01:15 GMT

Just an FYI - I recently upgraded from 3.1p7 to 3.2p4 and still hit this SNMP bug. I've submitted a case but posting in case others are also seeing this on the latest patch.

Workaround listed in CSCwe95624 worked for me.

Re: SNMPv3 on ISE 3.2 patch-2 stopped working after every reboot

stayd — Sat, 06 Jan 2024 12:22:02 GMT

Hi Minnesotakid,

so you installed patch 4 and the issue is still there ? Cisco said it will be fixed in patch 4.

Could you confirm it ?

Re: SNMPv3 on ISE 3.2 patch-2 stopped working after every reboot

Minnesotakid — Mon, 08 Jan 2024 19:40:58 GMT

@stayd Yes - after upgrading to 3.2 and then patching to patch 4, I still needed to remove and re-add my snmp configuration to get it work with our NMS. That was the workaround listed in the original bug.

Re: SNMPv3 on ISE 3.2 patch-2 stopped working after every reboot

adamscottmaster2013 — Mon, 08 Jan 2024 20:02:58 GMT

@Minnesotakid: I had two ISE 3.2 patch-3 environment. One environment is SNS-3655 cluster and the other environment is on VMWare ESXi. Both had the same issue in patch-3. The issue is resolved after patching to patch-4.

You might want to open a TAC case with Cisco so that they can investigate.

Re: SNMPv3 on ISE 3.2 patch-2 stopped working after every reboot

Minnesotakid — Mon, 08 Jan 2024 21:36:35 GMT

@adamscottmaster2013 Yep - I did that already, appreciate the note. Was replying in here just in case anyone experienced the same issue as me.

Re: SNMPv3 on ISE 3.2 patch-2 stopped working after every reboot

Yordan1 — Tue, 05 Mar 2024 11:46:41 GMT

i have the same issue with patch5... it is not resolved... we did all t-shoot steps which are in the bug CSCwe95624, same situation.

Any ideas? I don't want to open cases for things that should be solved.

Re: SNMPv3 on ISE 3.2 patch-2 stopped working after every reboot

Feds — Wed, 06 Mar 2024 02:52:23 GMT

Hi @Yordan1 ,
Just tested in lab with both v2c and v3, by installing patch 5 (from no patch): it looks like SNMP doesn't start after the first reload triggered by patching, but if you apply the workaround then at the next reload SNMP looks like being OK.
As I'm coincidentally standing up a new 3.2 node, I've also tested patch 4: 3.2 node with no patch, SNMP v2c and v3 configured, tested OK. Then installed patch 4 - SNMP didn't work after the post-patching reload. Port 161 not showing in "sh port | include 161".
I'm reloading the node again to confirm if it behaves the same as after installing patch 5. I'll confirm later on.

Cheers
Feds

Re: SNMPv3 on ISE 3.2 patch-2 stopped working after every reboot

Yordan1 — Wed, 06 Mar 2024 10:20:42 GMT

hi Feds,

Thanks for the explanation, yes, it works for me now!

regards