No accounting logs in report cisco ISE 2.4 patch 10

[anilkumar.cisco]

Hello Team,

we have ISE in VM machine.. and suddenly this report coming has gone off..

It was working earlier..

How we can re-activate it without VM reboot..

Any idea or troubleshooting advise will be highly appreciated..

the below is the status of ISE.. services..

show application status ise



ISE PROCESS NAME                       STATE            PROCESS ID

Database Listener                      running          3192

Database Server                        running          113 PROCESSES

Application Server                     running          6051

Profiler Database                      running          7422

ISE Indexing Engine                    running          19050

AD Connector                           running          21911

M&T Session Database                   running          6087

M&T Log Collector                      running          16816

M&T Log Processor                      running          16727

Certificate Authority Service          disabled

EST Service                            disabled

SXP Engine Service                     disabled

Docker Daemon                          running          8321

TC-NAC Service                         disabled



Wifi Setup Helper Container            disabled

pxGrid Infrastructure Service          disabled

pxGrid Publisher Subscriber Service    disabled

pxGrid Connection Manager              disabled

pxGrid Controller                      disabled

PassiveID WMI Service                  disabled

PassiveID Syslog Service               disabled

PassiveID API Service                  disabled

PassiveID Agent Service                disabled

PassiveID Endpoint Service             disabled

PassiveID SPAN Service                 disabled

DHCP Server (dhcpd)                    disabled

DNS Server (named)                     disabled

ISE RabbitMQ Container                 running          8648

[balaji.bandi]

May be Disk usage also to check :

Login CLI and post below information :

# dir

[anilkumar.cisco]

From PAN

/admin# dir

Directory of disk:/

       4096 Nov 13 2021 01:01:02  corefiles/

2434959857 Dec 02 2019 20:29:31  ise-patchbundle-2.4.0.357-Patch10-19091709.SPA.x86_64.tar.gz

1121470253 Nov 03 2018 10:31:57  ise-patchbundle-2.4.0.357-Patch4-18092802.SPA.x86_64.tar.gz

1138860565 Jan 25 2019 19:59:55  ise-patchbundle-2.4.0.357-Patch5-18112000.SPA.x86_64.tar.gz

      16384 Nov 03 2018 03:05:26  lost+found/

           Usage for disk: filesystem

                 9482919936 bytes total used

                19756482560 bytes free

                30829043712 bytes available

Secondary PAN

admin# dir

Directory of disk:/

       4096 Jan 04 2021 01:01:02  corefiles/

2434959857 Dec 02 2019 21:18:09  ise-patchbundle-2.4.0.357-Patch10-19091709.SPA.x86_64.tar.gz

1121470253 Nov 03 2018 10:15:33  ise-patchbundle-2.4.0.357-Patch4-18092802.SPA.x86_64.tar.gz

1138860565 Jan 25 2019 20:17:58  ise-patchbundle-2.4.0.357-Patch5-18112000.SPA.x86_64.tar.gz

      16384 Nov 03 2018 01:16:48  lost+found/

           Usage for disk: filesystem

                 4742651904 bytes total used

                24496750592 bytes free

                30829043712 bytes available

PSN

/admin# dir

Directory of disk:/

       4096 Jul 08 2021 04:01:03  corefiles/

2434959857 Dec 02 2019 20:59:27  ise-patchbundle-2.4.0.357-Patch10-19091709.SPA.x86_64.tar.gz

1138860565 Oct 17 2019 10:38:11  ise-patchbundle-2.4.0.357-Patch5-18112000.SPA.x86_64.tar.gz

      16384 Oct 17 2019 09:43:17  lost+found/

           Usage for disk: filesystem

                 3621105664 bytes total used

                25618296832 bytes free

                30829043712 bytes available

[anilkumar.cisco]

From above output..

we have free space around 19 GB in PAN, 24 GB in SAN and 25 GB in PSN.

Not sure what is the issue..

it could be hitting some bugs here..

[balaji.bandi]

Do you see Live events ? only accounting logs missing ?

[anilkumar.cisco]

Yes we are able to see Live logs.. but accounting logs are missing for both tacacs and radius 

[balaji.bandi]

but accounting logs are missing for both tacacs and radius 

this looks odd, you mentioned it was working suddendly not work, Do you see any FW in the path ?

i will pickup 1 network device and start debugging to clarify any FW in the path blocking this ?

[anilkumar.cisco]

the customer has done some certificate upload in month of Sep which required ISE reload..

He is suspecting after that only these logs were missing..

I will check for the firewall

Any other idea?

[thomas]

RADIUS Accounting Start and Stop status messages must be sent from the network access devices for sessions to be tracked in ISE.

Have you confirmed that RADIUS Accounting  is configured on the network device?  Nothing has changed recently with the configuration? Does it fail for all network devices or only certain devices?

Any firewalls blocking UDP:1813 traffic?

Can you see the Accounting packets arriving at ISE with a packet capture?

[anilkumar.cisco]

Yes Thomas,

Enclosed is the extract from Wireshark logs.. I am able to see both accouting and authentication packets..

But accounting is working in 1646 port and authentication is on 1812 port... not sure why. but accounting start and stop both messages are coming.. in ISE in port 1646 via wireshark message..

While analysing cisco ISE yesterday.. i found some alarm related to High load average and Insufficient Virtual Machine resources error on both PAN and SAN devices..

whereas PSN is Ok and did not rebooted from past 2 years..

they are not meeting  as per Cisco specification..

[balaji.bandi]

is this Wireshark output from ISE - Does your ISE Listens on Accounting 1646 / this odd for me, they normally 1812/1813  or 1645/1646

While analysing cisco ISE yesterday.. i found some alarm related to High load average and Insufficient Virtual Machine resources error on both PAN and SAN devices..

May be worth reboot and check

below reference diagram provides high level.

[anilkumar.cisco]

it appear odd to me as well..

Yes this capture was from ISE PSN.

Don't we have any other option apart from reboot..

[balaji.bandi]

Don't we have any other option apart from reboot..

ask yourself, Do you have any other option, have you tried any ? - we would like to hear learning lessons from you, if you tried.

[anilkumar.cisco]

i was thinking about enable and disable accounting loggings. and then restarting monitoring services..

[balaji.bandi]

Restart ISE Service  vs Reboot kit is the same (both have downtime) - if this is only single kit.

since it was long uptime, personally i take the call reboot device.