Cisco ISE No Livelogs after MnT Database Reset

[Niko99]

Hello,

I have opened a Cisco TAC regaring failing scheduled config Backups in our deployment. We are running Cisco ISE 3.1 Patch 10. Cisco TAC found out, that our /OPT Direcotry is above the treshold of 70 %. TAC deleted some Files but it did not went down below the threshold so he recommended a MnT Database Reset. I did that on the nodes but after the MnT Database Reset it does not show any Logs in the ISE GUI. In our Remote Logging target we set up we still see the Logs. TAC took a Support Bundle after that but could not replicate the issue. In the lab the Reset went through without any problems so he recommended to do the Reset again. But this did not help either.

Anyone with a similar issue an a solution ?

Any suggestions ?

[Marcelo Morais]

Hi @Niko99 ,

 the Reset M&T Database option is to reset the Database Files to the Factory Default so no Data retained up until the time of issuing the option:

ise/admin# application configure ise
Selection configuration option
 ...
 [4]Reset M&T Database
 ...

Reset M&T Database is used:

• If the Database Files are taking up to too much File System Space !!!
• Could solve the incorrect MAC to IP Binding at Live Session
• Could solve the blank page at Live Logs
• Reset the Authentication Summary report
• Could solve the blank RADIUS Accounting report (at Operations > Reports > Reports > Endpoint and Users) when it is Export To.

Hope this helps !!!

[Niko99]

Hello Marcelo,

• If the Database Files are taking up to too much File System Space !!!

This was the reason Cisco TAC recommended the M&T Database Reset. Our /OPT Directory took up too much space. But after the Reset the Visibility of our Logs was gone. Cisco TAC has not found the reason for the Issue yet.

We are currently working on a new Node, that will replace the corrupted one in our deployment. But if anyone got some suggestions regarding the current state and how we cen resolve the Issue in the old Node feel free to share them.

Best Regards

Niko

[Marcelo Morais]

Hi @Niko99 ,

 talking about OPT, if your OPT_Threshold is above 70, you probably won't be able to generate a Backup either, am I correct?

 To check the /OPT:

ise/admin# show disk
 Filesystem  Size  Used  Avail  Use%  Mounted on
 ...
 /dev/sda7   550G  390G  160G   71%   /opt
 ...

TAC must have accessed your Node via root to find the directories with the most used space so that it can delete them, something like this:

ade# df -h
ade# cd /opt
ade# du -sh *
ade# rm -rf *.log.*

This way, TAC is able to remove some old logs.

Note: what is your Disk Size: 300 GB, 600 GB ... ? Appliance or VM ?

Hope this helps !!!

[Niko99]

Hello Marcelo,

We gave the VM a Disk Size of 600GB and Used the  600GB For Medium Deployment .ova Template. However, the escalation-Team of Cisco informed us, that they can see, that the /OPT only Uses 300GB for some reason. They are currently checking the .ova we used. We currently working with the Workaround, that a Device-Admin Node is takeing the Persona of "Primary Monitoring" now. 

Kind Regards

Niko

[Marcelo Morais]

Hi @Niko99 ,

 interesting ... as if you had a Disk Size of 300GB and not 600GB.

PS: I'm curious about TAC's response about .OVA !!!

Best regards

[Scott Fella]

I had issues also with our MNT nodes and alerts about free space.  TAC deleted items and eventually the alerts just came back.  What was a fix was, I removed the secondary MNT from the cube, factory rest the secondary MNT and then joined it back to the cube and waited for the node to fully sync.  I promoted that node to primary and performed the steps again to factory reset the node and that seemed to fix all the issue we were having with the MNT. Might be something to try since you have multiple nodes.  I also did this to all our other nodes (20) to start fresh.  Seemed to really help with other issues we were having.