ISE backup hung every 15th day..

[dilnaazhum]

Hello All,

I am glad if someone help me with this.. I know we are running very old OS and unable to get support from CISCO TAC due to EOS..I am new to ISE and unable to get support..

I have seen the behavior of ISE backup hung every 15th day and once we restart the admin node backup will work as expected, today i have seen this log while executing the "sh app stat ise"

rpmdb: Thread/process 7538/139916087605152 failed: Thread died in Berkeley DB li
brary
error: db3 error(-30974) from dbenv->failchk: DB_RUNRECOVERY: Fatal error, run d
atabase recovery
error: cannot open Packages index using db3 - (-30974)

error: cannot open Packages database in /var/lib/rpm

admin# sh backup status
%% Configuration backup status
%% ----------------------------
% backup name: ISEConfigBackup
% repository: abcd
% start date: Sun Apr 28 00:30:14 BST 2019
% scheduled: yes
% triggered from: Admin web UI
% host: admin
% status: Backup is in progress...
% progress %: 50
% progress message: Completing ISE Backup Staging

ISE-3395-K9 - 1.4.0.253 Patch 8..

We are working to replace with new cisco ISE node 3595 which are in full deployment method (2 Admin, 2 PSN) 

https://access.redhat.com/solutions/2148761

can any one help me on this.. 

Thanks and regards

Afeez Mali

[Arne Bier · Accepted solution]

I have not seen this one before. 

When you say "restart node" do you mean application restart, or server reboot?

Since we don't get root access to the Linux shell, you can only "rattle the cage" and see what happens next.

I would do the following in this order

Purge any logs that you don't need (to make the backup smaller)

Make sure the repository directory doesn't contain too many files (i.e. remove old files)

Delete the config backup job and create a new one

Patch to the latest release (if not already done so)

Reboot the node

Try FTP instead of SFTP?  Maybe it's an SSL issue.  Who knows.

Become your own TAC engineer :-)  if you have the time, download the Support Bundle and go through the logs to see if you can spot the cause.  However it would possibly require deeper knowledge of which DEBUG level to set, and which file to analyse.   But that is probably what the TAC would be doing next.

The backups might be failing for a reason that you can never control or fix.  So ultimately if you have no support, then you are forced to go to ISE 2.4 or later and I would even recommend building that node from scratch (don't try any upgrade paths, because you drag a lot of technical debt with you!)

[dilnaazhum]

Hello Arne,

Thanks for your reply !!

We restart server (Primary admin node) on and every 15th day and later in the evening we can see successful backup.

We deleted logs on backup server which is using FTP, not SFTP.

Currently i am replacing trusted certificates which are of expired. Also checking the Local logging setting which says 7 days local setting, but i can see old logs while executing the sh logging command. probably i can go for " Delete Local Logs now" option as per https://community.cisco.com/t5/policy-and-access/how-to-delete-old-logs-in-ise/td-p/2455844.

I was going through the option of Support bundle, but i have not seen option such as for backup also not sure component name relate to backup..

This problem i have seen after remote logging target configuration enabled / logs are forward to Splunk server.

Appreciated if you could help me on component name relate to backup stuff in Bundle support, which i can take some risk to capture traffic / simulate that in LAB before trying in production environment..

Thanks and regards

Afeez Mali

[Arne Bier]

Hi @dilnaazhum 

I don't have an ISE 1.4 system and perhaps someone else can comment.  In ISE 2.4 I would think you need to enable DEBUG level on the "nsf" component.  This is the one that logs to the ise-psc.log - well, at least in ISE 2.4 it does.  Maybe it was different in ISE 1.4

What's stopping you from moving forward to a new ISE 2.4 deployment? ;-)

cheers

[dilnaazhum]

Hi Arne,

Thanks will check that and  It is downtime for complete migration..

Migration plan as follows:

Will build separate setup and will do Pre-test with traffic diversion of one switch / few SSID (guest and Production) to new setup. Once all testing complete, we take one window for full migration (by changing the IP address to the current one), so we looking for downtime.. or else we have to raise almost 300 change request if we do one by one.. :)

Thanks and regards

Afeez Mali