topic After the upgrade, runtime in Network Access Control

Upgrading an ACS deployment from 5.3 to 5.5.

andrea.meconi — Mon, 11 Mar 2019 04:43:16 GMT

Hello.

I'm using ACS on virtual machines with a primary and secondary instances.

The primary server also functions as a log collector in my deployment. I don't use AD. LDAP is configured.

Now I need to upgrade to 5.5.

Reading installation guide I find this note..

Upgrading to ACS 5.5 may fail if any LDAP identity store is configured without groups or attributes and
an AD identity store is not configured. To avoid this issue, before upgrading to ACS 5.5, either add
groups or attributes to the LDAP identity store or configure an AD identity store.

How can I verify this?

Thanks.

Regards.

Andrea

Hey Andrea,Go to External

edwardcollins7 — Tue, 20 May 2014 06:52:53 GMT

Hey Andrea,

Go to External Identity stores-->LDAP

Go to directory attributes, directory groups and check, it should not be empty, at least one of them should be populated.

Rate if Useful 🙂

Sharing knowledge makes you Immortal.

Regards,

Hello Edward and many thanks

andrea.meconi — Wed, 21 May 2014 12:27:16 GMT

Hello Edward and many thanks for your help.

Groups and attributes are empty. So I need to populate for the upgrade only and remove settings when upgrade is done!

And AD? What's about it?

Regards.

Andrea

Andrea,Yes populate them

edwardcollins7 — Thu, 22 May 2014 07:36:01 GMT

Andrea,

Yes populate them before upgrade and remove later.

No issues with AD, leave it in what ever condition it is.

Rate if Useful 🙂

Sharing knowledge makes you Immortal.

Regards,

Many thanks Edward.So with my

andrea.meconi — Thu, 22 May 2014 20:11:18 GMT

Many thanks Edward.

So with my deployment, where the primary server also functions as a log collector and only one secondary is present, I need to deactivated the secondary and delete it from instances table.

With two primary, and standalone server, I can upgrade it independently. When upgrade is done I can register to primary again.

Regards.

Andrea

After the upgrade, runtime

andrea.meconi — Fri, 23 May 2014 04:38:01 GMT

After the upgrade, runtime process is not monitored and application doesn't respond.

Any ideas?

Thanks.

acsdue/admin# application upgrade ACS_5.5.0.46.tar.gz patch
Do you want to save the current configuration ? (yes/no) [yes] ?
Generating configuration...
Saved the running configuration to startup successfully

% CARS Install application required post install reboot...

Broadcast message from root (pts/0) (Fri May 23 00:04:11 2014):

The system is going down for reboot NOW!

Application upgrade successful
acsdue/admin#
Connection was reset.

Last login: Thu May 22 23:13:39 2014 from 10.164.0.1
Copyright(c) 2013 Cisco Systems, Inc. All rights Reserved

acsdue/admin#
acsdue/admin# sh app status acs

ACS role: PRIMARY

Process 'database'                  running
Process 'management'                running
Process 'runtime'                   not monitored
Process 'ntpd'                      running
Process 'view-database'             running
Process 'view-jobmanager'           running
Process 'view-alertmanager'         running
Process 'view-collector'            running
Process 'view-logprocessor'         running

acsdue/admin#

Andrea,The procedure you

edwardcollins7 — Fri, 23 May 2014 05:51:16 GMT

Andrea,

The procedure you followed was correct.

Did you try starting the service manually?

"acs start runtime"

Rate if Useful 🙂

Sharing knowledge makes you Immortal.

Regards,

Yes Edward.I try starting it

andrea.meconi — Fri, 23 May 2014 08:54:22 GMT

Yes Edward.

I try starting it manually also.

Reimage maybe the solution....

Thanks.

Andrea

Re-image might be an option,

edwardcollins7 — Fri, 23 May 2014 12:53:57 GMT

Re-image might be an option, you might have a backup from the older version.

If you wish to share the logs, I can have a look at it.

Regards

Edward

Edward,I'm trying to reimage.

andrea.meconi — Mon, 26 May 2014 14:15:52 GMT

Edward,

I'm trying to reimage. After a successfully restore, application is initializing... forever....

acsuno/admin# sh app sta acs

Application initializing...
Status is not yet available.
Please check again in a minute.

acsuno/admin#

Any ideas?

Regards.

Which version is the backup

edwardcollins7 — Mon, 26 May 2014 14:22:51 GMT

Which version is the backup from?

What version are you at now?

Regards

Backup comes from 5.3, patch

andrea.meconi — Mon, 26 May 2014 15:01:32 GMT

Backup comes from 5.3, patch 9 and Pointed-PreUpgrade.

Restore to 5.5, fresh installation without any patches.

acsuno/admin# sh ver

Cisco Application Deployment Engine OS Release: 2.1
ADE-OS Build Version: 2.1.1.129
ADE-OS System Architecture: i386

Version information of installed applications
---------------------------------------------

Cisco ACS VERSION INFORMATION
-----------------------------
Version : 5.5.0.46
Internal Build ID : B.723

acsuno/admin#

After applying restore, there

edwardcollins7 — Tue, 27 May 2014 07:01:27 GMT

After applying restore, there is another encryption patch that you need to get from TAC.

Rate if Useful 🙂

Sharing knowledge makes you Immortal.

Regards,

Sorry Edward, is it related

andrea.meconi — Tue, 27 May 2014 13:49:41 GMT

Sorry Edward, is it related to bug ID CSCum67932? If so, this bug is resolved with cumulative patch 2.

Anyway I apply patch 3 and then restore with success.

When virtual machine reboots, service runtime is not monitored again: this is the same issue I have got with application bundle.

Ideas?

Thanks.

Regards.

Andrea

Ok good, yes that was the

edwardcollins7 — Wed, 28 May 2014 07:07:56 GMT

Ok good, yes that was the defect, Could you share the debugs:

>acs-config

>debug-log runtime level debug

Collect the support file from the ACS and attach it.

http://www.cisco.com/c/en/us/td/docs/net_mgmt/cisco_secure_access_control_system/5-5/user/guide/acsuserguide/viewer_troubleshooting.html#pgfId-1057672

Rate if Useful 🙂

Sharing knowledge makes you Immortal.

Regards,

Edward, many many thanks for

andrea.meconi — Wed, 28 May 2014 20:49:53 GMT

Edward, many many thanks for your help.

I have changed the debug level. Now I'm using CLI to download the bundle because GUI is not available when runtine service doesn't run.

acs support acsuno-debug repository patch encryption-passphrase null include-debug-logs 1

Regards.

Andrea

InfoHandler,28/05/2014,19:51

edwardcollins7 — Thu, 29 May 2014 07:05:54 GMT

InfoHandler,28/05/2014,19:51:39:066,WARN ,3049638800,ActiveDirectoryInfoHandler::startWatching: inotify_add_watch failed for /var/centrifydc, errno = No such file or directory (2),ActiveDirectoryInfoHandler.cpp:115
EventHandler,28/05/2014,19:52:01:514,ERROR,3036089232,caught exception while calling ConfigNotificationFlow: Method MethodCaller<ConfigNotificationFlow, StartConfigNotifFlowEvent>: Enum [Remote-Access=2] already exists,EventStack.cpp:218
Daemon,28/05/2014,19:52:01:514,ERROR,3086341872,rt_daemon init failed,RTDaemon.cpp:325

All the logs are in warn.

Could you turn on debugs and then try to restart the runtime service to trigger the problem again.

Hello Edward.I turn on debug

andrea.meconi — Thu, 29 May 2014 13:09:00 GMT

Hello Edward.

I turn on debug with this command

acsuno/admin# acs-config
Escape character is CNTL/D.

Username:
Password:

Please note that due to an internal error you are working in a Failover Mode.
Configuration changes may not take effect. Please contact support.

acsuno/acsadmin(config-acs)# debug-log runtime level debug
acsuno/acsadmin(config-acs)#

and restart runtime service...

acsuno/admin# acs start runtime
Starting runtime
acsuno/admin# sh app stat acs

ACS role: PRIMARY

Process 'database'                  running
Process 'management'                running
Process 'runtime'                   initializing
Process 'ntpd'                      running
Process 'view-database'             running
Process 'view-jobmanager'           running
Process 'view-alertmanager'         running
Process 'view-collector'            running
Process 'view-logprocessor'         running

acsuno/admin#

When runtime service is in "not monitored" state I collect the bundle with the command

acsuno/admin# acs support acsuno-debug-2905 repository patch encryption-passphrase null include-debug-logs 1
Collecting support information ...(acsuno-debug-2905.tar.gz)
Calculating disk size for /opt/backup/supportbundle-acsuno-debug-2905-1401358751
ACS support file 'acsuno-debug-2905.tar.gz' successfully copied to repository 'patch'
acsuno/admin#

Many thanks.

Regards.

Andrea

Andrea,Luckily, the log file

edwardcollins7 — Fri, 30 May 2014 07:15:07 GMT

Andrea,

Luckily, the log file has the problem captured.

Log snippet:

AcsLogs,29/05/2014,10:17:27:816,DEBUG,3036543888,NIL-CONTEXT,Log_Message=[2014-05-29 10:17:27.816 +00:00 0000000005 31108 FATAL Configuration-Notifications: Cold configuration restart failed, ACSVersion=acs-5.5.0.46-B.723, ], Startup/Shutdown ,MessageFormatter.cpp:94
ConfigNotificationFlow,29/05/2014,10:17:27:816,DEBUG,3036543888,NIL-CONTEXT,ConfigNotificationFlow::signalContolCondition: state=StartFailed,ConfigNotificationFlow.cpp:1107
EventHandler,29/05/2014,10:17:27:816,ERROR,3036543888,caught exception while calling back ConfigNotificationFlow: Method MethodCaller<ConfigNotificationFlow, MBSendEvent>: Enum [Remote-Access=2] already exists,EventStack.cpp:255
ConfigVersionManager,29/05/2014,10:17:27:816,DEBUG,3036543888,VersionManager::decrementVersionRef: version ID=1, ref count=4,VersionManager.cpp:159
ConfigVersionManager,29/05/2014,10:17:27:816,DEBUG,3036543888,VersionManager::decrementVersionRef: version ID=1, ref count=3,VersionManager.cpp:159
ConfigVersionManager,29/05/2014,10:17:27:816,DEBUG,3036543888,VersionManager::decrementVersionRef: version ID=1, ref count=2,VersionManager.cpp:159
ConfigVersionManager,29/05/2014,10:17:27:816,DEBUG,3036543888,VersionManager::decrementVersionRef: version ID=1, ref count=1,VersionManager.cpp:159
EventHandler,29/05/2014,10:17:27:816,DEBUG,3036543888,Stack: 0xb4610b48 Destroy stack,EventStack.cpp:37
Daemon,29/05/2014,10:17:27:816,ERROR,3086796528,rt_daemon init failed,RTDaemon.cpp:325
Daemon,29/05/2014,10:17:27:816,ERROR,3086796528,ConfigNotificationFlow::waitStart: unable to load configuration: fatal error,RTDaemon.cpp:326
Daemon,29/05/2014,10:17:27:816,ERROR,3086796528,rc = 4,RTDaemon.cpp:327

The interesting part is here:

MBSendEvent>: Enum [Remote-Access=2] already exists,EventStack.cpp:255

This is pointing to this attribute:

Dictionary attribute [CVPN3000/ASA/PIX7.x-IPSec-Tunnel-Type] enum registered : [2 - Remote-Access]

Are you using this somewhere?

if so delete it and then try to start the runtime, once started, you can re-configure them.

Now, if you are not using this, you are running into data corruption, only the devs can fix it.

You will have to open a case with TAC, now it would be quicker for you as you already know where the problem lies.

Rate if Useful 🙂

Sharing knowledge makes you Immortal.

Regards,

Many many thanks Edward.I'm

andrea.meconi — Fri, 30 May 2014 08:10:18 GMT

Many many thanks Edward.

I'm using this attribute to configure a VPN concentrator. To delete it I need to reconfigure more profiles.

How can I delete it using CLI, GUI is not available when runtime is down.

Regards.

Andrea