topic --- Primary --- in Network Access Control

Logging and Monitoring Dashboard - Troubleshooting

bryan.hepding1 — Mon, 11 Mar 2019 06:26:49 GMT

We've recently upgraded to ACS 5.8.0.32 with two ACS servers. One is the primary and the secondary is for log collecting. When we click on troubleshooting we are unable to see any live authentications. I've tried to stop and start the logprocessor as well as doing a replace-cleandb command which erased all logs. Any help would be greatly appreciated.

Thanks,

Bryan

From the primary please get

Jatin Katyal — Tue, 02 Feb 2016 14:48:07 GMT

From the primary please get the output of this command: sh acs-logs filename acsLogForward.log | last 80

From the secondary ( log collector) - get me the o/p of acsview show-dbsize

~ Jatin

--- Primary ---

bryan.hepding1 — Tue, 02 Feb 2016 14:56:34 GMT

Thanks for your quick reply, here is the output.

--- Primary ---

Daemon,24/01/2016,15:57:22:%Q,INFO ,0x7ffef6630760,Daemon ctor: process ID=4781,Daemon.cpp:22
Daemon,24/01/2016,15:57:22:%Q,INFO ,0x7ffef6630760,Daemon:init: daemon process ID=4784,Daemon.cpp:98
ConfigManager,24/01/2016,15:57:22:%Q,INFO ,0x7ffef6630760,ConfigManager init success,ConfigManager.cpp:110
ConfigManager,24/01/2016,15:57:24:%Q,INFO ,0x7ffef6630760,ConfigManager::createVersion version id 1,ConfigManager.cpp:917
ConfigManager,24/01/2016,15:57:24:%Q,INFO ,0x7ffef6630760,ConfigManager::activateVersion version id 1 activated,ConfigManager.cpp:944
Daemon,24/01/2016,15:57:24:%Q,INFO ,0x7ffef6630760,Log forwarding is suspended,LogForwardDaemon.cpp:308
HttpWorkerThread,24/01/2016,15:57:24:%Q,INFO ,0x7ffef6630760,HttpWorkerThread started,HttpWorkerThread.cpp:35
HttpHandler,24/01/2016,15:57:24:%Q,INFO ,0x7ffee3770700,HttpHandler::onHttpStartEvent: opened HTTP port 8087; RT Control is available,HttpHandler.cpp:94
Tailor,24/01/2016,15:57:24:%Q,WARN ,0x7ffef6630760,cannot open watermark file /opt/CSCOacs/runtime/config/logforward/watermark: No such file or directory,Tailor.cpp:4
38
Daemon,24/01/2016,15:57:24:%Q,ERROR,0x7ffef6630760,Daemon::createInitFile: failed to create init file. Error: Permission denied,Daemon.cpp:119
Daemon,24/01/2016,15:57:24:%Q,INFO ,0x7ffef6630760,initialized,LogForwardDaemon.cpp:111
Daemon,24/01/2016,15:58:44:%Q,INFO ,0x7ffef6630760,Configured syslog: *.*.*.*:20514,LogForwardDaemon.cpp:380
Daemon,24/01/2016,15:58:44:%Q,INFO ,0x7ffef6630760,Log forwarding is active,LogForwardDaemon.cpp:312
Daemon,24/01/2016,16:16:18:%Q,INFO ,0x7ffef6630760,Configured syslog: *.*.*.*:20514,LogForwardDaemon.cpp:380
Daemon,25/01/2016,08:26:37:%Q,INFO ,0x7f29f14c6760,Daemon ctor: process ID=2709,Daemon.cpp:22
Daemon,25/01/2016,08:26:37:%Q,INFO ,0x7f29f14c6760,Daemon:init: daemon process ID=2722,Daemon.cpp:98
ConfigManager,25/01/2016,08:26:37:%Q,INFO ,0x7f29f14c6760,ConfigManager init success,ConfigManager.cpp:110
ConfigManager,25/01/2016,08:26:39:%Q,INFO ,0x7f29f14c6760,ConfigManager::createVersion version id 1,ConfigManager.cpp:917
ConfigManager,25/01/2016,08:26:39:%Q,INFO ,0x7f29f14c6760,ConfigManager::activateVersion version id 1 activated,ConfigManager.cpp:944
Daemon,25/01/2016,08:26:39:%Q,INFO ,0x7f29f14c6760,Configured syslog: *.*.*.*:20514,LogForwardDaemon.cpp:380
Daemon,25/01/2016,08:26:39:%Q,INFO ,0x7f29f14c6760,Log forwarding is active,LogForwardDaemon.cpp:312
HttpWorkerThread,25/01/2016,08:26:39:%Q,INFO ,0x7f29f14c6760,HttpWorkerThread started,HttpWorkerThread.cpp:35
HttpHandler,25/01/2016,08:26:39:%Q,INFO ,0x7f29de602700,HttpHandler::onHttpStartEvent: opened HTTP port 8087; RT Control is available,HttpHandler.cpp:94
Daemon,25/01/2016,08:26:39:%Q,ERROR,0x7f29f14c6760,Daemon::createInitFile: failed to create init file. Error: Permission denied,Daemon.cpp:119
Daemon,25/01/2016,08:26:39:%Q,INFO ,0x7f29f14c6760,initialized,LogForwardDaemon.cpp:111
Daemon,25/01/2016,08:28:56:%Q,INFO ,0x7ff40d978760,Daemon ctor: process ID=2724,Daemon.cpp:22
Daemon,25/01/2016,08:28:56:%Q,INFO ,0x7ff40d978760,Daemon:init: daemon process ID=2736,Daemon.cpp:98
ConfigManager,25/01/2016,08:28:56:%Q,INFO ,0x7ff40d978760,ConfigManager init success,ConfigManager.cpp:110
ConfigManager,25/01/2016,08:28:58:%Q,INFO ,0x7ff40d978760,ConfigManager::createVersion version id 1,ConfigManager.cpp:917
ConfigManager,25/01/2016,08:28:58:%Q,INFO ,0x7ff40d978760,ConfigManager::activateVersion version id 1 activated,ConfigManager.cpp:944
Daemon,25/01/2016,08:28:58:%Q,INFO ,0x7ff40d978760,Configured syslog: *.*.*.*:20514,LogForwardDaemon.cpp:380
Daemon,25/01/2016,08:28:58:%Q,INFO ,0x7ff40d978760,Log forwarding is active,LogForwardDaemon.cpp:312
HttpWorkerThread,25/01/2016,08:28:58:%Q,INFO ,0x7ff40d978760,HttpWorkerThread started,HttpWorkerThread.cpp:35
HttpHandler,25/01/2016,08:28:58:%Q,INFO ,0x7ff3fabb8700,HttpHandler::onHttpStartEvent: opened HTTP port 8087; RT Control is available,HttpHandler.cpp:94
Daemon,25/01/2016,08:28:58:%Q,ERROR,0x7ff40d978760,Daemon::createInitFile: failed to create init file. Error: Permission denied,Daemon.cpp:119
Daemon,25/01/2016,08:28:58:%Q,INFO ,0x7ff40d978760,initialized,LogForwardDaemon.cpp:111
Daemon,25/01/2016,08:33:17:%Q,INFO ,0x7fca89ed8760,Daemon ctor: process ID=2658,Daemon.cpp:22
Daemon,25/01/2016,08:33:17:%Q,INFO ,0x7fca89ed8760,Daemon:init: daemon process ID=2661,Daemon.cpp:98
ConfigManager,25/01/2016,08:33:17:%Q,INFO ,0x7fca89ed8760,ConfigManager init success,ConfigManager.cpp:110
ConfigManager,25/01/2016,08:33:19:%Q,INFO ,0x7fca89ed8760,ConfigManager::createVersion version id 1,ConfigManager.cpp:917
ConfigManager,25/01/2016,08:33:19:%Q,INFO ,0x7fca89ed8760,ConfigManager::activateVersion version id 1 activated,ConfigManager.cpp:944
Daemon,25/01/2016,08:33:19:%Q,INFO ,0x7fca89ed8760,Configured syslog: *.*.*.*:20514,LogForwardDaemon.cpp:380
Daemon,25/01/2016,08:33:19:%Q,INFO ,0x7fca89ed8760,Log forwarding is active,LogForwardDaemon.cpp:312
HttpWorkerThread,25/01/2016,08:33:19:%Q,INFO ,0x7fca89ed8760,HttpWorkerThread started,HttpWorkerThread.cpp:35
HttpHandler,25/01/2016,08:33:19:%Q,INFO ,0x7fca5301d700,HttpHandler::onHttpStartEvent: opened HTTP port 8087; RT Control is available,HttpHandler.cpp:94
Daemon,25/01/2016,08:33:19:%Q,ERROR,0x7fca89ed8760,Daemon::createInitFile: failed to create init file. Error: Permission denied,Daemon.cpp:119
Daemon,25/01/2016,08:33:19:%Q,INFO ,0x7fca89ed8760,initialized,LogForwardDaemon.cpp:111
Daemon,25/01/2016,08:46:06:%Q,INFO ,0x7f0426f27760,Daemon ctor: process ID=2811,Daemon.cpp:22
Daemon,25/01/2016,08:46:06:%Q,INFO ,0x7f0426f27760,Daemon:init: daemon process ID=2825,Daemon.cpp:98
ConfigManager,25/01/2016,08:46:06:%Q,INFO ,0x7f0426f27760,ConfigManager init success,ConfigManager.cpp:110
ConfigManager,25/01/2016,08:46:08:%Q,INFO ,0x7f0426f27760,ConfigManager::createVersion version id 1,ConfigManager.cpp:917
ConfigManager,25/01/2016,08:46:08:%Q,INFO ,0x7f0426f27760,ConfigManager::activateVersion version id 1 activated,ConfigManager.cpp:944
Daemon,25/01/2016,08:46:08:%Q,INFO ,0x7f0426f27760,Configured syslog: *.*.*.*:20514,LogForwardDaemon.cpp:380
Daemon,25/01/2016,08:46:08:%Q,INFO ,0x7f0426f27760,Log forwarding is active,LogForwardDaemon.cpp:312
HttpWorkerThread,25/01/2016,08:46:08:%Q,INFO ,0x7f0426f27760,HttpWorkerThread started,HttpWorkerThread.cpp:35
HttpHandler,25/01/2016,08:46:08:%Q,INFO ,0x7f0418285700,HttpHandler::onHttpStartEvent: opened HTTP port 8087; RT Control is available,HttpHandler.cpp:94
Daemon,25/01/2016,08:46:08:%Q,ERROR,0x7f0426f27760,Daemon::createInitFile: failed to create init file. Error: Permission denied,Daemon.cpp:119
Daemon,25/01/2016,08:46:08:%Q,INFO ,0x7f0426f27760,initialized,LogForwardDaemon.cpp:111
Daemon,01/02/2016,13:59:13:%Q,INFO ,0x7f0426f27760,Configured syslog: *.*.*.*:20514,LogForwardDaemon.cpp:380
Daemon,01/02/2016,14:25:39:%Q,INFO ,0x7f0426f27760,Configured syslog: *.*.*.*:20514,LogForwardDaemon.cpp:380
Daemon,01/02/2016,14:48:36:%Q,INFO ,0x7fc15fc82760,Daemon ctor: process ID=2740,Daemon.cpp:22
Daemon,01/02/2016,14:48:36:%Q,INFO ,0x7fc15fc82760,Daemon:init: daemon process ID=2743,Daemon.cpp:98
ConfigManager,01/02/2016,14:48:36:%Q,INFO ,0x7fc15fc82760,ConfigManager init success,ConfigManager.cpp:110
ConfigManager,01/02/2016,14:48:38:%Q,INFO ,0x7fc15fc82760,ConfigManager::createVersion version id 1,ConfigManager.cpp:917
ConfigManager,01/02/2016,14:48:38:%Q,INFO ,0x7fc15fc82760,ConfigManager::activateVersion version id 1 activated,ConfigManager.cpp:944
Daemon,01/02/2016,14:48:38:%Q,INFO ,0x7fc15fc82760,Configured syslog: *.*.*.*:20514,LogForwardDaemon.cpp:380
Daemon,01/02/2016,14:48:38:%Q,INFO ,0x7fc15fc82760,Log forwarding is active,LogForwardDaemon.cpp:312
HttpWorkerThread,01/02/2016,14:48:38:%Q,INFO ,0x7fc15fc82760,HttpWorkerThread started,HttpWorkerThread.cpp:35
HttpHandler,01/02/2016,14:48:38:%Q,INFO ,0x7fc150dde700,HttpHandler::onHttpStartEvent: opened HTTP port 8087; RT Control is available,HttpHandler.cpp:94
Daemon,01/02/2016,14:48:38:%Q,ERROR,0x7fc15fc82760,Daemon::createInitFile: failed to create init file. Error: Permission denied,Daemon.cpp:119
Daemon,01/02/2016,14:48:38:%Q,INFO ,0x7fc15fc82760,initialized,LogForwardDaemon.cpp:111

--- Secondary ---

     Actual DB Size (bytes) : 69181440
     Actual DB Size (GBs) :0.06
     Physical DB Size (bytes):69230592
     Physical DB Size (GBs) :0.06
     Physical ACSviewlog file Size (GBs) :0

1. I could see an error log

Jatin Katyal — Tue, 02 Feb 2016 15:07:21 GMT

1. I could see an error log forwarding is suspended and then active again. I would still suggest you to check the current status of log forwarding on primary ACS under the log collector configuration.

Daemon,24/01/2016,15:57:24:%Q,INFO ,0x7ffef6630760,Log forwarding is suspended,LogForwardDaemon.cpp:308

Daemon,24/01/2016,15:58:44:%Q,INFO ,0x7ffef6630760,Log forwarding is active

2. Do you see logs in live authentication of primary ACS node if we switch the log collector from secondary to primary ACS? If you see then the next step is to ensure UDP 20514 is open between both the ACS.

~ Jatin

I moved the log collector to

bryan.hepding1 — Tue, 02 Feb 2016 15:27:00 GMT

I moved the log collector to point to the primary and nothing is showing up still. Should I still look to see if the UDP port 20514 is being blocked?

If the primary is also

Jatin Katyal — Tue, 02 Feb 2016 17:55:20 GMT

If the primary is also showing the same behavior you don't need to check the port 20514. What do you see monitoring and reports > ACS reports > AAA protocol.

~ Jatin

AAA Diagnostics - Invalid or

bryan.hepding1 — Wed, 03 Feb 2016 08:41:38 GMT

AAA Diagnostics - Invalid or unexpected EAP payload received. Code - 11500

RADIUS Accounting - No data

RADIUS Authentication - No data

TACACS Accounting - Able to see users logging into switches.

TACACS Authentication - No data

TACACS Authorization - No data

It seems few logging

Jatin Katyal — Wed, 03 Feb 2016 15:02:23 GMT

It seems few logging components are showing logs and other shows nothing. Normally this happens due to issue with acs-viewdb and you've already replaced it. Can you reload the primary ACS and also check opt size again.

~ Jatin

So you would like me to

bryan.hepding1 — Wed, 03 Feb 2016 15:09:31 GMT

So you would like me to restart the primary ACS? I did this yesterday and it fixed nothing. I can try again though if you recommend. If this is the case I need to wait for people to go home, so that I do not disrupt their work.

Still not working.

bryan.hepding1 — Wed, 03 Feb 2016 18:28:10 GMT

Still not working.

     acsview show-dbsize
     Actual DB Size (bytes) : 118243328
     Actual DB Size (GBs) :0.11
     Physical DB Size (bytes):118259712
     Physical DB Size (GBs) :0.11
     Physical ACSviewlog file Size (GBs) :0.02

Can you assist me with

bryan.hepding1 — Thu, 04 Feb 2016 06:05:49 GMT

Can you assist me with anything else to solve this problem?

Jatin, did you give up on me

bryan.hepding1 — Fri, 05 Feb 2016 11:04:35 GMT

Jatin, did you give up on me ? lol

No way - Just too busy. What

Jatin Katyal — Fri, 05 Feb 2016 14:45:19 GMT

No way - Just too busy. What is your email address?

~ Jatin

Email here

bryan.hepding1 — Fri, 05 Feb 2016 18:13:33 GMT

email address here

Any updates? We are still

bryan.hepding1 — Thu, 11 Feb 2016 07:18:29 GMT

Any updates? We are still experiencing this issue, Jatin.