cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Announcements
736
Views
0
Helpful
6
Replies
khgrant
Cisco Employee

NSO stops logging and starts after ncs_cmd -c reopen_logs

 

Hello NSO Team,

 

We have two NSO nodes in production with HA enabled with manual failover, installed with "--run-as-user admin".

 

We notice the master NSO stopped logging. And, it could start logging again after executing the "/opt/ncs/current/bin/ncs_cmd -c reopen_logs" command as follows:

 

[admin@mivic10s ncs]$ ls -altr

total 5564

-rw-r--r--.  1 admin ncsadmin       0 Feb 22 11:06 snmp.log

-rw-r--r--.  1 admin ncsadmin       0 Feb 22 11:06 netconf.log

-rw-r--r--.  1 admin ncsadmin       0 Feb 22 11:06 ncs-python-vm.log

-rw-r--r--.  1 admin ncsadmin       0 Feb 22 11:06 webui-browser.log

-rw-r--r--.  1 admin ncsadmin      13 Feb 24 12:43 ncserr.log.siz

-rw-r--r--.  1 admin ncsadmin   16809 Feb 28 03:37 audit.log-20160228.gz

-rw-r--r--.  1 admin ncsadmin    4904 Feb 28 03:37 devel.log-20160228.gz

-rw-r--r--.  1 admin ncsadmin   18052 Feb 28 03:37 ncs-java-vm.log-20160228.gz

-rw-r--r--.  1 admin ncsadmin    7526 Feb 28 03:37 ncs.log-20160228.gz

-rw-r--r--.  1 admin ncsadmin   50034 Feb 28 03:37 172.22.9.137:8080.access-20160228.gz

-rw-r--r--.  1 admin ncsadmin  159360 Feb 28 03:37 xpath.trace-20160228.gz

-rw-r--r--.  1 admin ncsadmin   46983 Mar  1 18:54 ncserr.log.1

-rw-r--r--.  1 admin ncsadmin 1491043 Mar  1 18:54 ncserr.log.2

-rw-r--r--.  1 admin ncsadmin   45504 Mar  1 20:05 ncserr.log.3

-rw-r--r--.  1 admin ncsadmin      42 Mar  1 20:05 ncserr.log.idx

-rw-r--r--.  1 admin ncsadmin 3085083 Mar  1 20:05 ncserr.log.4

-rw-r--r--.  1 admin ncsadmin       0 Mar  4 09:59 ncs.log

-rw-r--r--.  1 admin ncsadmin  151945 Mar  4 17:18 ncserr.log.5

-rw-r--r--.  1 admin ncsadmin     753 Mar  6 03:17 devel.log-20160306.gz

-rw-r--r--.  1 admin ncsadmin   21023 Mar  6 03:17 audit.log-20160306.gz

-rw-r--r--.  1 admin ncsadmin    7588 Mar  6 03:17 ncs-java-vm.log-20160306.gz

-rw-r--r--.  1 admin ncsadmin   53726 Mar  6 03:17 172.22.9.137:8080.access-20160306.gz

-rw-r--r--.  1 admin ncsadmin  476288 Mar  6 03:17 xpath.trace-20160306.gz

drwxr-xr-x. 15 root  root        4096 Mar  6 03:17 ..

drwxr-xr-x.  2 admin ncsadmin    4096 Mar  6 03:17 .

[admin@mivic10s ncs]$ /opt/ncs/current/bin/ncs_cmd -c reopen_logs

CMD_MAAPI is true [mtid = 0]

[admin@mivic10s ncs]$ ls -altr

total 5568

-rw-r--r--.  1 admin ncsadmin       0 Feb 22 11:06 snmp.log

-rw-r--r--.  1 admin ncsadmin       0 Feb 22 11:06 netconf.log

-rw-r--r--.  1 admin ncsadmin       0 Feb 22 11:06 ncs-python-vm.log

-rw-r--r--.  1 admin ncsadmin       0 Feb 22 11:06 webui-browser.log

-rw-r--r--.  1 admin ncsadmin      13 Feb 24 12:43 ncserr.log.siz

-rw-r--r--.  1 admin ncsadmin   16809 Feb 28 03:37 audit.log-20160228.gz

-rw-r--r--.  1 admin ncsadmin    4904 Feb 28 03:37 devel.log-20160228.gz

-rw-r--r--.  1 admin ncsadmin   18052 Feb 28 03:37 ncs-java-vm.log-20160228.gz

-rw-r--r--.  1 admin ncsadmin    7526 Feb 28 03:37 ncs.log-20160228.gz

-rw-r--r--.  1 admin ncsadmin   50034 Feb 28 03:37 172.22.9.137:8080.access-20160228.gz

-rw-r--r--.  1 admin ncsadmin  159360 Feb 28 03:37 xpath.trace-20160228.gz

-rw-r--r--.  1 admin ncsadmin   46983 Mar  1 18:54 ncserr.log.1

-rw-r--r--.  1 admin ncsadmin 1491043 Mar  1 18:54 ncserr.log.2

-rw-r--r--.  1 admin ncsadmin   45504 Mar  1 20:05 ncserr.log.3

-rw-r--r--.  1 admin ncsadmin      42 Mar  1 20:05 ncserr.log.idx

-rw-r--r--.  1 admin ncsadmin 3085083 Mar  1 20:05 ncserr.log.4

-rw-r--r--.  1 admin ncsadmin       0 Mar  4 09:59 ncs.log

-rw-r--r--.  1 admin ncsadmin  151945 Mar  4 17:18 ncserr.log.5

-rw-r--r--.  1 admin ncsadmin     753 Mar  6 03:17 devel.log-20160306.gz

-rw-r--r--.  1 admin ncsadmin   21023 Mar  6 03:17 audit.log-20160306.gz

-rw-r--r--.  1 admin ncsadmin    7588 Mar  6 03:17 ncs-java-vm.log-20160306.gz

-rw-r--r--.  1 admin ncsadmin   53726 Mar  6 03:17 172.22.9.137:8080.access-20160306.gz

-rw-r--r--.  1 admin ncsadmin  476288 Mar  6 03:17 xpath.trace-20160306.gz

drwxr-xr-x. 15 root  root        4096 Mar  6 03:17 ..

-rw-r--r--.  1 admin ncsadmin       0 Mar  9 13:35 devel.log

-rw-r--r--.  1 admin ncsadmin       0 Mar  9 13:35 xpath.trace

-rw-r--r--.  1 admin ncsadmin       0 Mar  9 13:35 172.22.9.137:8080.access

-rw-r--r--.  1 admin ncsadmin       0 Mar  9 13:35 ncs-java-vm.log

-rw-r--r--.  1 admin ncsadmin     583 Mar  9 13:37 audit.log

drwxr-xr-x.  2 admin ncsadmin    4096 Mar  9 13:37 .

[admin@mivic10s ncs]$

 

We did not change the logrotate configuration from the one coming in with nso installation and you can see it was compressing the files after rotating with dateeext. However, we suspected if the postrorate script might not have executed properly.

 

But with all the same configs, the other node is logging as expected and logrotate is working as expected.

 

 

Has anyone seen a similar issue?

 

 

Regards,

 

-Fatih

 

1 ACCEPTED SOLUTION

Accepted Solutions
khgrant
Cisco Employee

 

Hi Team,

 

 

I can confirm that below was issue was due to SELinux on the Linux. So, logrotate has been working fine after disabling it.

 

 

Regards,

 

-Fatih

 

View solution in original post

6 REPLIES 6
khgrant
Cisco Employee

 

Sounds as a bug, file a proper report to NSO team (how ???) and be sure to include a debugdump (after the reopen_logs cmd was executed)

 

 

/klacke

 

 

khgrant
Cisco Employee

 

>

 

>Sounds as a bug, file a proper report to NSO team (how ???) and be sure

 

>to include a debugdump (after the reopen_logs cmd was executed)

 

 

(Roque) In this case it is a paid customer and Fatih can issue a ticket via RT/TAC.

 

 

Regards,

 

Roque

 

khgrant
Cisco Employee

 

Hello Klacke,

 

 

We are suspecting SELinux might have caused this. I will update once we confirm.

 

Thanks for reply.

 

 

Regards,

 

-Fatih

 

khgrant
Cisco Employee

 

Hi Team,

 

 

I can confirm that below was issue was due to SELinux on the Linux. So, logrotate has been working fine after disabling it.

 

 

Regards,

 

-Fatih

 

View solution in original post

khgrant
Cisco Employee

 

NSO logging should work fine with standard security measures in place. We should expect SELinux to be required in many environments.

 

 

Are there changes that could be made to SELinux to enable this to work?

 

khgrant
Cisco Employee

 

Hi Chris,

 

 

We observed the following logs:

 

[root@mivic10s audit]# grep -i logrotate audit.log* audit.log.1:type=AVC msg=audit(1456627021.689:9648): avc:  denied { execute } for  pid=27968 comm="sh" name="ncs_cmd" dev=dm-3 ino=2621463

 

scontext=system_u:system_r:logrotate_t:s0-s0:c0.c1023

 

tcontext=unconfined_u:object_r:usr_t:s0 tclass=file audit.log.1:type=SYSCALL msg=audit(1456627021.689:9648): arch=c000003e

 

syscall=59 success=no exit=-13 a0=1d0a360 a1=1d08cf0 a2=1d08e60 a3=20

 

items=0 ppid=27967 pid=27968 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0

 

egid=0 sgid=0 fsgid=0 tty=(none) ses=1315 comm="sh" exe="/bin/bash"

 

subj=system_u:system_r:logrotate_t:s0-s0:c0.c1023 key=(null) audit.log.1:type=AVC msg=audit(1456627021.690:9649): avc:  denied { execute } for  pid=27968 comm="sh" name="ncs_cmd" dev=dm-3 ino=2621463

 

scontext=system_u:system_r:logrotate_t:s0-s0:c0.c1023

 

tcontext=unconfined_u:object_r:usr_t:s0 tclass=file audit.log.1:type=SYSCALL msg=audit(1456627021.690:9649): arch=c000003e

 

syscall=21 success=no exit=-13 a0=1d0a360 a1=1 a2=0 a3=20 items=0

 

ppid=27967 pid=27968 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0

 

sgid=0 fsgid=0 tty=(none) ses=1315 comm="sh" exe="/bin/bash"

 

subj=system_u:system_r:logrotate_t:s0-s0:c0.c1023 key=(null) audit.log.1:type=AVC msg=audit(1457230622.003:18005): avc:  denied { execute } for  pid=29064 comm="sh" name="ncs_cmd" dev=dm-3 ino=2621463

 

scontext=system_u:system_r:logrotate_t:s0-s0:c0.c1023

 

tcontext=unconfined_u:object_r:usr_t:s0 tclass=file audit.log.1:type=SYSCALL msg=audit(1457230622.003:18005): arch=c000003e

 

syscall=59 success=no exit=-13 a0=2202360 a1=2200cf0 a2=2200e60 a3=20

 

items=0 ppid=29063 pid=29064 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0

 

egid=0 sgid=0 fsgid=0 tty=(none) ses=2554 comm="sh" exe="/bin/bash"

 

subj=system_u:system_r:logrotate_t:s0-s0:c0.c1023 key=(null) audit.log.1:type=AVC msg=audit(1457230622.003:18006): avc:  denied { execute } for  pid=29064 comm="sh" name="ncs_cmd" dev=dm-3 ino=2621463

 

scontext=system_u:system_r:logrotate_t:s0-s0:c0.c1023

 

tcontext=unconfined_u:object_r:usr_t:s0 tclass=file audit.log.1:type=SYSCALL msg=audit(1457230622.003:18006): arch=c000003e

 

syscall=21 success=no exit=-13 a0=2202360 a1=1 a2=0 a3=20 items=0

 

ppid=29063 pid=29064 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0

 

sgid=0 fsgid=0 tty=(none) ses=2554 comm="sh" exe="/bin/bash"

 

subj=system_u:system_r:logrotate_t:s0-s0:c0.c1023 key=(null) audit.log.1:type=AVC msg=audit(1457836501.248:25393): avc:  denied { execute } for  pid=25806 comm="sh" name="ncs_cmd" dev=dm-3 ino=2621463

 

scontext=system_u:system_r:logrotate_t:s0-s0:c0.c1023

 

tcontext=unconfined_u:object_r:usr_t:s0 tclass=file audit.log.1:type=SYSCALL msg=audit(1457836501.248:25393): arch=c000003e

 

syscall=59 success=no exit=-13 a0=2272360 a1=2270cf0 a2=2270e60 a3=20

 

items=0 ppid=25805 pid=25806 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0

 

egid=0 sgid=0 fsgid=0 tty=(none) ses=3756 comm="sh" exe="/bin/bash"

 

subj=system_u:system_r:logrotate_t:s0-s0:c0.c1023 key=(null) audit.log.1:type=AVC msg=audit(1457836501.249:25394): avc:  denied { execute } for  pid=25806 comm="sh" name="ncs_cmd" dev=dm-3 ino=2621463

 

scontext=system_u:system_r:logrotate_t:s0-s0:c0.c1023

 

tcontext=unconfined_u:object_r:usr_t:s0 tclass=file audit.log.1:type=SYSCALL msg=audit(1457836501.249:25394): arch=c000003e

 

syscall=21 success=no exit=-13 a0=2272360 a1=1 a2=0 a3=20 items=0

 

ppid=25805 pid=25806 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0

 

sgid=0 fsgid=0 tty=(none) ses=3756 comm="sh" exe="/bin/bash"

 

subj=system_u:system_r:logrotate_t:s0-s0:c0.c1023 key=(null) [root@mivic10s audit]#

 

 

 

Our system-install is using non-root user, admin. I haven't tried, but if logrotate could be executed from admin user, then I would not expect SELinux be a problem for log rotation -due to ncs_cmd denial.

 

 

Regards,

 

-Fatih

 

 

Create
Recognize Your Peers
Content for Community-Ad