The ncs-java-vm.log file disappeared

[yuwa4@cisco.com]

Hi All,

    After i system-install a new NSO, several days later, the ncs-java-vm.log file disappeared itself until I restart ncs.

    Does anybody suffer this issue?

    Thank you!

[yfherzog]

Hi,

Did it disappear or just stopped logging (is the file still there but not updating, or has it been deleted altogether).

In any case, My first suspect will be logrotate.

In some cases (sometime it has to do with conflicts with SELinux), once the logs are rotated (probably once a week with default configs), NSO logging is not resumed until you either restart NSO or reload NSO (ncs --reload).

You can confirm that by changing the logrotate configurations to run more frequently (e.g. hourly, daily), and confirm that this is happening when logs are being rotated.

In many cases, adding copytruncate to the ncs logrotate configurations file solves the issue.

Yftach

[adam.allen]

I've seen a case where the python-vm.* logs have just had any log's added - but I've not never been able to even start to describe what the conditions might have been. A request packages reload would get the logs back logging. I suspect he java-vm logs would have stopped logging too - but I guess because our service is written with python that's what stood out.

Is the ideal selinux config documented (or some well known gotchas)?

Adam

[yfherzog]

I hadn't seen any other issue with SELinux other than this one, and this is solved by adding the copytruncate option to the logrotate config file, so as far as I'm aware no specific SELinux configurations are required by NSO in most cases.

[yuwa4@cisco.com]

Hi Yftach,

    Really thanks for your help.

    it is logrotate issue exactly.

    i found a command "/opt/ncs/current/bin/ncs_cmd -c reopen_logs" in file /etc/logrotate.d/ncs which function seem like to reopen nso logging system. After i command it menualy, everying became OK again.

    And i suspect the issue maybe that after system logrotated, it is failed in running that command to reopen nso logging system.

[yfherzog]

Glad to hear that's indeed the issue!

From past experience, adding copytruncate to this ncs logrotate config file solves the issue.

Did it work for you? Did you solve it in some other way?

Yftach

[yuwa4@cisco.com]

Hi Yftach,

    I just run command "/opt/ncs/current/bin/ncs_cmd -c reopen_logs", then i think the issue was resolved temporary.

    this command was written in logrotate file, normally, every time the ncs log rotated, this command will be run.

    After running this command, every log file will be empty. if the log file isn't existed, it will create a new one. So i think this command's function cover the copytruncate's somehow.

    But i also think this issue depends on the linux version.

    Mine is CentOS 7.0, and i think its logrotated system doesn't work well.

    if people want to use the logrotate, i think need to write a crontab to run "

 /usr/sbin/logrotate -f /etc/logrotate.conf" or "/opt/ncs/current/bin/ncs_cmd -c reopen_logs" regularly.

".

[xiaoxi.cheng]

We have the following for our logrotate config for NCS log files:

/var/log/ncs/*.log 

{

  ...

  postrotate 

      /opt/ncs/current/bin/ncs_cmd -c reopen_logs

  endscript

}

So you do not need another cron job.

[yuwa4@cisco.com]

Hi Xiaoxi,

    Yes, we have this configuration.

    But this command is run in logrotate system.

    So, at first, the logrotate should work well.

    What's confused me much is that our system can only rotate two or three times automaticly. Then i cannot find fresh log file or compressed log files generated.