NSO: audit.log - a package is "polluting" the log - can we force a specific package to be silent ?

ben_piret · ‎09-29-2020

Hello all,

I have an operational/design question.

We have a package in NSO called dbnet-link. This is a package that to writes the links that we introduce in NSO to our external mysql network database.

This is a flat structure with request/response in the yang model.

We have activated the audit logs in NSO but the specific module described above is “polluting” our audit.log especially when it updates the backpointer (see below for an extract but it is far more long)

Is there a way to forbid a package or an element in the model to send logs into the audit log ?

I read something about tailf:suppess-echo but I am not sure it will do the job ?

Any advice on this ?

Kind regards.

Benoit

<INFO> 29-Sep-2020::13:47:09.501 nsowin ncs[10492]: audit user: mhe/68100 commit thandle 1707099 /dbnet-link:dbnet-link attribute "backpointer" set to "[ /l3vpn:l3vpn[l3vpn:servicename='ABV']/l3vpn:link_type[l3vpn:link_type_name='CABLE']/l3vpn:pelinks[l3vpn:name='C01A4260ABV1.10015'] /l3vpn:l3vpn[l3vpn:servicename='ACB']/l3vpn:link_type[l3vpn:link_type_name='CABLE']/l3vpn:pelinks[l3vpn:name='C01A4690ACB3.10410'] /l3vpn:l3vpn[l3vpn:servicename='ACD']/l3vpn:link_type[l3vpn:link_type_name='CABLE']/l3vpn:pelinks[l3vpn:name='C01A5500ACD9.10700'] /l3vpn:l3vpn[l3vpn:servicename='ACD']/l3vpn:link_type[l3vpn:link_type_name='CABLE']/l3vpn:pelinks[l3vpn:name='C01A5500ACDA.10701'] /l3vpn:l3vpn[l3vpn:servicename='ACD']/l3vpn:link_type[l3vpn:link_type_name='CABLE']/l3vpn:pelinks[l3vpn:name='C01A5500ACDC.10703'] /l3vpn:l3vpn[l3vpn:servicename='ACF']/l3vpn:link_type[l3vpn:link_type_name='CABLE']/l3vpn:pelinks[l3vpn:name='C01A6240ACF2.10917'] /l3vpn:l3vpn[l3vpn:servicename='ACM']/l3vpn:link_type[l3vpn:link_type_name='ETHERNET']/l3vpn:pelinks[l3vpn:name='FE1S1831ACM1.22020002'] /l3vpn:l3vpn[l3vpn:servicename='ACN']/l3vpn:link_type[l3vpn:link_type_name='CABLE']/l3vpn:pelinks[l3vpn:name='C02A5620ACN1.37200000'] /l3vpn:l3vpn[l3vpn:servicename='ACZ'

ben_piret · ‎09-30-2020

Hi all,

in fact what i would need is this :

/ncs-config/cli/show-service-meta-data (boolean) [false]

If 'true', backpointers and refcounts are displayed by default when showing the configuration. The default can be overridden by the pipe flags 'display service-meta' and 'hide service-meta'.

so I tried this in my ncs.conf :

<audit-log>
<enabled>true</enabled>
<file>
<name>${NCS_LOG_DIR}/audit.log</name>
<enabled>true</enabled>
</file>
<show-service-meta-data>
<enabled>false</enabled>
</show-service-meta-data>
</audit-log>

but ok it did not work. Any advise ?

Benoit

vleijon · ‎09-30-2020

It should be false but that will not change the audit log, that is only a change for the cli.

ben_piret · ‎09-30-2020

yes I set it as false and I know it is not for the audit.log but it would be very helpful to have this feature in there as well...

Each time somebody is entering a link into NSO, it produces 175266 chars (for the last link entered but growing with each) for just one audit log line.

As such, it is not so usefull...

If nobody knows how to do that - how could I enter a feature request ?

Benoit

vleijon · ‎09-30-2020

I never crank the audit trace up this far. But what I am guessing is that you turned on audit-log-commit and that you like everything except the attribute messages? As far as I can see there is no way of turning them on independently.

You may try turning off audit-log-commit and turning on audit-network-log and see if that fulfill your logging needs.

If you want to file a feature request you can file a regular ticket.

ben_piret · ‎10-03-2020

yes but when I disable the audit log and with only the network log, I don't have any log for the modules that speak with our external database... So it s'not enough in operational environment.

Benoit

gmuloche · ‎10-08-2020

Hello,

Looking at the shared logs, it looks like all the l3vpn service instances are having a backpointer on your dbnet-link element. It looks like any l3vpn services is seen as an owner of the dbnet-link element - I guess it comes from how you are writing from your service instance to this dbnet package. Is this dbnet-link element a container or a list? What I am heading towards is not a solution for silencing the log (I don’t think it is possible or at least I don’t know how to do so) but there may be an option to look at the way you are writing towards this element (maybe not making the l3vpn service owning the dbnet-link element but rather some sub-elements relevant to it) to prevent having such a long list of backpointers and hence a smaller log.

Regarding your latest answer regarding not having logs - I guess you always have some java or python logs for the package but that they are not enough?

ben_piret · ‎10-08-2020

yes the dbnet-link is an adaptation of this ;https://github.com/hniska/external-id-allocation

it holds a request and response list with all the links that the l3vpn package needs to record towards oiur external database (so almost 100% of our links)

And the presence or absence of an entry within the dbnet-link request/response is used in the code.

So I just need to not have the backpointer changes in the audit.log.

gmuloche · ‎10-14-2020

Hello,

I had a discussion with Hakan who is the owner of the repository. The newest version (and only one maintained) of the repository is here: https://github.com/NSO-developer/external-id-allocation

Following our discussion the YANG model was changed to remove the presence container at the top level of the module (you can see the change here: https://github.com/NSO-developer/external-id-allocation/commit/a1ea7cf007a8bed4c424eea952f1c6439140ceb7#diff-085af003eaa27a446d026c3e4540afa8ceed70a5744ce800c071e35ee6d262a9L27)

Indeed the fact that it is a presence container make all the backpointers being inserted at the level of the container when actually it makes more sense that each service is only "owner" (understand insert a backpointer) at the "request" level.

Now while this does not answer your question about the huge log or how to remove the backpointer from there - changing this in your YANG model (provided you test that it works in your environment as I don't know how you have adapted the repo) will make the "too many backpointers on one object" situation disappear.

ben_piret · ‎10-15-2020

Hello,

okeee.

I tested this in our lab environment (i.e. moved the service-point down at the level of the request and removing the "presence true" at the container level) and it seems indeed that the backpointer logs are not inserted anymore in the audit.log - I will do some further regression testing but it looks promising...

container external-id-allocation {

container create-kickers {

}

list request {

uses ncs:service-data;

ncs:servicepoint external-id-allocation-servicepoint;

key name;

leaf name {

type string;

}

ben_piret · ‎10-16-2020

Hi,

in fact it works for my east-west packages (I mean the packages that speak with our internal network databases).

But,... I have a "legacy" package which holds a list of list of list with the service-point at the top level list (needed) and I have the same problem.

See the log attached (line 28 and 59 - 2*30k CHARS for just one line log) - so at the end although the solution proposed works fine for some packages, I would still need to either :

- silent a package i.e. this package is not allowed to send logs in the audit.log

- filter the service-meta-data inputs out of the audit.log (function that seems to be available on CLI ?)

Should I open a TAC Service Request as suggested above to request this as a future feature ?

Benoit

ben_piret · ‎11-06-2020

Hi all,

I still don't have a solution so I have opened a SR with the TAC to see how/when/if I could have this feature.

It would be a pity for us to de-activate completely the audit.log in ncs.conf as it brings values (operationnaly, I mean)

Thanks for your support.

Benoit