ncs-netsim start returns failure

[sm000x]

Hi, I use NSO-5.7.3 ncs-netsim to create vendor netconf device, but when I issues "ncs-netsim start", I got failure. There is no problem to create simulator with Juniper/Cisco yang.

ncs-netsim create-network /opt/ncs/packages/ncs-5.7.3-rdm8upf-2023.03.01-2023.03.02.1.tar.gz 3 v
DEVICE v0 CREATED
DEVICE v1 CREATED
DEVICE v2 CREATED

However, I got the following error

ncs-netsim start
CDB boot error: Init transaction failed to validate: application communication failure
Hint: Try using staged start using ConfD start-phases or the --ignore-initial-validation option
Daemon died status=10
DEVICE v0 FAIL
CDB boot error: Init transaction failed to validate: application communication failure
Hint: Try using staged start using ConfD start-phases or the --ignore-initial-validation option
Daemon died status=10
DEVICE v1 FAIL
CDB boot error: Init transaction failed to validate: application communication failure
Hint: Try using staged start using ConfD start-phases or the --ignore-initial-validation option
Daemon died status=10
DEVICE v2 FAIL

Does anyone know wht did I do wrong?

THX

Shauh-Wen Ma

[jvikman]

Check the netsim directories for their logs, perhaps there's a `confderr.log`? If so print it with ncs --printlog <path_to_log_file_here>.

[sm000x]

Hi, Jvikman:

Thank you for the replying. I checked the logs: netsim/upf/upf0/logs, there is
ls -ltr
total 28
-rw-r--r--. 1 root root 0 Mar 23 08:36 audit.log
-rw-r--r--. 1 root root 0 Mar 23 08:36 netconf.log
-rw-r--r--. 1 root root 0 Mar 23 08:36 snmp.log
-rw-r--r--. 1 root root 13 Mar 23 08:36 confderr.log.siz
-rw-r--r--. 1 root root 18 Mar 23 08:36 confderr.log.idx
-rw-r--r--. 1 root root 8 Mar 23 08:36 confderr.log.1
-rw-r--r--. 1 root root 995 Mar 23 08:36 devel.log
-rw-r--r--. 1 root root 11220 Mar 23 08:36 confd.log

"ncs --printlog confderr.log.1" displays nothing. I tail -f the devel.log and confd.log, but I do not understand the messages.

tail -f devel.log
23-Mar-2023::08:36:34.878 mtnjdslncs06 confd[<0.69.0>]: confd embedded apps in early_phase0:
[]
23-Mar-2023::08:36:35.148 mtnjdslncs06 confd[<0.97.0>]: confd mmap_schema handle_info got msg: timeout
23-Mar-2023::08:36:35.169 mtnjdslncs06 confd[<0.9.0>]: confd embedded apps in phase0:
[]
23-Mar-2023::08:36:35.297 mtnjdslncs06 confd[<0.133.0>]: devel-cdb Loaded oper data from ./cdb/O.cdb (0 bytes data in 0.000s)
23-Mar-2023::08:36:35.358 mtnjdslncs06 confd[<0.135.0>]: devel-cdb Loaded configuration from ./cdb/A.cdb (0 bytes data, 0 transactions in 0.000s)
23-Mar-2023::08:36:35.379 mtnjdslncs06 confd[<0.9.0>]: devel-cdb init files found in ./cdb: aaa_init.xml
23-Mar-2023::08:36:35.415 mtnjdslncs06 confd[<0.69.0>]: confd Set running datastore status to valid
23-Mar-2023::08:36:35.447 mtnjdslncs06 confd[<0.145.0>]: devel-c no registration found for callpoint appplicationRestartWarning/validate of type=external path /PFCP:no-pfcp

tail -f confd.log
23-Mar-2023::08:36:34.952 mtnjdslncs06 confd[35052]: - Loading file ./cna-li-x1.fxs
23-Mar-2023::08:36:34.953 mtnjdslncs06 confd[35052]: - Loading file ./cna-interface-mgr.fxs
23-Mar-2023::08:36:34.955 mtnjdslncs06 confd[35052]: - Loading file ./cna-cfgmgr.fxs
23-Mar-2023::08:36:34.955 mtnjdslncs06 confd[35052]: - Loading file ./bgpmgr.fxs
23-Mar-2023::08:36:34.956 mtnjdslncs06 confd[35052]: - Skipping file ./ietf-restconf.fxs: namespace already loaded
23-Mar-2023::08:36:35.110 mtnjdslncs06 confd[35052]: - Starting to listen for Internal IPC on 127.0.0.1:5010
23-Mar-2023::08:36:35.379 mtnjdslncs06 confd[35052]: - CDB load: processing file: ./cdb/aaa_init.xml
23-Mar-2023::08:36:35.447 mtnjdslncs06 confd[35052]: - no registration found for callpoint appplicationRestartWarning/validate of type=external
23-Mar-2023::08:36:35.458 mtnjdslncs06 confd[35052]: - CDB boot error: Init transaction failed to validate: application communication failure
Hint: Try using staged start using ConfD start-phases or the --ignore-initial-validation option

Thank you very much
sm000x

[jvikman]

Ok, I see the problem now. It was actually there in the 1st errors you showed. There should be a validation application starting with the netsim devices, but that validation application doesn't start for some reason:

23-Mar-2023::08:36:35.447 mtnjdslncs06 confd[<0.145.0>]: devel-c no registration found for callpoint appplicationRestartWarning/validate of type=external path /PFCP:no-pfcp

You can also see that in the 1st messages (I should've figured it out from that):

CDB boot error: Init transaction failed to validate: application communication failure

Check that the tarballs is intended for your architecture or see if you can extract the packages and rebuild them yourself?

[sm000x]

Hi, Jvikman:

I built the package myself. I still have the same issue.

Thank you
sm000x

[jvikman]

Does the netsim have a validation application?

What do you get if you grep for the validation point:

grep -RHin -A 2 applicationRestartWarning *

[sm000x]

Hi, Jvikman:

Sorry for the delay. Somehow I missed your reply.

I did this:

cd rdm8upf/src/yang

pwd
/opt/ncs/packages/rdm8upf/src/yang

grep -RHin -A 2 applicationRestartWarning *

And I got nothing back.

THX

sm000x

[jvikman]

Do a grep higher up, maybe there are pre-compiled fxs:s?

You can also look for 

/PFCP:no-pfcp 

[sm000x]

Hi, Jvikman:

Thank you for the suggestion.

This time I found several "tailf:validate appplicationRestartWarning".
tailf:validate appplicationRestartWarning
{
tailf:dependency "../provision-admin-state";
tailf:opaque "Warning: provision-admin-state must be disabled.";
}
}

Somehow I did not find it last time, sorry about that.

Should I comment out the appplicationRestartWarning ?

Thank you
Sm000x

[jvikman · Accepted solution]

Hi Sm00x,

Great, the package with that tailf:validate statement should have an application that registers a callback for that validation point, but for some reason it fails to do so. If it's a python package you should have a ncs-python-vm-<package>.log file where a possible problem could be logged.

BR,

Johan

[sm000x]

Hi, Johan:

Thank you for the information. The package is java package, however, I do not fi
nd any java-vm-<package>.log file, either in netsim directory or in NSO log dire
ctory.

However, I took out all the "tailf:validate appplicationRestartWarning" statemen
ts in the package and recompiled again. This time the netsim starts fine.

Thank you so much for your help. This is really a helpful information.

sm000x