Device is locked when using cronjob to checksync all the device managed by NSO

[Chris Wang]

We have a daily job to check-sync all the devices managed, and generate the report to result in that some of the devices are locked in the report. But when I manually check-sync or sync-from the locked devices by CLI, it is OK, and could be sync-from.

Error message is like "ORD10-WXBB-PE01: Device is locked in a check-sync operation by session 290795"

The devices locked are all type of neds: cisco-ios, cisco-iosxr, cisco-nx...

What's the reason of this behavior?

NSO version: 4.6.1

NED:

   cisco-ios pakcage version is 5.9.3
   cisco-asa version is 6.0.4
   cisco-fmc version is 1.0.4
   cisco-iosxr version is 6.6.2.1
   cisco-nx version is 5.6
   citrix-netscaler version is 3.0.23

[joepak]

How is the cronjob configured? Can you provide a sample?

In this case, we should figure out if it's NSO or the cronjob. Since you manually check and it works fine, I assume NSO may not like what is being retrieved from the cronjob.

You said 'some' of the devices are locked, is it random devices each time or the same devices?

[Chris Wang]

cronjob is by calling the NSO northbound API: _operations/check-sync.
The locked devices of the API call "_operations/check-sync" return is random.

[joepak]

Okay so when you state you do manual check-syncs, it is done via NB API? How often is the cronjob ran to perform check-sync? Do the following netconf/localhost888/devel/java-vm logs say anything?

[Chris Wang]

yes, the manual check-sync is done via NB API and we run 1 time a day.

One thing I find is that, we run both service check-sync and device check-sync via NB API, will it impact the device lock?

[joepak]

I honestly can't say.. If both are ran at the same time, the test you can perform is to isolate each task to see if the redundancy of errors/device-lock changes.