07-27-2022 06:45 AM - edited 07-27-2022 06:56 AM
Hi.
I have an interesting problem. I have allocation in resource-manager id-manager that was done by a service instance that is long gone, yet somehow allocation is still there.
admin@2532e48bb2a1# show running-config resource-pools id-pool POOL allocation DUMMY_340
resource-pools id-pool POOL
allocation DUMMY_340
username bot
allocating-service --ERROR--
!
!
Issue is that this allocation doesn't want to get deleted in any nice way. CLI or such. Via APIs it doesn't budge either:
with ncs.maapi.single_write_trans("admin", "system", db=ncs.OPERATIONAL) as th:
id_pool_l = ncs.maagic.get_root(th).ralloc__resource_pools.id_pool
del id_pool_l['POOL'].allocation['DUMMY_340']
th.apply()
will cause:
Traceback (most recent call last):
File "/root/pool-error/delete.py", line 23, in <module>
del id_pool_l['POOL'].allocation['DUMMY_340']
File "/opt/ncs/current/src/ncs/pyapi/ncs/maagic.py", line 1108, in __delitem__
self._backend._delete(self._path + keystr)
File "/opt/ncs/current/src/ncs/pyapi/ncs/maagic.py", line 130, in _delete
self.delete(path)
File "/opt/ncs/current/src/ncs/pyapi/ncs/maapi.py", line 1202, in proxy
return real(self2.maapi.msock, self2.th, *args, **kwargs)
_ncs.error.Error: item is not writable (4):
I am kinda hoping for a low lever CDB API as a last grasping for straws but it seems I'm not able to figure out the path:
sock_cdb = socket.socket()
cdb.connect(sock_cdb, type=cdb.DATA_SOCKET, ip='127.0.0.1', port=_ncs.NCS_PORT)
cdb.start_session2(sock_cdb, cdb.OPERATIONAL, cdb.LOCK_WAIT | cdb.LOCK_REQUEST)
path = "/ralloc:resource-pools/id-pool[name='POOL']/allocation[id='DUMMY_340']"
new_value = cdb.get(sock_cdb, path)
cdb.end_session(sock_cdb)
sock_cdb.close()
______________________________________________________________
Traceback (most recent call last):
File "/root/pool-error/delete.py", line 13, in <module>
new_value = cdb.get(sock_cdb, path)
_ncs.error.Error: badly formatted or nonexistent path (8): Bad path </ralloc:resource-pools/id-pool[name='POOL']/allocation[id='DUMMY_340']>
I tried different combinations of paths with prefixes like: "/ralloc:resource-pools/idalloc:id-pool[name='POOL']/allocation[id='DUMMY_340']". At this point I think I would still get writeable error, but now I'm more interested why the path doesn't work.
Any ideas? I'm out of those.
Solved! Go to Solution.
07-27-2022 02:34 PM
He's talking about rollback that can be disabled in ncs.conf.
/ncs-config/rollback/enabled (boolean) [false]
When set to true a rollback file will be created whenever the running configuration is modified.
07-27-2022 09:09 PM
Hi
Yes the rollback feature I am referring to is disabled in ncs.conf, and then run "ncs --reload" from the unix shell.
With regards to you cdp path question it has to be in the "keypath" format not "xpath".
path = "/ralloc:resource-pools/id-pool{POOL}/allocation{DUMMY_340}"
eg. try "show running-config resource-pools id-pool | display keypath"
07-27-2022 12:57 PM
Hi
I have had something similar. And In my case it was solved by temporarily disabling rollback. While rollback was disabled I could do the delete via the CLI.
If this does not help, you might be able to get more information about the issue with the "ncs --printlog ncserr.log.[last updated]"
br.
Kristoffer Larsen
07-27-2022 02:13 PM
Hey.
You meant "services global-settings collect-forward-diff false"? Just tried it, it failed with Aborted: Internal error. The ncserr is not readable.
07-27-2022 02:34 PM
He's talking about rollback that can be disabled in ncs.conf.
/ncs-config/rollback/enabled (boolean) [false]
When set to true a rollback file will be created whenever the running configuration is modified.
07-27-2022 09:09 PM
Hi
Yes the rollback feature I am referring to is disabled in ncs.conf, and then run "ncs --reload" from the unix shell.
With regards to you cdp path question it has to be in the "keypath" format not "xpath".
path = "/ralloc:resource-pools/id-pool{POOL}/allocation{DUMMY_340}"
eg. try "show running-config resource-pools id-pool | display keypath"
07-28-2022 12:44 AM - edited 07-28-2022 12:47 AM
It worked! Thanks! How on earth did you figured that out if you don't mind me asking?
07-28-2022 01:19 AM
I do most of my NSO development in erlang, and run my development instances in the foreground, where stack traces of these internal errors can be seen. And then experience tells me that when data is displayed in the CLI like "--ERROR--" it usually means a stack trace is produced.
You can also see historic stack traces with the "ncs --printlog" eg:
kri@desk:~/ncs-nbp/logs$ ncs --printlog ncserr.log.1
Eshell V10.7.2.9 (abort with ^G)
{"12-Feb-2021::12:26:18.096",
{error_report,<0.1853.0>,
{<0.1857.0>,crash_report,
[[{initial_call,{cli_backend_server,'-start_link/4-fun-0-',[]}},
{pid,<0.1857.0>},
{registered_name,[]},
{error_info,
{error,
{badmatch,<<>>},
[{cli_backend_server,read_init2,5,
[{file,"cli_backend_server.erl"},{line,1129}]},
{cli_backend_server,init,5,
[{file,"cli_backend_server.erl"},{line,1002}]},
{proc_lib,init_p,3,[{file,"proc_lib.erl"},{line,234}]}]}},
{ancestors,[<0.1853.0>,cli_server,cli_sup,<0.522.0>]},
{message_queue_len,1},
{messages,[terminal_input_ping]},
{links,[<0.1853.0>]},
{dictionary,[]},
{trap_exit,false},
{status,running},
{heap_size,376},
{stack_size,27},
{reductions,364}],
[]]}}}
1> 12-Feb-2021::12:29:30.067 <0.1866.0> <0.1998.0> rollback:814: cs_trans_rng:show_delta_rev(): throw:{error,{cs_error,undefined,bad_value,bad_value,undefined,undefined,undefined,{[{{13,666}},8235672,{<<"ANON">>,<<"ANON">>,{10,1}},760089515,[702437083|205751944],1787219742,[734664653|962606743]]},undefined,false}}
[{cs,val2str,7,[{file,"cs.erl"},{line,1856}]},
{cs_trans_rng,qval2str,7,[{file,"cs_trans_rng.erl"},{line,5714}]},
{cs_trans_rng,emit_leaf,7,[{file,"cs_trans_rng.erl"},{line,7870}]},
{cs_trans_rng,get_leaf,7,[{file,"cs_trans_rng.erl"},{line,3573}]},
{lists,foldl,3,[{file,"lists.erl"},{line,1263}]},
{cs_trans_rng,get_cs_delta1,8,[{file,"cs_trans_rng.erl"},{line,6763}]},
{cs_trans_rng,'-show_delta_rev/14-fun-4-',19,
[{file,"cs_trans_rng.erl"},{line,6187}]},
{cs_trans_diff_iterate,sh_diff_iterate_x,10,
[{file,"cs_trans_diff_iterate.erl"},{line,323}]},
{xds_ram,tts_fold_children,7,[{file,"xds_ram.erl"},{line,1259}]},
{cs_trans_diff_iterate,sh_diff_iterate_x,10,
[{file,"cs_trans_diff_iterate.erl"},{line,491}]},
{xds_ram,tts_fold_children,7,[{file,"xds_ram.erl"},{line,1259}]},
{cs_trans_diff_iterate,sh_diff_iterate_x,10,
[{file,"cs_trans_diff_iterate.erl"},{line,499}]},
{xds_ram,tts_fold_children,7,[{file,"xds_ram.erl"},{line,1259}]},
{cs_trans_diff_iterate,sh_diff_iterate_x,10,
[{file,"cs_trans_diff_iterate.erl"},{line,499}]},
{cs_trans_diff_iterate,sh_diff_iterate,7,
[{file,"cs_trans_diff_iterate.erl"},{line,131}]},
{cs_trans_rng,'-show_delta_rev/14-fun-11-',3,
[{file,"cs_trans_rng.erl"},{line,6360}]},
{cs_trans_rng,show_delta_rev,14,[{file,"cs_trans_rng.erl"},{line,6358}]},
{rollback,create_pending2,7,[{file,"rollback.erl"},{line,776}]},
{rollback,create_pending,7,[{file,"rollback.erl"},{line,656}]},
{cs_trans,do_rollback,8,[{file,"cs_trans.erl"},{line,17361}]},
{cs_trans,sh_validate2,13,[{file,"cs_trans.erl"},{line,12451}]},
{cs_trans,sh_validate0,7,[{file,"cs_trans.erl"},{line,12321}]},
{cs_trans,handle_call,3,[{file,"cs_trans.erl"},{line,4714}]},
{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,677}]}]
1>
From this trace example you can see that the rollback feature is trying to convert the internal Erlang representation to a string, but likely due to some yang models being removed it does not know the string representation for all the numeric tag values.
And it all sort of makes sense, if you have a look at some of these rollback files, since they need to contain the reverse of your delete action, and if you can't "see" what data you currently have it is hard to generate the reverse.
07-28-2022 02:20 AM
Thanks a lot, very useful stuff.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide