topic Re: update_netconf fails with error in RADKit Discussions

update_netconf fails with error

mv4 — Mon, 12 Feb 2024 09:21:17 GMT

Hi,

everytime I try to call update_netconf() on any object from inventory, I get the following error:

>>> pe1=service.inventory['pe1']
>>> res = pe1.update_netconf().wait()
>>> res.result.result
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<frozen radkit_client.sync.from_async.autowrap>", line 252, in __get__
File "<frozen radkit_client.sync.from_async.autowrap>", line 80, in new_method
File "<frozen radkit_client.sync.from_async.portal>", line 104, in do_async
File "<frozen radkit_client.sync.from_async.portal>", line 231, in do_async
File "<frozen radkit_client.sync.from_async.portal>", line 219, in do_async_coro
File "<frozen radkit_client.sync.from_async.portal>", line 207, in in_loop
File "<frozen radkit_client.sync.from_async.portal>", line 229, in func_as_coro
File "<frozen radkit_client.sync.from_async.autowrap>", line 78, in in_loop
File "<frozen radkit_client.async_.device>", line 2733, in result
File "<frozen radkit_client.async_.event_loop_thread>", line 53, in result
File "<frozen radkit_client.async_.event_loop_thread>", line 104, in raise_
radkit_common.protocol.device_actions.DeviceActionError: Device action failed: Netconf connection error: Server closed the connection unexpectedly
>>> pe1.netconf
[UNAVAILABLE] <radkit_client.sync.netconf.netconf_api.SingleDeviceNetconfAPI object at 0x120f77dd0>

I tried this via direct-rpc as well as via cloud-rpc with the same result. Also tried on other devices in my lab with the same result. Does anyone know where the problem lies and how to fix this?

Re: update_netconf fails with error

Haitham Jaradat — Mon, 12 Feb 2024 10:27:34 GMT

Looking at the error message

DeviceActionError: Device action failed: Netconf connection error: Server closed the connection unexpectedly

It seems to be an issue on the device side, either Netconf is disabled or not configured properly. What kind of device is this? If a platform SME sees this they could help with more insight on how to troubleshoot the issue.

Re: update_netconf fails with error

mv4 — Thu, 15 Feb 2024 14:36:58 GMT

Thanks for quick response.

Platform is ASR9901 running 7.9.21. I tried this also on NCS540 platform running 7.5.2 with the same exact result.

RP/0/RSP0/CPU0:pe_1#show run netconf
netconf agent tty
!

RP/0/RSP0/CPU0:pe_1#show run netconf-yang agent
netconf-yang agent
ssh
!

There are some netconf related messages in service.log:

MainThread DirectNetconfRunnerConnection(0x7FC58CB056D0) [NETCONF] downloading schemas from device (deleting local directory if present)
MainThread DirectNetconfRunnerConnection(0x7FC58CB056D0) [NETCONF] can't extract module and revision from capability, ignoring it [capability='urn:ietf:params:netconf:base:1.1']
MainThread DirectNetconfRunnerConnection(0x7FC58CB056D0) [NETCONF] can't extract module and revision from capability, ignoring it [capability='urn:ietf:params:netconf:capability:candidate:1.0']
MainThread DirectNetconfRunnerConnection(0x7FC58CB056D0) [NETCONF] can't extract module and revision from capability, ignoring it [capability='urn:ietf:params:netconf:capability:rollback-on-error:1.0']
MainThread DirectNetconfRunnerConnection(0x7FC58CB056D0) [NETCONF] can't extract module and revision from capability, ignoring it [capability='urn:ietf:params:netconf:capability:validate:1.1']
MainThread DirectNetconfRunnerConnection(0x7FC58CB056D0) [NETCONF] can't extract module and revision from capability, ignoring it [capability='urn:ietf:params:netconf:capability:confirmed-commit:1.1']
MainThread DirectNetconfRunnerConnection(0x7FC58CB056D0) [NETCONF] can't extract module and revision from capability, ignoring it [capability='urn:ietf:params:netconf:capability:notification:1.0']
MainThread DirectNetconfRunnerConnection(0x7FC58CB056D0) [NETCONF] can't extract module and revision from capability, ignoring it [capability='urn:ietf:params:netconf:capability:interleave:1.0']
MainThread DeviceActionResponder(0x7FC58EF22F50) [] device request finished
MainThread RPCServer(0x7FC59383A050) [AUDIT,RPC] RPC request finished [rpc_name='get-netconf-capabilities' identity='superadmin' connection_type='DIRECT']

Re: update_netconf fails with error

Haitham Jaradat — Mon, 12 Feb 2024 12:15:49 GMT

Checked the code, and this error will be raised when parsing fails for any reason. it's not the most accurate error message. I will file a request to make it more clear and show the data that failed parsing, at least that will help with the troubleshooting.

Back to the issue, what version of RADKit are you using? (client and service). I'll try and test this locally and see what I get.

Re: update_netconf fails with error

mv4 — Mon, 12 Feb 2024 12:17:13 GMT

Both client and service are 1.6.4 (latest as far as I know).

Re: update_netconf fails with error

Haitham Jaradat — Mon, 12 Feb 2024 13:40:14 GMT

Thanks for the information. You are correct, at the moment 1.6.4 is the latest version. Will test locally and come back to you soon (48 hours max).

Re: update_netconf fails with error

Haitham Jaradat — Mon, 12 Feb 2024 13:49:52 GMT

I almost forgot. In the service logs, you should see a pointer to the detailed log file for that NETCONF session something like
`2024-02-12T13:43:03.780Z INFO | forwarder/hajarada@cisco.com/wHMDEFka | MainThread radkit_service.session_log [] Session log initialized [filepath='/radkit/session_logs/service/20240212-134303-wHMDEFka-NETCONF-c8000v-12.log']`

Can you check that out and see if there is anything there pointing to connectivity or authentication issues?

Re: update_netconf fails with error

mv4 — Mon, 12 Feb 2024 13:59:13 GMT

Yes I already checked those log files. There is nothing indicating any failure or misbehaviour. It just shows the "hello" xml tag with capabilities listing sent from router. It looks like trace log of netconf xml protocol. Couple of START READ and END READ sections (log file attached).

Re: update_netconf fails with error

mv4 — Wed, 21 Feb 2024 11:24:44 GMT

Just curious if you had any chance to look into this?

Re: update_netconf fails with error

mv4 — Tue, 05 Mar 2024 09:17:21 GMT

Just updated to 1.6.5 (both service and client) and the result is the same:

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<frozen radkit_client.sync.from_async.autowrap>", line 252, in __get__
File "<frozen radkit_client.sync.from_async.autowrap>", line 80, in new_method
File "<frozen radkit_client.sync.from_async.portal>", line 104, in do_async
File "<frozen radkit_client.sync.from_async.portal>", line 231, in do_async
File "<frozen radkit_client.sync.from_async.portal>", line 219, in do_async_coro
File "<frozen radkit_client.sync.from_async.portal>", line 207, in in_loop
File "<frozen radkit_client.sync.from_async.portal>", line 229, in func_as_coro
File "<frozen radkit_client.sync.from_async.autowrap>", line 78, in in_loop
File "<frozen radkit_client.async_.device>", line 2733, in result
File "<frozen radkit_client.async_.event_loop_thread>", line 53, in result
File "<frozen radkit_client.async_.event_loop_thread>", line 104, in raise_
radkit_common.protocol.device_actions.DeviceActionError: Device action failed: Netconf connection error: Server closed the connection unexpectedly

Re: update_netconf fails with error

mv4 — Tue, 26 Mar 2024 07:04:11 GMT

Any news?

Re: update_netconf fails with error

mv4 — Tue, 02 Apr 2024 04:55:25 GMT

I get the same traceback even after updating (both service and client) to latest version 1.6.6:

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<frozen radkit_client.sync.from_async.autowrap>", line 250, in __get__
File "<frozen radkit_client.sync.from_async.autowrap>", line 80, in new_method
File "<frozen radkit_client.sync.from_async.portal>", line 106, in do_async
File "<frozen radkit_client.sync.from_async.portal>", line 292, in do_async
File "<frozen radkit_client.sync.from_async.portal>", line 233, in do_async_coro
File "<frozen radkit_client.sync.from_async.portal>", line 221, in in_loop
File "<frozen radkit_client.sync.from_async.portal>", line 290, in func_as_coro
File "<frozen radkit_client.sync.from_async.autowrap>", line 78, in in_loop
File "<frozen radkit_client.async_.device>", line 2733, in result
File "<frozen radkit_client.async_.event_loop_thread>", line 54, in result
File "<frozen radkit_client.async_.event_loop_thread>", line 105, in raise_
radkit_common.protocol.device_actions.DeviceActionError: Device action failed: Netconf connection error: Server closed the connection unexpectedly

It is unfortunate that nobody from cisco is interested in this anymore. RadKit seemed to be quite usable product, but without working netconf, it is not anymore.

Re: update_netconf fails with error

Rana Khalil — Tue, 02 Apr 2024 12:03:06 GMT

Dear @mv4

Apologies for any inconveniences.

This is Rana Khalil from RADKit support team, I will be looking into your issue with my team.

Since you already upgraded to 1.6.6, please share with us the radkit support package after reproducing the issue.

The support package can be capture from a CLI tool called radkit-control and needs to be ran from the same machine as the RADKit Service.

The command syntax is:

radkit-control system support-package --output radkit-support-package.json

Please let me know if you need assistance with that.

Re: update_netconf fails with error

mv4 — Wed, 03 Apr 2024 06:42:02 GMT

Hi,

I successfully generated the support package. However I'm not very comfortable to post it on the public forum. Any possibility to ship it directly?

Re: update_netconf fails with error

Rana Khalil — Wed, 03 Apr 2024 07:23:04 GMT

Hello @mv4 ,

Please feel free to send it over to my email rankhali@cisco.com

Regards,

Rana