update_netconf fails with error

[mv4]

Hi,

everytime I try to call update_netconf() on any object from inventory, I get the following error:

>>> pe1=service.inventory['pe1']
>>> res = pe1.update_netconf().wait()
>>> res.result.result
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<frozen radkit_client.sync.from_async.autowrap>", line 252, in __get__
File "<frozen radkit_client.sync.from_async.autowrap>", line 80, in new_method
File "<frozen radkit_client.sync.from_async.portal>", line 104, in do_async
File "<frozen radkit_client.sync.from_async.portal>", line 231, in do_async
File "<frozen radkit_client.sync.from_async.portal>", line 219, in do_async_coro
File "<frozen radkit_client.sync.from_async.portal>", line 207, in in_loop
File "<frozen radkit_client.sync.from_async.portal>", line 229, in func_as_coro
File "<frozen radkit_client.sync.from_async.autowrap>", line 78, in in_loop
File "<frozen radkit_client.async_.device>", line 2733, in result
File "<frozen radkit_client.async_.event_loop_thread>", line 53, in result
File "<frozen radkit_client.async_.event_loop_thread>", line 104, in raise_
radkit_common.protocol.device_actions.DeviceActionError: Device action failed: Netconf connection error: Server closed the connection unexpectedly
>>> pe1.netconf
[UNAVAILABLE] <radkit_client.sync.netconf.netconf_api.SingleDeviceNetconfAPI object at 0x120f77dd0>

I tried this via direct-rpc as well as via cloud-rpc with the same result. Also tried on other devices in my lab with the same result. Does anyone know where the problem lies and how to fix this? 

[Haitham Jaradat]

Looking at the error message

DeviceActionError: Device action failed: Netconf connection error: Server closed the connection unexpectedly

It seems to be an issue on the device side, either Netconf is disabled or not configured properly. What kind of device is this? If a platform SME sees this they could help with more insight on how to troubleshoot the issue.

[mv4]

Thanks for quick response.

Platform is ASR9901 running 7.9.21. I tried this also on NCS540 platform running 7.5.2 with the same exact result.

RP/0/RSP0/CPU0:pe_1#show run netconf
netconf agent tty
!

RP/0/RSP0/CPU0:pe_1#show run netconf-yang agent
netconf-yang agent
ssh
!

 There are some netconf related messages in service.log:

MainThread DirectNetconfRunnerConnection(0x7FC58CB056D0) [NETCONF] downloading schemas from device (deleting local directory if present)
MainThread DirectNetconfRunnerConnection(0x7FC58CB056D0) [NETCONF] can't extract module and revision from capability, ignoring it [capability='urn:ietf:params:netconf:base:1.1']
MainThread DirectNetconfRunnerConnection(0x7FC58CB056D0) [NETCONF] can't extract module and revision from capability, ignoring it [capability='urn:ietf:params:netconf:capability:candidate:1.0']
MainThread DirectNetconfRunnerConnection(0x7FC58CB056D0) [NETCONF] can't extract module and revision from capability, ignoring it [capability='urn:ietf:params:netconf:capability:rollback-on-error:1.0']
MainThread DirectNetconfRunnerConnection(0x7FC58CB056D0) [NETCONF] can't extract module and revision from capability, ignoring it [capability='urn:ietf:params:netconf:capability:validate:1.1']
MainThread DirectNetconfRunnerConnection(0x7FC58CB056D0) [NETCONF] can't extract module and revision from capability, ignoring it [capability='urn:ietf:params:netconf:capability:confirmed-commit:1.1']
MainThread DirectNetconfRunnerConnection(0x7FC58CB056D0) [NETCONF] can't extract module and revision from capability, ignoring it [capability='urn:ietf:params:netconf:capability:notification:1.0']
MainThread DirectNetconfRunnerConnection(0x7FC58CB056D0) [NETCONF] can't extract module and revision from capability, ignoring it [capability='urn:ietf:params:netconf:capability:interleave:1.0']
MainThread DeviceActionResponder(0x7FC58EF22F50) [] device request finished
MainThread RPCServer(0x7FC59383A050) [AUDIT,RPC] RPC request finished [rpc_name='get-netconf-capabilities' identity='superadmin' connection_type='DIRECT']

[Haitham Jaradat]

Checked the code, and this error will be raised when parsing fails for any reason. it's not the most accurate error message. I will file a request to make it more clear and show the data that failed parsing, at least that will help with the troubleshooting.

Back to the issue, what version of RADKit are you using? (client and service). I'll try and test this locally and see what I get. 

[mv4]

Both client and service are 1.6.4 (latest as far as I know).

[Haitham Jaradat]

Thanks for the information. You are correct, at the moment 1.6.4 is the latest version. Will test locally and come back to you soon (48 hours max).

[mv4]

Just curious if you had any chance to look into this?

[mv4]

Any news?

[Haitham Jaradat]

I almost forgot. In the service logs, you should see a pointer to the detailed log file for that NETCONF session something like
`2024-02-12T13:43:03.780Z INFO | forwarder/hajarada@cisco.com/wHMDEFka | MainThread radkit_service.session_log [] Session log initialized [filepath='/radkit/session_logs/service/20240212-134303-wHMDEFka-NETCONF-c8000v-12.log']`

Can you check that out and see if there is anything there pointing to connectivity or authentication issues?

[mv4]

Yes I already checked those log files. There is nothing indicating any failure or misbehaviour. It just shows the "hello" xml tag with capabilities listing sent from router. It looks like trace log of netconf xml protocol. Couple of START READ and END READ sections (log file attached).

[mv4]

Just updated to 1.6.5 (both service and client) and the result is the same:

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<frozen radkit_client.sync.from_async.autowrap>", line 252, in __get__
File "<frozen radkit_client.sync.from_async.autowrap>", line 80, in new_method
File "<frozen radkit_client.sync.from_async.portal>", line 104, in do_async
File "<frozen radkit_client.sync.from_async.portal>", line 231, in do_async
File "<frozen radkit_client.sync.from_async.portal>", line 219, in do_async_coro
File "<frozen radkit_client.sync.from_async.portal>", line 207, in in_loop
File "<frozen radkit_client.sync.from_async.portal>", line 229, in func_as_coro
File "<frozen radkit_client.sync.from_async.autowrap>", line 78, in in_loop
File "<frozen radkit_client.async_.device>", line 2733, in result
File "<frozen radkit_client.async_.event_loop_thread>", line 53, in result
File "<frozen radkit_client.async_.event_loop_thread>", line 104, in raise_
radkit_common.protocol.device_actions.DeviceActionError: Device action failed: Netconf connection error: Server closed the connection unexpectedly

[mv4]

I get the same traceback even after updating (both service and client) to latest version 1.6.6:

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<frozen radkit_client.sync.from_async.autowrap>", line 250, in __get__
File "<frozen radkit_client.sync.from_async.autowrap>", line 80, in new_method
File "<frozen radkit_client.sync.from_async.portal>", line 106, in do_async
File "<frozen radkit_client.sync.from_async.portal>", line 292, in do_async
File "<frozen radkit_client.sync.from_async.portal>", line 233, in do_async_coro
File "<frozen radkit_client.sync.from_async.portal>", line 221, in in_loop
File "<frozen radkit_client.sync.from_async.portal>", line 290, in func_as_coro
File "<frozen radkit_client.sync.from_async.autowrap>", line 78, in in_loop
File "<frozen radkit_client.async_.device>", line 2733, in result
File "<frozen radkit_client.async_.event_loop_thread>", line 54, in result
File "<frozen radkit_client.async_.event_loop_thread>", line 105, in raise_
radkit_common.protocol.device_actions.DeviceActionError: Device action failed: Netconf connection error: Server closed the connection unexpectedly

It is unfortunate that nobody from cisco is interested in this anymore. RadKit seemed to be quite usable product, but without working netconf, it is not anymore. 

[Rana Khalil]

Dear @mv4

Apologies for any inconveniences.

This is Rana Khalil from RADKit support team, I will be looking into your issue with my team.

Since you already upgraded to 1.6.6, please share with us the radkit support package after reproducing the issue.

The support package can be capture from a CLI tool called radkit-control and needs to be ran from the same machine as the RADKit Service.

The command syntax is:

radkit-control system support-package --output radkit-support-package.json

Please let me know if you need assistance with that.

[mv4]

Hi,

I successfully generated the support package. However I'm not very comfortable to post it on the public forum. Any possibility to ship it directly?

[Rana Khalil]

Hello @mv4 ,

Please feel free to send it over to my email rankhali@cisco.com

Regards,

Rana