Good morning. I'm working through getting a netconf ned working for a Ciena 3304 using the NSO in Docker project. I can get everything to compile correctly, netsim works just fine. When I finally get to the point of testing the NED against an actual device, the wheels start to fall off.
I'm currently using NSO 5.3.4.2 in this build. I'll show the output and process with comments throughout. But the TL;DR; is that when I attempt to do a sync-from against the device I get an error of "missing element" and trace log showing "(badxml)".
Relevant output:
admin@ncs# devices device lab-ciena sync-from
result false
info lab-ciena: missing element: priority in /ncs:devices/ncs:device[ncs:name='lab-ciena']/ncs:config/sys:system/ciena-users:authentication/ciena-users:authentication
Enabling trace logging on the device shows the following in the logs:
>>>>out 1-Sep-2021::18:42:04.016 user: admin/49 thandle 95 hostname aee7a56c07bf device lab-ciena session-id=27
<rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"
message-id="2">
<get-config>
<source>
<running/>
</source>
<filter>
<system xmlns="urn:ietf:params:xml:ns:yang:ietf-system"/>
<nacm xmlns="urn:ietf:params:xml:ns:yang:ietf-netconf-acm"/>
</filter>
</get-config>
</rpc>
<<<<in 1-Sep-2021::18:42:04.039 user: admin/49 thandle 95 hostname aee7a56c07bf device lab-ciena session-id=27 (badxml)
<?xml version="1.0" encoding="UTF-8"?><rpc-reply message-id="2" xmlns:ncx="http://netconfcentral.org/ns/yuma-ncx" ncx:last-modified="2000-01-05T22:03:24Z" ncx:etag="315" xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"><data><nacm xmlns="urn:ietf:params:xml:ns:yang:ietf-netconf-acm"><enable-nacm>true</enable-nacm><read-default>permit</read-default><write-default>deny</write-default><exec-default>permit</exec-default><enable-external-groups>true</enable-external-groups></nacm><system xmlns="urn:ietf:params:xml:ns:yang:ietf-system"><contact>Customer Support, Ciena</contact><hostname>Ciena</hostname><location>Not Specified</location><clock><timezone-utc-offset>0</timezone-utc-offset><timestamp xmlns="http://www.ciena.com/ns/yang/ciena-time">local</timestamp></clock><authentication xmlns="http://www.ciena.com/ns/yang/ciena-users"><authentication><method>local</method><priority>1</priority><scope>all</scope></authentication><authentication><method>tacacs</method><priority>2
>>>>out 1-Sep-2021::18:42:04.040 user: admin/49 thandle 95 hostname aee7a56c07bf device lab-ciena session-id=27 NCS closeTake note of the line above that is beginning with "<<<<in" it shows "(badxml)". Looking at the end of the returned XML you can see it's truncated at "<priority>2". I was wondering if maybe the device isn't returning a full document in the response for the config, so I fired up ncclient and ran the same query directly against the device. This shows that the device IS returning the full response:
Python 3.8.10 (default, Jun 2 2021, 10:49:15)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.27.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: from ncclient import manager
In [2]: import xml.dom.minidom
In [3]: with manager.connect(host='*********', port=830, username='****', password='****', hostkey_verify=False) as m:
...: c = m.get_config(source='running', filter='<filter><system xmlns="urn:ietf:params:xml:ns:yang:ietf-system"/><nacm xmlns="urn:ietf:params:xml:ns:yang:ietf-netconf-acm"/></filter>').data_xml
...: x = xml.dom.minidom.parseString(c)
...: print(x.toprettyxml())
...:
<?xml version="1.0" ?>
<data xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0" xmlns:ncx="http://netconfcentral.org/ns/yuma-ncx">
<nacm xmlns="urn:ietf:params:xml:ns:yang:ietf-netconf-acm">
<enable-nacm>true</enable-nacm>
<read-default>permit</read-default>
<write-default>deny</write-default>
<exec-default>permit</exec-default>
<enable-external-groups>true</enable-external-groups>
</nacm>
<system xmlns="urn:ietf:params:xml:ns:yang:ietf-system">
<contact>Customer Support, Ciena</contact>
<hostname>Ciena</hostname>
<location>Not Specified</location>
<clock>
<timezone-utc-offset>0</timezone-utc-offset>
<timestamp xmlns="http://www.ciena.com/ns/yang/ciena-time">local</timestamp>
</clock>
<authentication xmlns="http://www.ciena.com/ns/yang/ciena-users">
<authentication>
<method>local</method>
<priority>1</priority>
<scope>all</scope>
</authentication>
<authentication>
<method>tacacs</method>
<priority>2</priority>
<scope>serial</scope>
</authentication>
<max-admin-users>5</max-admin-users>
<max-limited-users>5</max-limited-users>
<max-super-users>5</max-super-users>
<password-policy>
<disallow-dict-words>off</disallow-dict-words>
<disallow-username>off</disallow-username>
<max-repeated-chars>0</max-repeated-chars>
<min-character-change>0</min-character-change>
<min-length>0</min-length>
<min-lowercase-chars>0</min-lowercase-chars>
<min-numeric-chars>0</min-numeric-chars>
<min-special-chars>0</min-special-chars>
<min-uppercase-chars>0</min-uppercase-chars>
</password-policy>
<user>
<access-level>admin</access-level>
<max-session-limit>0</max-session-limit>
<name>admin</name>
</user>
<user>
<access-level>diag</access-level>
<max-session-limit>0</max-session-limit>
<name>gss</name>
</user>
<user>
<access-level>super</access-level>
<max-session-limit>0</max-session-limit>
<name>netconf</name>
</user>
<user>
<access-level>super</access-level>
<max-session-limit>0</max-session-limit>
<name>su</name>
</user>
<user>
<access-level>limited</access-level>
<max-session-limit>0</max-session-limit>
<name>user</name>
</user>
<username-policy>
<min-length>1</min-length>
</username-policy>
</authentication>
<chassis xmlns="http://www.ciena.com/ns/yang/ciena-chassis">
<dying-gasp>
<inform>off</inform>
<state>disabled</state>
</dying-gasp>
<temperature>
<high-threshold>65</high-threshold>
<low-threshold>0</low-threshold>
</temperature>
</chassis>
<configuration xmlns="http://www.ciena.com/ns/yang/ciena-configuration-management">
<default-load-filename>startup-config</default-load-filename>
<default-save-filename>startup-config</default-save-filename>
</configuration>
<dns-client xmlns="http://www.ciena.com/ns/yang/ciena-dns-client">
<admin-status>enabled</admin-status>
<domain-name/>
</dns-client>
<ntp xmlns="http://www.ciena.com/ns/yang/ciena-ntp">
<admin-state>enabled</admin-state>
<associations>
<multicast-server/>
<peer-server>
<peering-admin-state>enabled</peering-admin-state>
</peer-server>
</associations>
<authentication>
<auth-admin-state>disabled</auth-admin-state>
</authentication>
<mode>polling</mode>
<polling-interval-max>16</polling-interval-max>
<polling-interval-min>16</polling-interval-min>
<sync-notification>off</sync-notification>
</ntp>
<radius xmlns="http://www.ciena.com/ns/yang/ciena-radius">
<admin-state>enabled</admin-state>
<dot1x-accounting>
<admin-state>disabled</admin-state>
<greylist-timeout>600</greylist-timeout>
<interval>86400</interval>
<retries>3</retries>
<search-method>priority</search-method>
<timeout>1</timeout>
</dot1x-accounting>
<dot1x-authentication>
<config-reauth>off</config-reauth>
<greylist-timeout>600</greylist-timeout>
<retries>3</retries>
<search-method>priority</search-method>
<timeout>1</timeout>
</dot1x-authentication>
<preferred-source-address>default</preferred-source-address>
<user-accounting>
<admin-state>disabled</admin-state>
<retries>3</retries>
<search-method>priority</search-method>
<timeout>1</timeout>
</user-accounting>
<user-login>
<retries>3</retries>
<search-method>priority</search-method>
<timeout>1</timeout>
</user-login>
</radius>
<server xmlns="http://www.ciena.com/ns/yang/ciena-system">
<scp>disabled</scp>
<sftp>disabled</sftp>
</server>
</system>
</data>
I feel as though I've exhausted all the possible options with regard to troubleshooting this. I was going to grab a pcap between the devices, but the Ciena won't do netconf over anything other than ssh. Also, additional logging on that device isn't great. Is there any debugging flags I can enable in NSO itself to really dig more into this? I might try dropping back to an 4.x release of NSO to see if that has any positive impact.
Any tips from experts on how to troubleshoot this beyond what I've done above would be a huge help!
