Solved: Strange pioneer/NSO behavior with JunOS

Iman1 · ‎08-29-2017

Sorry there are probably multiple questions below, I try to follow them in a logical order. The main problem/question is that Pioneer cannot fetch/download the yang modules from the device, however there are other anomalies seen too as I mention below.

NSO version 4.4.2. Device is JunOS 17.2R1.13. Created netconf NED from modules that were manually downloaded from device (because pioneer fetch didn't work).

Trying to download modules via pioneer:

Even though device advertises netconf-monitoring capability, Pioneer fetch-list reports that "Device does not report support for netconf-monitoring":

admin@ncs> request devices device qfx1 pioneer netconf hello

hello-reply <?xml version="1.0"?>

<nc:hello xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0">

<nc:capabilities>

<nc:capability>urn:ietf:params:netconf:base:1.0</nc:capability>

<nc:capability>urn:ietf:params:netconf:capability:candidate:1.0</nc:capability>

<nc:capability>urn:ietf:params:netconf:capability:confirmed-commit:1.0</nc:capability>

<nc:capability>urn:ietf:params:netconf:capability:validate:1.0</nc:capability>

<nc:capability>urn:ietf:params:netconf:capability:url:1.0?protocol=http,ftp,file</nc:capability>

<nc:capability>urn:ietf:params:xml:ns:netconf:base:1.0</nc:capability>

<nc:capability>urn:ietf:params:xml:ns:netconf:capability:candidate:1.0</nc:capability>

<nc:capability>urn:ietf:params:xml:ns:netconf:capability:confirmed-commit:1.0</nc:capability>

<nc:capability>urn:ietf:params:xml:ns:netconf:capability:validate:1.0</nc:capability>

<nc:capability>urn:ietf:params:xml:ns:netconf:capability:url:1.0?protocol=http,ftp,file</nc:capability>

<capability>urn:ietf:params:xml:ns:yang:ietf-netconf-monitoring</capability>

<nc:capability>http://xml.juniper.net/netconf/junos/1.0</nc:capability>

<nc:capability>http://xml.juniper.net/dmi/system/1.0</nc:capability>

</nc:capabilities>

<nc:session-id>26045</nc:session-id>

</nc:hello>

[ok][2017-08-29 17:15:36]

admin@ncs> request devices device qfx1 pioneer yang fetch-list

Retrieving module list from device

Device does not report support for netconf-monitoring, trying anyway

Found out the names for a total of 0 modules

hello message: 0

netconf-monitoring subtree: 0

netconf-monitoring xpath: 0

message Marked 0 modules for download, skipped 0

yang-directory /tmp/download/qfx1

[ok][2017-08-29 17:16:10]

I found that when rfc-compliant (see details here) is deactivated from JunOS device configuration, pioneer now reports that "device supports netconf-monitoring", but strangely there is no difference in netconf-monitoring support from device perspective, device still advertises the netconf-monitoring capability same as before (before deactivating rfc-compliant).

admin@ncs> request devices device qfx1 pioneer netconf hello

hello-reply <?xml version="1.0"?>

<capability>urn:ietf:params:netconf:base:1.0</capability>

<capability>urn:ietf:params:netconf:capability:candidate:1.0</capability>

<capability>urn:ietf:params:netconf:capability:confirmed-commit:1.0</capability>

<capability>urn:ietf:params:netconf:capability:validate:1.0</capability>

<capability>urn:ietf:params:netconf:capability:url:1.0?scheme=http,ftp,file</capability>

<capability>urn:ietf:params:xml:ns:netconf:base:1.0</capability>

<capability>urn:ietf:params:xml:ns:netconf:capability:candidate:1.0</capability>

<capability>urn:ietf:params:xml:ns:netconf:capability:confirmed-commit:1.0</capability>

<capability>urn:ietf:params:xml:ns:netconf:capability:validate:1.0</capability>

<capability>urn:ietf:params:xml:ns:netconf:capability:url:1.0?protocol=http,ftp,file</capability>

<capability>urn:ietf:params:xml:ns:yang:ietf-netconf-monitoring</capability>

<capability>http://xml.juniper.net/netconf/junos/1.0</capability>

<capability>http://xml.juniper.net/dmi/system/1.0</capability>

</capabilities>

<session-id>27757</session-id>

</hello>

[ok][2017-08-29 18:09:05]

admin@ncs> request devices device qfx1 pioneer yang fetch-list

Retrieving module list from device

Device supports netconf-monitoring

Found out the names for a total of 0 modules

hello message: 0

netconf-monitoring subtree: 0

netconf-monitoring xpath: 0

message Marked 0 modules for download, skipped 0

yang-directory /tmp/download/qfx1

[ok][2017-08-29 18:09:12]

Other very strange symptoms - running various 'request' commands consecutively, returns continuously changing results between "error" and "ok"

admin@ncs> request devices device qfx1 connect

result false

info Device qfx1 does not advertise any known YANG modules

[ok][2017-08-29 17:27:33]

admin@ncs> request devices device qfx1 connect

result true

info (admin) Connected to qfx1 - 172.25.205.195:830

[ok][2017-08-29 17:27:35]

admin@ncs> request devices device qfx1 check-sync

result error

info Device qfx1 does not advertise any known YANG modules

[ok][2017-08-29 17:43:58]

admin@ncs> request devices device qfx1 check-sync

result out-of-sync

info got: 2017-08-29 23:26:48 UTC expected: 2017-08-25 18:08:03 UTC

[ok][2017-08-29 17:44:00]

admin@ncs> *** ALARM out-of-sync: got: 2017-08-29 23:26:48 UTC expected: 2017-08-25 18:08:03 UTC

admin@ncs> request devices device qfx1 compare-config

info Device qfx1 does not advertise any known YANG modules

[ok][2017-08-29 17:44:28]

admin@ncs> request devices device qfx1 compare-config

[ok][2017-08-29 17:44:31]

admin@ncs> request devices device qfx1 compare-config

info Device qfx1 does not advertise any known YANG modules

[ok][2017-08-29 17:44:33]

admin@ncs> request devices device qfx1 compare-config

[ok][2017-08-29 17:44:36]

Jan Lindblad · ‎08-30-2017

This is unfortunately an issue with the Juniper YANG implementation.

Juniper pretty much invented NETCONF and have had good support for this for ages. YANG on the other hand is only very recently supported by Juniper. They have been relying on a special form of XML Schema (XSD) for describing their model. Because of this, at Tail-f, we developed a tool that translates these XSD schemas to YANG, and those generated YANGs are then what's used in the Juniper NED.

Now that it's possible to obtain Juniper official YANG modules for JunOS devices, I and many others have tried to build NEDs out of them. This works mostly fine, but then a major hurdle hits. Even the newest JunOS devices don't announce the YANG namespaces as required by the NETCONF protocol. So all those new YANG models are useless anyway.

That's why you see the messages "Device qfx1 does not advertise any known YANG modules" Even if the YANGs are loaded into NSO, the device denies any knowledge of these namespaces. So then NSO and the device has nothing to talk about.

I discussed the issue with the Juniper team at EANTC NETCONF/YANG interop in February this year. They blushed a little and said it would be fixed "soon". To date, I haven't heard anyone report a sighting of a JunOS device where this issue has been fixed.

So I'm afraid you'll have to go back to using the official NSO JunOS NED for now, or possibly get the JunOS equipment updated.

View solution in original post

Jan Lindblad · ‎08-30-2017

This is unfortunately an issue with the Juniper YANG implementation.

Juniper pretty much invented NETCONF and have had good support for this for ages. YANG on the other hand is only very recently supported by Juniper. They have been relying on a special form of XML Schema (XSD) for describing their model. Because of this, at Tail-f, we developed a tool that translates these XSD schemas to YANG, and those generated YANGs are then what's used in the Juniper NED.

Now that it's possible to obtain Juniper official YANG modules for JunOS devices, I and many others have tried to build NEDs out of them. This works mostly fine, but then a major hurdle hits. Even the newest JunOS devices don't announce the YANG namespaces as required by the NETCONF protocol. So all those new YANG models are useless anyway.

That's why you see the messages "Device qfx1 does not advertise any known YANG modules" Even if the YANGs are loaded into NSO, the device denies any knowledge of these namespaces. So then NSO and the device has nothing to talk about.

I discussed the issue with the Juniper team at EANTC NETCONF/YANG interop in February this year. They blushed a little and said it would be fixed "soon". To date, I haven't heard anyone report a sighting of a JunOS device where this issue has been fixed.

So I'm afraid you'll have to go back to using the official NSO JunOS NED for now, or possibly get the JunOS equipment updated.

Iman1 · ‎08-30-2017

Thanks Jan for the awesome explanation.

So the main issue is Juniper not announcing module namespaces in hello, correct?

I also noticed the following in Juniper's log which look like Juniper and NSO also have syntax issues in exchanging netconf-monitoring transactions. Is this just another hurdle in Juniper's implementation of netconf-monitoring yang? (incoming is from NSO)

Aug 30 00:38:18 [NETCONF] - [29642] Incoming: <?xml version="1.0" encoding="UTF-8"?>

</rpc>]]>]]>

Aug 30 00:38:18 [NETCONF] - [29642] Outgoing: <nc:rpc-reply xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0" xmlns:junos="http://xml.juniper.net/junos/17.2R1/junos" xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="1">

Aug 30 00:38:18 [NETCONF] - [29642] Outgoing: <nc:rpc-error>

Aug 30 00:38:18 [NETCONF] - [29642] Outgoing: <nc:error-type>protocol</nc:error-type>

Aug 30 00:38:18 [NETCONF] - [29642] Outgoing: <nc:error-tag>operation-failed</nc:error-tag>

Aug 30 00:38:18 [NETCONF] - [29642] Outgoing: <nc:error-severity>error</nc:error-severity>

Aug 30 00:38:18 [NETCONF] - [29642] Outgoing: <nc:error-message>syntax error, expecting <config-text/>, <netconf-state>, <configuration>, or <configuration/></nc:error-message>

Aug 30 00:38:18 [NETCONF] - [29642] Outgoing: <nc:error-info>

Aug 30 00:38:18 [NETCONF] - [29642] Outgoing: <nc:bad-element>netconf-state</nc:bad-element>

Aug 30 00:38:18 [NETCONF] - [29642] Outgoing: </nc:error-info>

Aug 30 00:38:18 [NETCONF] - [29642] Outgoing: </nc:rpc-error>

Aug 30 00:38:18 [NETCONF] - [29642] Debug: The last token parsed by mgd [29642] was [netconf-state] and gram data current token [netconf-state]

Aug 30 00:38:18 [NETCONF] - [29642] Outgoing: </nc:rpc-reply>

Aug 30 00:38:18 [NETCONF] - [29642] Outgoing: ]]>]]>

And by the way... all these Juniper issues aside, do you think NSO's behavior is a bit suspect too? If you refer to last part of my previous post, NSO seems to be unstable in its response to consecutive issues of the same command. It alternates between reporting the issue correctly, and reporting false "ok/successful" message.