Error with python script to fetch devices from NSO

[jeyriku]

Hi Team !

I am trying to create a script to fetch a list of devices recorded into NSO thanks to this code :

#!/usr/local/bin/python3
import requests
from pprint import pprint


nso_server = {
   "ip": "10.10.0.1",
   "username": "admin",
   "password": "admin",
   "port": "8080",
}

headers = {
      "Accept" : "application/yang-data+xml", 
      "Content-Type" : "application/yang-data+xml", 
   }

module = "tailf-ncs:devices"

url = f"http://{nso_server['ip']}:{nso_server['port']}/restconf/data/{module}"

response = requests.get(url, headers=headers, auth=(nso_server['username'], nso_server['password']), verify=False).json()
devices = response['tailf-ncs:devices']['device']

# print out all the device name and IP
for device in devices:
    if 'test' not in device['name']:
        dev_name = device['name']
        url_sync = url+'/'+f"device={dev_name}"+'/'+'sync-from'
        sync = requests.post(url_sync, headers=headers, auth=(nso_server['username'], nso_server['password']), verify=False)
    print(f"device:{device['name']} -ip:{device['address']} -port:{device['port']} -sync result:{sync.text}")

As i am a python padawan, i can not even understand the errors returned when script execution fails :

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/urllib3/response.py", line 700, in _update_chunk_length
    self.chunk_left = int(line, 16)
ValueError: invalid literal for int() with base 16: b''

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/urllib3/response.py", line 441, in _error_catcher
    yield
  File "/usr/local/lib/python3.9/site-packages/urllib3/response.py", line 767, in read_chunked
    self._update_chunk_length()
  File "/usr/local/lib/python3.9/site-packages/urllib3/response.py", line 704, in _update_chunk_length
    raise InvalidChunkLength(self, line)
urllib3.exceptions.InvalidChunkLength: InvalidChunkLength(got length b'', 0 bytes read)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 760, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/usr/local/lib/python3.9/site-packages/urllib3/response.py", line 575, in stream
    for line in self.read_chunked(amt, decode_content=decode_content):
  File "/usr/local/lib/python3.9/site-packages/urllib3/response.py", line 796, in read_chunked
    self._original_response.close()
  File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.9/site-packages/urllib3/response.py", line 458, in _error_catcher
    raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/jeyriku/Documents/dev/ztp/Python_&_Rest/python_rest_v0.1.py", line 22, in <module>
    response = requests.get(url, headers=headers, auth=(nso_server['username'], nso_server['password']), verify=False).json()
  File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 687, in send
    r.content
  File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 838, in content
    self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
  File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 763, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))

Some help would be appreciated !!!

Thanks in advance !

Kind Regards,

Jerems

[jeyriku]

It could be file permission related...let's check !

[jeyriku]

It's not permission related but urlib3 failure !

Let's Dig into that then !

A curl request does not lead to such issue.

[Nabsch · Accepted solution]

Hello ,

I changed your script and it's working

#!/usr/local/bin/python3
import requests
from pprint import pprint


nso_server = {
   "ip": "10.10.0.1",
   "username": "admin",
   "password": "admin",
   "port": "8080",
}

headers = {
      "Accept" : "application/yang-data+json",
      "Content-Type" : "application/yang-data+json"
   }

module = "tailf-ncs:devices/"
path="tailf-ncs:devices/device?fields=name;address;port"
url = f"http://{nso_server['ip']}:{nso_server['port']}/restconf/data/{path}"
device_url=f"http://{nso_server['ip']}:{nso_server['port']}/restconf/data/{module}"
response = requests.get(url, headers=headers, auth=(nso_server['username'], nso_server['password']), verify=False).json()
devices = response['tailf-ncs:device']

# print out all the device name and IP
for device in devices:
    if 'test' not in device['name']:
        dev_name = device['name']
        url_sync = device_url+'/'+f"device={dev_name}"+'/'+'sync-from'
        sync = requests.post(url_sync, headers=headers, auth=(nso_server['username'], nso_server['password']), verify=False)
    print(f"device:{device['name']} -ip:{device['address']} -port:{device['port']} -sync result:{sync.text}")

Here the output that i got

device:NETSIM0 -ip:127.0.0.1 -port:10031 -sync result:{
  "tailf-ncs:output": {
    "result": false,
    "info": "Failed to authenticate towards device NETSIM0: SSH host key mismatch"
  }
}

device:TEST-IOS-XR-3.00 -ip:127.0.0.1 -port:10025 -sync result:{
  "tailf-ncs:output": {
    "result": true
  }
}

device:TEST-IOS-XR-3.01 -ip:127.0.0.1 -port:10026 -sync result:{
  "tailf-ncs:output": {
    "result": true
  }
}

[jeyriku]

Hi @Nabsch,

I was missing the correct parameter typo:

?fields

in my restconf get method.

Moreover, The expected format was json instead of xml.

Why do i get an error when trying xml ?

Thank you so much,

Jerems

[Nabsch]

Hello,

It's not working because when you try to retrieve your devices and you have more than one device. So NSO will try to send more than 1 device and you will have more than one root Element which is not Valid in XML.

I try on device it's fails but working on authgroup because i have only one.

nabil@DESKTOP-8ECTID4:~$  curl -i -u admin:admin "http://localhost:8080/restconf/data/tailf-ncs:devices/device?fields=name;address;port" -H "Accept :application/yang-data+xml"
HTTP/1.1 400 Bad Request
Date: Mon, 23 May 2022 22:54:42 GMT
Cache-Control: private, no-cache, must-revalidate, proxy-revalidate
Content-Length: 230
Content-Type: application/yang-data+xml
X-Cisco-NSO-Trace-ID: a4e9f8ab-d6a1-4ed7-927f-de7e11404c39
Pragma: no-cache
Content-Security-Policy: default-src 'self'; block-all-mixed-content; base-uri 'self'; frame-ancestors 'none';
Strict-Transport-Security: max-age=15552000; includeSubDomains
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block


<errors xmlns="urn:ietf:params:xml:ns:yang:ietf-restconf">
  <error>
    <error-type>application</error-type>
    <error-tag>invalid-value</error-tag>
    <error-message>too many instances: 6</error-message>
  </error>
</errors>
nabil@DESKTOP-8ECTID4:~$ curl -i -u admin:admin "http://localhost:8080/restconf/data/tailf-ncs:devices/authgroups/group" -H "Accept :application/yang-data+xml"
HTTP/1.1 200 OK
Date: Mon, 23 May 2022 22:50:04 GMT
Last-Modified: Fri, 20 May 2022 19:13:03 GMT
Cache-Control: private, no-cache, must-revalidate, proxy-revalidate
Etag: "1653-74064-499717"
Content-Type: application/yang-data+xml
Transfer-Encoding: chunked
X-Cisco-NSO-Trace-ID: 9727915c-d6a3-40ac-8696-4d1d5455d35e
Pragma: no-cache
Content-Security-Policy: default-src 'self'; block-all-mixed-content; base-uri 'self'; frame-ancestors 'none';
Strict-Transport-Security: max-age=15552000; includeSubDomains
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block


<group xmlns="http://tail-f.com/ns/ncs"  xmlns:ncs="http://tail-f.com/ns/ncs">
  <name>default</name>
  <umap>
    <local-user>admin</local-user>
    <remote-name>admin</remote-name>
    <remote-password>$9$09ec/rAIrWPCWZZIv87YrZD8Q2oJa0RET3sX0JMadic=</remote-password>
  </umap>
  <umap>
    <local-user>oper</local-user>
    <remote-name>oper</remote-name>
    <remote-password>$9$afUGLCtUdztJBsxAnS6yULzCCqv10jzfe1f8OWw6pYc=</remote-password>
  </umap>
</group>

You have another option is to switch to Collection.

nabil@DESKTOP-8ECTID4:~$ curl -i -u admin:admin "http://localhost:8080/restconf/data/tailf-ncs:devices/device?fields=name;address;port"  -H "Accept : application/vnd.yang.collection+xml"
HTTP/1.1 200 OK
Date: Mon, 23 May 2022 22:56:22 GMT
Last-Modified: Fri, 20 May 2022 19:13:03 GMT
Cache-Control: private, no-cache, must-revalidate, proxy-revalidate
Etag: "1653-74064-499717"
Content-Type: application/vnd.yang.collection+xml
Transfer-Encoding: chunked
X-Cisco-NSO-Trace-ID: 3b8fb897-2894-4099-9ac6-4c0766e5224d
Pragma: no-cache
Content-Security-Policy: default-src 'self'; block-all-mixed-content; base-uri 'self'; frame-ancestors 'none';
Strict-Transport-Security: max-age=15552000; includeSubDomains
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block

<collection xmlns="http://tail-f.com/ns/restconf/collection/1.0">
  <device xmlns="http://tail-f.com/ns/ncs"  xmlns:ncs="http://tail-f.com/ns/ncs">
    <name>TEST-IOS-XR-3.00</name>
    <address>127.0.0.1</address>
    <port>10025</port>
  </device>
  <device xmlns="http://tail-f.com/ns/ncs"  xmlns:ncs="http://tail-f.com/ns/ncs">
    <name>TEST-IOS-XR-3.01</name>
    <address>127.0.0.1</address>
    <port>10026</port>
  </device>

Here a link that might help you