ANNOUNCEMENT - The community will be down for maintenace this Thursday August 13 from 12:00 AM PT to 02:00 AM PT. As a precaution save your work.
cancel
Showing results for 
Search instead for 
Did you mean: 
cancel

Implementing GRPC/Telemetry on XR devices

4832
Views
10
Helpful
2
Comments

 

 

 

Introduction

 

The objective of this document is to enable you to configure telemetry with pipeline collector. This is perfect replacement for SNMP with more scalability and performance.

Any platform variants of ASR9000 running eXR (64bit)  and Fretta running 61x release and above supports streaming telemetry. GRPC is not supported on cXR (32bit) images.

 

SNMP Vs Telemetry

 

SNMP

Telemetry

Pull model

Push model

No Encoding

Encoding (cGPB, key-Value GPB, JSON)

Polling period ( 5-30mins)

Streaming ( 1sec onwards)

Less scalable

Highly scalable

Moderate performance(slow OID’s)

High Performance

Traps

Event driven telemetry (Upon state change/frequency)

SNMPv3 MD5/SHA/DES

TLS Authentication

Data based on OID's

Data based on Yang models

 

 

 

Use cases

 

 These three modes are supported

  • Dial-IN
  • Dial-OUT
  • Event driven

 

Dial In mode: Sensor paths and subscriptions are configured on the router. The collector establishes a session and subscribes dynamically to subscriptions on the router. The router streams data through the established session.

 

 Screen Shot 2018-06-05 at 5.47.15 AM.png

Dial out mode:  Sensor paths, destinations are configured and attached to one or more telemetry subscriptions. The router tries to establish a session every 1minute(based on the subscription) to each destination for each subscription. The data is then streamed out for each session according to the specified sample interval.

 Screen Shot 2018-06-05 at 5.50.09 AM.png

 In both Dial in and Dial out cases, data is streamed from the router.

 

Event driven Telemetry objective is to send updates only when the event occurs in the subscribed Sensor paths after dumping all the data at first. Currently it’s with limited path support from r63x releases.

Authentication: Telemetry also  supports TLS for encryption of data. When TLS is enabled, the server will have to send a certificate to authenticate itself to the collector. The collector validates the certificate using the certificate authority that signed it and then generates session keys to encrypt the session.

When collector wants to receive updates regarding the state of data on the router, the collector creates a subscription. A subscription can consist of one or more paths.  You can see keep seeing data of the paths that you have subscribed indefinitely until you cancel the subscription or until the session terminates.

 

emsd (extensible manageability service daemon) process is responsible for streaming telemetry operation. After configuring grpc ensure that emsd process is started. Show process emsd should give you the details about the process.

Configuration

 

  • Configure GRPC:

RP/0/RSP0/CPU0:BX03#show run grpc

grpc

 port 57400

 no-tls

 

  • Configure telemetry MDT dial-in config:

The below has 2 config sets

(i) sensor group

(ii) subscription

Sensor group consists of sensor paths which describes the yang model

 

RP/0/RSP0/CPU0:BX03#show run telemetry model-driven

telemetry model-driven

 sensor-group 1minute

  sensor-path Cisco-IOS-XR-asr9k-np-oper:hardware-module-np/nodes/node/nps/np/load-utilization

  sensor-path Cisco-IOS-XR-telemetry-model-driven-oper:telemetry-model-driven/destinations/destination

 !

 subscription 1minute

  sensor-group-id 1minute sample-interval 60000       à interval in ms (milliseconds)

 

Verification

 

After configuring GRPC and telemetry configs ensure the sensor paths are getting resolved.

Execute the following command to check if the sensor path is resolved.

 

RP/0/RSP0/CPU0:BX03#    show telemetry model-driven subscription

Subscription:  1minute                  State: NA             à No active session

-------------

  Sensor groups:

  Id                               Interval(ms)        State    

  1minute                          60000               Resolved 

 

If there are active sessions then the session state will be ACTIVE and destination group will show the collector address.

 

RP/0/RSP0/CPU0:BX03#    show telemetry model-driven subscription

Subscription:  1minute                  State: ACTIVE

-------------

  Sensor groups:

  Id                               Interval(ms)        State    

  1minute                          60000               Resolved 

 

  Destination Groups:

  Id                 Encoding            Transport   State   Port    Vrf     IP           

  DialIn_1014        self-describing-gpb dialin      Active  59698           123.100.105.79

    No TLS           

 

To get the detailed stats, please use the CLI with subscription name:

 

RP/0/RSP0/CPU0:BX03#show telemetry model-driven subscription 1minute

Subscription:  1minute

-------------

  State:       ACTIVE

  Sensor groups:

  Id: 1minute

    Sample Interval:      60000 ms

    Sensor Path:          Cisco-IOS-XR-telemetry-model-driven-oper:telemetry-model-driven/destinations/destination

    Sensor Path State:    Resolved

    Sensor Path:          Cisco-IOS-XR-asr9k-np-oper:hardware-module-np/nodes/node/nps/np/load-utilization

    Sensor Path State:    Resolved

 

  Destination Groups:

  Group Id: DialIn_1014

    Destination IP:       123.100.105.79

    Destination Port:     59698

    Encoding:             self-describing-gpb

    Transport:            dialin

    State:                Active

    No TLS               

    Total bytes sent:     80314

    Total packets sent:   36

    Last Sent time:       2018-06-02 20:56:38.4023157852 +0000

 

  Collection Groups:

  ------------------

    Id: 100

    Sample Interval:      60000 ms

    Encoding:             self-describing-gpb

    Num of collection:    18

    Collection time:      Min:     3 ms Max:     6 ms

    Total time:           Min:     3 ms Avg:     5 ms Max:    10 ms

    Total Deferred:       0

    Total Send Errors:    0

    Total Send Drops:     0

    Total Other Errors:   0

    No data Instances:    0

    Last Collection Start:2018-06-02 20:56:26.4010942852 +0000

    Last Collection End:  2018-06-02 20:56:26.4010950852 +0000

    Sensor Path:          Cisco-IOS-XR-telemetry-model-driven-oper:telemetry-model-driven/destinations/destination

 

    Id: 101

    Sample Interval:      60000 ms

    Encoding:             self-describing-gpb

    Num of collection:    18

    Collection time:      Min:   670 ms Max:   702 ms

    Total time:           Min:   676 ms Avg:   683 ms Max:   711 ms

    Total Deferred:       0

    Total Send Errors:    0

    Total Send Drops:     0

    Total Other Errors:   0

    No data Instances:    0

    Last Collection Start:2018-06-02 20:56:37.4022478852 +0000

    Last Collection End:  2018-06-02 20:56:38.4023157852 +0000

    Sensor Path:          Cisco-IOS-XR-asr9k-np-oper:hardware-module-np/nodes/node/nps/np/load-utilization

 

After the collector establishes a session with the router, we can see that router sends stream of data. In order to get detailed information about subscriptions on the router, we run ‘show telemetry model-driven <Subscription id > internal’. This command gives the state of the subscription which is ACTIVE as data was being streamed at that particular point of time, sensor group id 1minute is attached with the subscription, sensor path is the path which the collector has subscribed to, sample interval is time interval between 2 samples of data and Sensor path state tells us if the path we have given is valid or not.

 

A destination group is also attached to the subscription. Destination ip and port are generated dynamically. Data is encoded using self-describing-gpb. As collector initialized the session, the transport being used in this case is Dial In. State tells us about the state of the subscription, No TLS says that transport layer security is disabled. Total bytes sent and Total packets sent gives us stats about amount of data collected. Collection groups are also attached to the subscription. This group tells us about the some of the statistics related to the subscription. The Sysdb Path gives us the path from which data is being collected internally.

 

Telemetry Debugging:

Not streaming data? Path not resolved? Unable to connect to Router?

Here are few simple checks to perform:

  1. When grpc is configured check emsd process is running
  2. Check for configurations present in the router such as  subscription, sensor group, sensor path, destination group(in case of dial-out), encoding and transport.        
  3. Check that the sensor paths configured on the router are in Resolved state . 

When Sensor path is Resolved and No data streamed:

  1. run sysdbcon -m for operational data, when operational data doesn’t exist the solution is to configure the specific feature on the router.

When sensor path is Not Resolved:

  1. There could be an error in the specified path
  2. The data model may not be supported by the router

Note: GRPC does not support both ipv4 and ipv6 address family co-existence.

 

When Management IP is not reachable:

  1. Ensure that MAC address and IP addresses are reflected in XR and TPNNS. To verify that please do    the following. Also type check the route entry for the collector by executing "route" from bash prompt.    

       RP/0/RSP0/CPU0:BX03#bash 
       [BX03:~]$ 

        [BX03:~]$ ifconfig Mg0_RSP0_CPU0_0          
        Mg0_RSP0_CPU0_0 Link encap:Ethernet  HWaddr 00:c1:64:6e:9c:40  
          inet addr:1.78.28.103  Mask:255.255.0.0
          inet6 addr: fe80::2c1:64ff:fe6e:9c40/64 Scope:Link
          UP RUNNING NOARP MULTICAST  MTU:1500  Metric:1
          RX packets:55677096 errors:0 dropped:0 overruns:0 frame:0
          TX packets:55528442 errors:0 dropped:0 overruns:0 carrier:1
          collisions:0 txqueuelen:1000 
          RX bytes:6037939100 (5.6 GiB)  TX bytes:6125568522 (5.7 GiB)

 

 Setting up the collector (pipeline client)

 

 Clone from git for the pipeline client as mentioned below:

 

[root@csg-tacacs telemetry]# git clone https://github.com/cisco/bigmuddy-network-telemetry-pipeline.git

Initialized empty Git repository in /auto/tftp-vista/narvenka/telemetry/bigmuddy-network-telemetry-pipeline/.git/

remote: Counting objects: 14615, done.

remote: Total 14615 (delta 0), reused 0 (delta 0), pack-reused 14615

Receiving objects: 100% (14615/14615), 37.72 MiB | 2.63 MiB/s, done.

Resolving deltas: 100% (4014/4014), done.

 

bigmuddy-network-telemetry-pipeline/ directory created under the current directory.

Copy the pipeline.conf into bin/ directory to reduce the runtime args.

 

MDT can be operated in 2 modes. Dial-in and Dial-out SYN first.

Either way the data will be pushed out of the router.

 

Configure Pipeline

 

The sample pipeline dial-in configs are:

 

##############################################

# Example of a gRPC dialin (pipeline connects to router over gRPC)

[CX]                           à hostname

stage = xport_input

#

# grpc: in the role client connecting to router doing server side

# streaming.

#

type = grpc

#

# Encoding requested by grpc client: gpbcompact, gpbkv

# This is only pertinent to dialin and describes the request

# to the router - i.e. do I want compact or k/v payload

encoding = gpbkv

#

# Encapsulation pushed by client: gpb, gpbkv, gpbcompact

# As of 6.1.1 release, we support gpb (which is a common

# header used to carry compact and k/v gpb). In older

# releases, it might be necessary to configure compact or k/v.

# encap = gpb

#

# Server to connect to

#

server = 1.78.28.103:57400   à router IP:GRPC port

#

# Subscription IDs to subscribe to, as configured on the router. Set

# is specified as a comma separated string of subscription ID names.

#

subscriptions = 1minute         à Subscriptions can we added with a , delimeter

#

#

# Access control options; TLS,  username/password

#

# TLS enabled or not, CA cert in PEM, and server name to accept.

#

# tls = false

# tls_pem = ca.pem

# tls_servername = tlsservername

#

# username/password

#

# Public key encrypted password is specified here. The pem with RSA

# key used to encrypt passwords is passed in as a cli argument.

#

# The encrypted password will be generated if pipeline is run, -pem is

# specified, and password option is NOT included in the section. In

# that user will be prompted for user/pass, and a new separate copy of

# configuration will be created for convenience.

#

# Passphrase-less RSA key pair can be generated using ssh-keygen if

# necessary: 'ssh-keygen -t rsa -b 4096'

#

#username = lab

#password = lab

#

# To enable dumping data as it is rxed, uncomment the following, and

# run with --debug option.

#

# logdata = on

###########################################

 

After editing the above config in pipeline.conf file just save it. So the streamed data will be in dump.txt. This is based on the directory configured in the pipeline.conf

##############################################

# Example of a tap stage; dump content to file, or at least count messages

#

[inspector]

stage = xport_output

#

# Module type: tap is only supported in xport_output stage currently.

#

type = tap

#

# File to dump decoded messages

#

file = dump.txt

##############################################

 

 

Pipeline execution

 

To start streaming just execute like below. The username and password is tacacs or local which can be authenticated by grpc. If tacacs is enabled then you need to use tacacs username and password else local username password can be used.

 

[root@csg-tacacs bin]# ./pipeline

Startup pipeline

Load config from [pipeline.conf], logging in [pipeline.log]

 

CRYPT Client [cx],[1.78.28.103:57400]

 Enter username: lab

 Enter password:

Wait for ^C to shutdown

 

All set, just check for dump.txt for the streamed data!

 

Pipeline execution on mac using docker

Make sure docker is installed in your mac before continuing the below step

docker file location is: https://github.com/cisco/bigmuddy-network-telemetry-pipeline/tree/master/docker

 

steps to follow:

Download the big muddy zip file from

https://github.com/cisco/bigmuddy-network-telemetry-pipeline

After download unzip and change the directory to docker.

Then execute the following.

 

AWP-M-72K6:docker mymac$ docker build -t pipeline .

Sending build context to Docker daemon  74.66MB

Step 1/10 : FROM debian:stable-slim

stable-slim: Pulling from library/debian

c262086186dd: Pull complete

Digest: sha256:76e4d780ebdd81315c1d67e0a044fabc06db5805352e3322594360d3990be1b6

Status: Downloaded newer image for debian:stable-slim

---> 414b5dbe710f

Step 2/10 : MAINTAINER Christian Cassar <ccassar@cisco.com>

---> Running in bd11994e835f

Removing intermediate container bd11994e835f

---> 098d36542ca1

Step 3/10 : ADD pipeline.conf /data/pipeline.conf

---> a5baaa3f2833

Step 4/10 : ADD metrics.json /data/metrics.json

---> 3a1150d58676

Step 5/10 : ADD metrics_gpb.json /data/metrics_gpb.json

---> a01a0719ca40

Step 6/10 : ADD pipeline /pipeline

---> db5c9e3b500e

Step 7/10 : VOLUME ["/data"]

---> Running in 03d9203a11fc

Removing intermediate container 03d9203a11fc

---> b7361654df3f

Step 8/10 : WORKDIR /

---> Running in d3869ad4afc3

Removing intermediate container d3869ad4afc3

---> 79538467f7fe

Step 9/10 : ENTRYPOINT ["/pipeline"]

---> Running in 9df783306916

Removing intermediate container 9df783306916

---> e3954f0d5584

Step 10/10 : CMD ["-log=/data/pipeline.log","-config=/data/pipeline.conf"]

---> Running in 0744900c0486

Removing intermediate container 0744900c0486

---> 17f5793f23f7

Successfully built 17f5793f23f7

Successfully tagged pipeline:latest

NARVENKA-M-72K6:docker venkatnagarajan$ docker images

REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE

pipeline            latest              17f5793f23f7        11 seconds ago      130MB

debian              stable-slim         414b5dbe710f        2 days ago          55.3MB

AWP-M-72K6:docker mymac$

 

Just run pipeline using docker:

AWP-M-72K6:docker mymac$ docker run --name pipeline pipeline

Startup pipeline

Load config from [/data/pipeline.conf], logging in [/data/pipeline.log]

Wait for ^C to shutdown

 

All set now.

 

Comments
Cisco Employee

Hi is there a way to run pipeline in MAC OS?

 

María E.

Cisco Employee

Yes Maria, you can run via mac using docker. We have docker file:

https://github.com/cisco/bigmuddy-network-telemetry-pipeline/tree/master/docker

Let me know if you have any questions.

 

Regards,

 

Venkat Nagarajan

 

Content for Community-Ad

This widget could not be displayed.