- Introduction
- SNMP Vs Telemetry
- Use cases
- Configuration
- Verification
- Telemetry Debugging:
- Setting up the collector (pipeline client)
- Configure Pipeline
- Pipeline execution
- Pipeline execution on mac using docker
Introduction
The objective of this document is to enable you to configure telemetry with pipeline collector. This is perfect replacement for SNMP with more scalability and performance.
Any platform variants of ASR9000 running eXR (64bit) and Fretta running 61x release and above supports streaming telemetry. GRPC is not supported on cXR (32bit) images.
SNMP Vs Telemetry
|
SNMP |
Telemetry |
|
Pull model |
Push model |
|
No Encoding |
Encoding (cGPB, key-Value GPB, JSON) |
|
Polling period ( 5-30mins) |
Streaming ( 1sec onwards) |
|
Less scalable |
Highly scalable |
|
Moderate performance(slow OID’s) |
High Performance |
|
Traps |
Event driven telemetry (Upon state change/frequency) |
|
SNMPv3 MD5/SHA/DES |
TLS Authentication |
|
Data based on OID's |
Data based on Yang models |
Use cases
These three modes are supported
- Dial-IN
- Dial-OUT
- Event driven
Dial In mode: Sensor paths and subscriptions are configured on the router. The collector establishes a session and subscribes dynamically to subscriptions on the router. The router streams data through the established session.
Dial out mode: Sensor paths, destinations are configured and attached to one or more telemetry subscriptions. The router tries to establish a session every 1minute(based on the subscription) to each destination for each subscription. The data is then streamed out for each session according to the specified sample interval.
In both Dial in and Dial out cases, data is streamed from the router.
Event driven Telemetry objective is to send updates only when the event occurs in the subscribed Sensor paths after dumping all the data at first. Currently it’s with limited path support from r63x releases.
Authentication: Telemetry also supports TLS for encryption of data. When TLS is enabled, the server will have to send a certificate to authenticate itself to the collector. The collector validates the certificate using the certificate authority that signed it and then generates session keys to encrypt the session.
When collector wants to receive updates regarding the state of data on the router, the collector creates a subscription. A subscription can consist of one or more paths. You can see keep seeing data of the paths that you have subscribed indefinitely until you cancel the subscription or until the session terminates.
emsd (extensible manageability service daemon) process is responsible for streaming telemetry operation. After configuring grpc ensure that emsd process is started. Show process emsd should give you the details about the process.
Configuration
- Configure GRPC:
RP/0/RSP0/CPU0:BX03#show run grpc
grpc
port 57400
no-tls
- Configure telemetry MDT dial-in config:
The below has 2 config sets
(i) sensor group
(ii) subscription
Sensor group consists of sensor paths which describes the yang model
RP/0/RSP0/CPU0:BX03#show run telemetry model-driven
telemetry model-driven
sensor-group 1minute
sensor-path Cisco-IOS-XR-asr9k-np-oper:hardware-module-np/nodes/node/nps/np/load-utilization
sensor-path Cisco-IOS-XR-telemetry-model-driven-oper:telemetry-model-driven/destinations/destination
!
subscription 1minute
sensor-group-id 1minute sample-interval 60000 à interval in ms (milliseconds)
Verification
After configuring GRPC and telemetry configs ensure the sensor paths are getting resolved.
Execute the following command to check if the sensor path is resolved.
RP/0/RSP0/CPU0:BX03# show telemetry model-driven subscription
Subscription: 1minute State: NA à No active session
-------------
Sensor groups:
Id Interval(ms) State
1minute 60000 Resolved
If there are active sessions then the session state will be ACTIVE and destination group will show the collector address.
RP/0/RSP0/CPU0:BX03# show telemetry model-driven subscription
Subscription: 1minute State: ACTIVE
-------------
Sensor groups:
Id Interval(ms) State
1minute 60000 Resolved
Destination Groups:
Id Encoding Transport State Port Vrf IP
DialIn_1014 self-describing-gpb dialin Active 59698 123.100.105.79
No TLS
To get the detailed stats, please use the CLI with subscription name:
RP/0/RSP0/CPU0:BX03#show telemetry model-driven subscription 1minute
Subscription: 1minute
-------------
State: ACTIVE
Sensor groups:
Id: 1minute
Sample Interval: 60000 ms
Sensor Path: Cisco-IOS-XR-telemetry-model-driven-oper:telemetry-model-driven/destinations/destination
Sensor Path State: Resolved
Sensor Path: Cisco-IOS-XR-asr9k-np-oper:hardware-module-np/nodes/node/nps/np/load-utilization
Sensor Path State: Resolved
Destination Groups:
Group Id: DialIn_1014
Destination IP: 123.100.105.79
Destination Port: 59698
Encoding: self-describing-gpb
Transport: dialin
State: Active
No TLS
Total bytes sent: 80314
Total packets sent: 36
Last Sent time: 2018-06-02 20:56:38.4023157852 +0000
Collection Groups:
------------------
Id: 100
Sample Interval: 60000 ms
Encoding: self-describing-gpb
Num of collection: 18
Collection time: Min: 3 ms Max: 6 ms
Total time: Min: 3 ms Avg: 5 ms Max: 10 ms
Total Deferred: 0
Total Send Errors: 0
Total Send Drops: 0
Total Other Errors: 0
No data Instances: 0
Last Collection Start:2018-06-02 20:56:26.4010942852 +0000
Last Collection End: 2018-06-02 20:56:26.4010950852 +0000
Sensor Path: Cisco-IOS-XR-telemetry-model-driven-oper:telemetry-model-driven/destinations/destination
Id: 101
Sample Interval: 60000 ms
Encoding: self-describing-gpb
Num of collection: 18
Collection time: Min: 670 ms Max: 702 ms
Total time: Min: 676 ms Avg: 683 ms Max: 711 ms
Total Deferred: 0
Total Send Errors: 0
Total Send Drops: 0
Total Other Errors: 0
No data Instances: 0
Last Collection Start:2018-06-02 20:56:37.4022478852 +0000
Last Collection End: 2018-06-02 20:56:38.4023157852 +0000
Sensor Path: Cisco-IOS-XR-asr9k-np-oper:hardware-module-np/nodes/node/nps/np/load-utilization
After the collector establishes a session with the router, we can see that router sends stream of data. In order to get detailed information about subscriptions on the router, we run ‘show telemetry model-driven <Subscription id > internal’. This command gives the state of the subscription which is ACTIVE as data was being streamed at that particular point of time, sensor group id 1minute is attached with the subscription, sensor path is the path which the collector has subscribed to, sample interval is time interval between 2 samples of data and Sensor path state tells us if the path we have given is valid or not.
A destination group is also attached to the subscription. Destination ip and port are generated dynamically. Data is encoded using self-describing-gpb. As collector initialized the session, the transport being used in this case is Dial In. State tells us about the state of the subscription, No TLS says that transport layer security is disabled. Total bytes sent and Total packets sent gives us stats about amount of data collected. Collection groups are also attached to the subscription. This group tells us about the some of the statistics related to the subscription. The Sysdb Path gives us the path from which data is being collected internally.
Telemetry Debugging:
Not streaming data? Path not resolved? Unable to connect to Router?
Here are few simple checks to perform:
- When grpc is configured check emsd process is running
- Check for configurations present in the router such as subscription, sensor group, sensor path, destination group(in case of dial-out), encoding and transport.
- Check that the sensor paths configured on the router are in Resolved state .
When Sensor path is Resolved and No data streamed:
- run sysdbcon -m for operational data, when operational data doesn’t exist the solution is to configure the specific feature on the router.
When sensor path is Not Resolved:
- There could be an error in the specified path
- The data model may not be supported by the router
Note: GRPC does not support both ipv4 and ipv6 address family co-existence.
When Management IP is not reachable:
- Ensure that MAC address and IP addresses are reflected in XR and TPNNS. To verify that please do the following. Also type check the route entry for the collector by executing "route" from bash prompt.
RP/0/RSP0/CPU0:BX03#bash
[BX03:~]$
[BX03:~]$ ifconfig Mg0_RSP0_CPU0_0
Mg0_RSP0_CPU0_0 Link encap:Ethernet HWaddr 00:c1:64:6e:9c:40
inet addr:1.78.28.103 Mask:255.255.0.0
inet6 addr: fe80::2c1:64ff:fe6e:9c40/64 Scope:Link
UP RUNNING NOARP MULTICAST MTU:1500 Metric:1
RX packets:55677096 errors:0 dropped:0 overruns:0 frame:0
TX packets:55528442 errors:0 dropped:0 overruns:0 carrier:1
collisions:0 txqueuelen:1000
RX bytes:6037939100 (5.6 GiB) TX bytes:6125568522 (5.7 GiB)
Setting up the collector (pipeline client)
Clone from git for the pipeline client as mentioned below:
[root@csg-tacacs telemetry]# git clone https://github.com/cisco/bigmuddy-network-telemetry-pipeline.git
Initialized empty Git repository in /auto/tftp-vista/narvenka/telemetry/bigmuddy-network-telemetry-pipeline/.git/
remote: Counting objects: 14615, done.
remote: Total 14615 (delta 0), reused 0 (delta 0), pack-reused 14615
Receiving objects: 100% (14615/14615), 37.72 MiB | 2.63 MiB/s, done.
Resolving deltas: 100% (4014/4014), done.
bigmuddy-network-telemetry-pipeline/ directory created under the current directory.
Copy the pipeline.conf into bin/ directory to reduce the runtime args.
MDT can be operated in 2 modes. Dial-in and Dial-out SYN first.
Either way the data will be pushed out of the router.
Configure Pipeline
The sample pipeline dial-in configs are:
##############################################
# Example of a gRPC dialin (pipeline connects to router over gRPC)
[CX] à hostname
stage = xport_input
#
# grpc: in the role client connecting to router doing server side
# streaming.
#
type = grpc
#
# Encoding requested by grpc client: gpbcompact, gpbkv
# This is only pertinent to dialin and describes the request
# to the router - i.e. do I want compact or k/v payload
encoding = gpbkv
#
# Encapsulation pushed by client: gpb, gpbkv, gpbcompact
# As of 6.1.1 release, we support gpb (which is a common
# header used to carry compact and k/v gpb). In older
# releases, it might be necessary to configure compact or k/v.
# encap = gpb
#
# Server to connect to
#
server = 1.78.28.103:57400 à router IP:GRPC port
#
# Subscription IDs to subscribe to, as configured on the router. Set
# is specified as a comma separated string of subscription ID names.
#
subscriptions = 1minute à Subscriptions can we added with a , delimeter
#
#
# Access control options; TLS, username/password
#
# TLS enabled or not, CA cert in PEM, and server name to accept.
#
# tls = false
# tls_pem = ca.pem
# tls_servername = tlsservername
#
# username/password
#
# Public key encrypted password is specified here. The pem with RSA
# key used to encrypt passwords is passed in as a cli argument.
#
# The encrypted password will be generated if pipeline is run, -pem is
# specified, and password option is NOT included in the section. In
# that user will be prompted for user/pass, and a new separate copy of
# configuration will be created for convenience.
#
# Passphrase-less RSA key pair can be generated using ssh-keygen if
# necessary: 'ssh-keygen -t rsa -b 4096'
#
#username = lab
#password = lab
#
# To enable dumping data as it is rxed, uncomment the following, and
# run with --debug option.
#
# logdata = on
###########################################
After editing the above config in pipeline.conf file just save it. So the streamed data will be in dump.txt. This is based on the directory configured in the pipeline.conf
##############################################
# Example of a tap stage; dump content to file, or at least count messages
#
[inspector]
stage = xport_output
#
# Module type: tap is only supported in xport_output stage currently.
#
type = tap
#
# File to dump decoded messages
#
file = dump.txt
##############################################
Pipeline execution
To start streaming just execute like below. The username and password is tacacs or local which can be authenticated by grpc. If tacacs is enabled then you need to use tacacs username and password else local username password can be used.
[root@csg-tacacs bin]# ./pipeline
Startup pipeline
Load config from [pipeline.conf], logging in [pipeline.log]
CRYPT Client [cx],[1.78.28.103:57400]
Enter username: lab
Enter password:
Wait for ^C to shutdown
All set, just check for dump.txt for the streamed data!
Pipeline execution on mac using docker
Make sure docker is installed in your mac before continuing the below step
docker file location is: https://github.com/cisco/bigmuddy-network-telemetry-pipeline/tree/master/docker
steps to follow:
Download the big muddy zip file from
https://github.com/cisco/bigmuddy-network-telemetry-pipeline
After download unzip and change the directory to docker.
Then execute the following.
AWP-M-72K6:docker mymac$ docker build -t pipeline .
Sending build context to Docker daemon 74.66MB
Step 1/10 : FROM debian:stable-slim
stable-slim: Pulling from library/debian
c262086186dd: Pull complete
Digest: sha256:76e4d780ebdd81315c1d67e0a044fabc06db5805352e3322594360d3990be1b6
Status: Downloaded newer image for debian:stable-slim
---> 414b5dbe710f
Step 2/10 : MAINTAINER Christian Cassar <ccassar@cisco.com>
---> Running in bd11994e835f
Removing intermediate container bd11994e835f
---> 098d36542ca1
Step 3/10 : ADD pipeline.conf /data/pipeline.conf
---> a5baaa3f2833
Step 4/10 : ADD metrics.json /data/metrics.json
---> 3a1150d58676
Step 5/10 : ADD metrics_gpb.json /data/metrics_gpb.json
---> a01a0719ca40
Step 6/10 : ADD pipeline /pipeline
---> db5c9e3b500e
Step 7/10 : VOLUME ["/data"]
---> Running in 03d9203a11fc
Removing intermediate container 03d9203a11fc
---> b7361654df3f
Step 8/10 : WORKDIR /
---> Running in d3869ad4afc3
Removing intermediate container d3869ad4afc3
---> 79538467f7fe
Step 9/10 : ENTRYPOINT ["/pipeline"]
---> Running in 9df783306916
Removing intermediate container 9df783306916
---> e3954f0d5584
Step 10/10 : CMD ["-log=/data/pipeline.log","-config=/data/pipeline.conf"]
---> Running in 0744900c0486
Removing intermediate container 0744900c0486
---> 17f5793f23f7
Successfully built 17f5793f23f7
Successfully tagged pipeline:latest
NARVENKA-M-72K6:docker venkatnagarajan$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
pipeline latest 17f5793f23f7 11 seconds ago 130MB
debian stable-slim 414b5dbe710f 2 days ago 55.3MB
AWP-M-72K6:docker mymac$
Just run pipeline using docker:
AWP-M-72K6:docker mymac$ docker run --name pipeline pipeline
Startup pipeline
Load config from [/data/pipeline.conf], logging in [/data/pipeline.log]
Wait for ^C to shutdown
All set now.
