Introduction
ZTP or Zero Touch Provisioning was introduced to reduce the human interaction in the provision of XR devices. ZTP helps in auto provisioning after the software installation of the router using iPXE.
This document is intended for first time deployments, troubleshooting and for lab recreates.
ZTP can perform the following:
- Auto-Configuration update: Download and apply the XR configuration after image installation
- Script execution: Download and execute user defined script file. Various methods/functions can be implemented as a part of script execution:
- Multiple packages update
- SMU installation
- Verifications
eXR decides which of the actions to perform based on contents of the first line in the file.
- If the file starts with “!! IOS XR” it denotes a configuration file, Perform apply_config.
- Similarly, the files starting with “#! /bin/bash” or “#! /bin/sh” or “#!/usr/bin/python”denotes a script file. Perform script execution.
Note1: Support for python is introduced from eXR 6.2.2
Note2: eXR iPXE supports TFTP,FTP & HTTP. HTTPS is not supported as system cannot predict which signature to verify.
Linux Server Configuration
iPXE is an enhancement over PXE require TFTP/FTP/HTTP for image/configuration download and uses DHCP to get/provide information to regarding image & configuration.
DHCP configuration: /etc/dhcp/dhcpd.conf
In the later example, we will review the packet capture to confirm this operation.
HTTP Server requirements
HTTP server must be reachable from the Mgmt Ethernet interface.
Troubleshooting TIPS
After setting up Linux server, Perform DHCP/HTTP server reachability and functionality Check.
In this setup, a single Linux server is being used as DHCP/HTTP server. If you have separate servers for these functions, you may need to verify the below steps on all the servers.
[root@xxxxxxxxxx]# service dhcpd status
Redirecting to /bin/systemctl status dhcpd.service
- dhcpd.service - DHCPv4 Server Daemon
Loaded: loaded (/usr/lib/systemd/system/dhcpd.service; disabled; vendor preset: disabled)
Active: active (running) since Mon 2017-05-29 10:30:59 PDT; 15h ago
Docs: man:dhcpd(8)
man:dhcpd.conf(5)
Main PID: 26913 (dhcpd)
Status: "Dispatching packets..."
CGroup: /system.slice/dhcpd.service
└─26913 /usr/sbin/dhcpd -f -cf /etc/dhcp/dhcpd.conf -user dhcpd -group dhcpd --no-pid
<SNIP>
Hint: Some lines were ellipsized, use -l to show in full.
[root@xxxxxxxx]# service httpd status
Redirecting to /bin/systemctl status httpd.service
- httpd.service - The Apache HTTP Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; vendor preset: disabled)
Active: active (running) since Fri 2017-05-26 05:50:30 PDT; 3 days ago
Docs: man:httpd(8)
man:apachectl(8)
Process: 28088 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=0/SUCCESS)
Process: 11036 ExecReload=/usr/sbin/httpd $OPTIONS -k graceful (code=exited, status=0/SUCCESS)
Main PID: 28095 (httpd)
Status: "Total requests: 0; Current requests/sec: 0; Current traffic: 0 B/sec"
CGroup: /system.slice/httpd.service
├─11037 /usr/sbin/httpd -DFOREGROUND
├─11038 /usr/sbin/httpd -DFOREGROUND
├─11039 /usr/sbin/httpd -DFOREGROUND
├─11040 /usr/sbin/httpd -DFOREGROUND
├─11041 /usr/sbin/httpd -DFOREGROUND
├─26998 /usr/sbin/httpd -DFOREGROUND
├─27426 /usr/sbin/httpd -DFOREGROUND
├─27427 /usr/sbin/httpd -DFOREGROUND
├─27428 /usr/sbin/httpd -DFOREGROUND
├─27889 /usr/sbin/httpd -DFOREGROUND
└─28095 /usr/sbin/httpd -DFOREGROUND
May 26 05:50:30 xxxxx systemd[1]: Starting The Apache HTTP Server...
May 26 05:50:30 xxxxx systemd[1]: Started The Apache HTTP Server.
May 27 03:16:01 xxxxx systemd[1]: Reloaded The Apache HTTP Server.
May 28 03:37:01 xxxxx systemd[1]: Reloaded The Apache HTTP Server.
Enable Router to get IP address using DHCP:
Interface MgmtEth 0/RP0/CPU0/0
Ipv4 address dhcp
Shut/no shut
Verify HTTP server is working by open a browser to the IP address/server name. Ie.http://<ip - servername>/
If DHCP or HTTP fails to work, you may be hitting firewall issues or reachability problems.
To check the firewall properties on the server perform the following commands or add the specific protocols:
Use IP tables command to verify firewall rules:
Iptables –L –n
Chain IN_public_allow (1 references)
target prot opt source destination
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:67 ctstate NEW
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:80 ctstate NEW
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:22 ctstate NEW
Use the following firewall command to list the entries allowed:
[root@xxxxxxxxx ~]# firewall-cmd --list-all
public (active)
target: default
icmp-block-inversion: no
interfaces: enp2s0f0
sources:
services: dhcp dhcpv6-client http ssh
ports:
protocols:
masquerade: no
forward-ports:
sourceports:
icmp-blocks:
rich rules:
Use the following command to permanently allow ports:
firewall-cmd --permanent --add-service=http
firewall-cmd --permanent --add-service=dhcp
Enable Packet capture on DHCP server:
tcpdump –i <interface id> port (bootpc & bootps & port http) –s 0(buffer size) –w <dest. File name>
ex: tcpdump -i enp2s0f0 port 67 or port 68 or port 80 -s 0 -w iPXEboot.pcap
Invoking ZTP
Manually Initiating ZTP
ZTP can be manually invoked using “ZTP initiate” exec CLI. By default, the interface used for ZTP is MGMTEth. If you want to initiate this process on other interface you can use the following option:
ZTP initiate interface <type> <number> <cr>
OR
ZTP initiate <cr>
Boot the router using iPXE
Following the command, router will reload and initiate iPXE boot. From Calvados/SysAdmin console, initiate the following CLI:
RP/0/RP0/CPU0:NCS-5502-A#admin
Tue May 30 22:03:24.462 UTC
root connected from 127.0.0.1 using console on xr-vm_node0_RP0_CPU0
sysadmin-vm:0_RP0# hw-module location all bootmedia network reload ?
Possible completions:
| Output modifiers
<cr>
sysadmin-vm:0_RP0# hw-module location all bootmedia network reload
Note: User-class information is interpreted as "Malformed option" due to bug in wireshark.
After reloading, the router initiates DHCP Discover, notice that user-class is populated with iPXE option:
The DHCP Offer from the server includes the boot file name in Option 67:
Following the response from DHCP Server, Router initiates the image download:
Following a successful image download, Router goes through image installation. This is time consuming and could take several minutes. After the router successfully boots with the downloaded image. It initiates another DHCP request:
In this Discover, you can notice that user-class info includes “*.exr-config”. Since the DHCP is configured to either return “Config file” or “Script” (ie. Else statement in etc/dhcp/dhcpd.conf). DHCP server returns the required file information in option 67:
Post-Install Automation
As explained before, ZTP kicks in after the image installation. In the above example, we see a config download post installation of the script. Alternately, a shell script or python script can be included as a response to the DHCP request by the server.
Various automation can be performed post-installation on the eXR software. Some of the examples are verification of BGP neighbors, routing information etc.
Here in this example, the following script installs all required packages and applies the Config.
#!/bin/bash
#############################################################################
# *** Be careful this is powerful and can potentially destroy your system ***
# *** !!! Use at your own risk !!! ***
#
# * Script file should be saved on the backend HTTP server
# * Take backup of the configuration and save them outside the device
# * Do not try this in live production setup
#
#
# script Action: Install additional packages, SMUs & apply configuration
#
################################################################################
source /disk0:/ztp/ztp_helper.sh
export LOG_FILE=/disk0:/ztp/user-script.log
export HTTP_SERVER=http://172.16.58.115
export SYSLOG_SERVER=172.16.58.115
export SYSLOG_PORT=514
export RPM_PATH=images
export CONFIG_PATH=images
#Config
export INITIAL_CONFIG=NCS-5502-A.cfg
export FINAL_CONFIG=NCS-5502-A.cfg.full
#Packages
ISIS_RPM=ncs5500-isis-1.1.0.0-r612.x86_64.rpm
K9SEC_RPM=ncs5500-k9sec-2.2.0.0-r612.x86_64.rpm
MCAST_RPM=ncs5500-mcast-2.0.0.0-r612.x86_64.rpm
MGBL_RPM=ncs5500-mgbl-3.0.0.0-r612.x86_64.rpm
MPLS_RPM=ncs5500-mpls-2.1.0.0-r612.x86_64.rpm
MPLSTE_RPM=ncs5500-mpls-te-rsvp-2.2.0.0-r612.x86_64.rpm
OSPF_RPM=ncs5500-ospf-1.0.0.0-r612.x86_64.rpm
#SMU1=SMU1
#SMU2=SMU2
declare -a packageList=("${ISIS_RPM}" "${K9SEC_RPM}" "${MCAST_RPM}" "${MGBL_RPM}" \
"${MPLS_RPM}" "${MPLSTE_RPM}" "${OSPF_RPM}")
function ztp_log() {
# Sends logging information to local file and syslog server
syslog "$1"
echo "$(date +"%b %d %H:%M:%S") "$1 >> $LOGFILE
}
function syslog() {
# Sends syslog messages with netcat (nc)
echo "ztp-script: "$1 | nc -u -q1 $SYSLOG_SERVER $SYSLOG_PORT
}
function download_config(){
ip netns exec tpnns /usr/bin/wget ${HTTP_SERVER}/${CONFIG_PATH}/$1 -O /harddisk:/new-config 2>&1
if [[ "$?" != 0 ]]; then
ztp_log "### Error downloading system configuration ###"
else
ztp_log "### Downloading system configuration complete ###";
fi
}
function apply_config(){
# Applies initial configuration
ztp_log "### Applying initial system configuration ###";
xrapply_with_reason "Initial ZTP configuration" /harddisk:/new-config 2>&1 >> $LOGFILE;
ztp_log "### Checking for errors ###";
local config_status=$(xrcmd "show configuration failed");
if [[ $config_status ]]; then
echo $config_status
ztp_log "!!! Error encounter applying configuration file, review the log !!!!";
fi
ztp_log "### Applying system configuration complete ###";
}
function install_pkg_smu(){
#Download packages
for RPM in "${packageList[@]}"
do
ztp_log "Downloading Packages ${RPM}"
ip netns exec tpnns /usr/bin/wget ${HTTP_SERVER}/${RPM_PATH}/${RPM} -O /harddisk:/$RPM 2>&1
if [[ "$?" != 0 ]]; then
ztp_log "### Error downloading $RPM ###"
else
ztp_log "### Downloading $RPM complete ###";
fi
done
xrcmd "install update source /harddisk:/ $K9SEC_RPM $MCAST_RPM $ISIS_RPM $OSPF_RPM $MGBL_RPM $MPLS_RPM $MPLSTE_RPM" 2>&1 >> $LOGFILE
local complete=0
while [ "$complete" = 0 ]; do
complete=`xrcmd "show install active" | grep k9sec | head -n1 | wc -l`
ztp_log "Waiting for k9sec package to be activated"
sleep 5
done
xrcmd "install commit" 2>&1 >> $LOGFILE
ztp_log "### XR Package Commit"
rm -f /harddisk:/$K9SEC_RPM /harddisk:/$MCAST_RPM /harddisk:/$MCAST_RPM /harddisk:/$ISIS_RPM /harddisk:/$OSPF_RPM /harddisk:/$MGBL_RPM /harddisk:/$MPLSTE_RPM /harddisk:/$MPLS_RPM
ztp_log "### XR PACKAGE INSTALL COMPLETE ###"
}
ztp_log "Updating Packages for XR"
download_config ${INITIAL_CONFIG};
apply_config;
install_pkg_smu;
download_config ${FINAL_CONFIG};
apply_config;
Debugging:
To troubleshoot the script, the above script logs into syslog server as well as to a local file disk0:/ztp/ztp.log.
To view the operation of the script, Open a terminal to the router. type "run" to log into the linux shell & do "tail -f /disk0:/ztp/ztp.log"
Additional resources & references:
https://xrdocs.github.io/software-management/tutorials/2016-08-26-working-with-ztp/
https://xrdocs.github.io/software-management/tutorials/2016-07-27-ipxe-deep-dive/
https://xrdocs.github.io/software-management/blogs/2016-10-14-ios-xr-packages-and-security/
