01-28-2021 05:53 AM
Hi, We are working on a proof of concept for staging C9300 switches via ZTP.
We tested out everything in the lab first so we knew we had a valid DHCP configuration and Python script.
When we unboxed some of the new switches and attempted to provision them we consistently ran into the same issue across several new switches.
They would boot up, pull a dhcp lease, pull the ztp python script and then hang. Out of the box these C9300 were running Cisco IOS XE Software, Version 16.12.04. We fell back to a simpler python script that just executed show commands but saw the same behavior.
It would hang here.
--- System Configuration Dialog --- Would you like to enter the initial configuration dialog? [yes/no]: Loading http://192.168.1.33/ztp_python_base.py Loading http://192.168.1.33/ztp_python_base.py Loading http://192.168.1.33/ztp_python_base.py Loading http://192.168.1.33/ztp_python_base.py The boot up sequence would hang here so we would then power cycle the switch: Initializing Hardware...
It seemed as though guestshell would never run.
We finally got things working by going into each switch and entering iox.
Switch#sh iox detail IOx Infrastructure Summary: --------------------------- IOx service (CAF) : Not Running IOx service (HA) : Not Running IOx service (IOxman) : Not Running Libvirtd : Running Dockerd : Not Running Switch(config)#iox Switch(config)# *Jan 27 15:37:37.514: %UICFGEXP-6-SERVER_NOTIFIED_START: Switch 1 R0/0: psd: Server iox has been notified to start sh iox det Switch(config)#do sh iox detail IOx Infrastructure Summary: --------------------------- IOx service (CAF) 1.8.0.5 : Running IOx service (HA) : Not Running IOx service (IOxman) : Not Ready Libvirtd : Running Dockerd : Running
We would wait a few minutes for the Docker service to come up and then wr erase (thus removing the iox command from any configuration) and start the process again and then everything worked as expected.
--- System Configuration Dialog --- Would you like to enter the initial configuration dialog? [yes/no]: Loading http://192.168.1.33/ztp.py Loading http://192.168.1.33/ztp.py day0guestshell installed successfully Current state is: DEPLOYED day0guestshell activated successfully Current state is: ACTIVATED day0guestshell started successfully Current state is: RUNNING Guestshell enabled successfully HTTP server statistics: Accepted connections total: 0====== STARTING ZTP INITIALIZATION SCRIPT ====== === Obtaining serial number of device.. === --- FOC2344L33F --- Setting configuration file variable.. --- --- FOC2344L33F.cfg ====== Software Version Check Cisco IOS XE Software, Version 16.12.04 ====== --- No upgrade is required!!! --- <snip successful execution of entire python script>
I'm trying to understand what went wrong. I can't imaging that ZTP requires that one console into each switch and run the iox command. I saw a similar posting ( too late) that indicated ZTP required DNS servers. Was it as simple as that and not providing DNS servers in the DHCP lease prevented guestshell from coming up? If so, that was not the experience in the lab (and in fact the lab DHCP server also does not set DNS servers) but in the lab we did not use a brand new out of the box switch.
Having to console into each switch to execute the iox command makes ZTP far less attractive so I'm hoping to ge a better understanding on what went wrong and why from this community!
Thanks in advance
Solved! Go to Solution.
01-30-2021 01:16 PM
Hi @Claudia de Luna and @bigevilbeard you may be hitting CSCvw63161 with 16.12.4 ZTP/Guestshell, there is a workaround with DHCP AutoInstall (for "low" touch) and the manual workaround is:
mkdir flash:guest-share write erase reload
01-28-2021 07:06 AM
Hey @Claudia de Luna i am not sure of your issue here, will leave for others to comment - have you looked at https://developer.cisco.com/codeexchange/github/repo/tdorssers/ztp wondered if this would help here?
Hope this helps.
01-29-2021 10:37 AM
Hey @bigevilbeard , I had not seen that and I love it! Thank you. I think my issue is a bit more fundamental in that if Guestshell does not spin up we are dead in the water. Not sure why we would need to console into each brand new switch and enable iox, defeats the purpose of ZTP. I'm hoping I'm doing something dumb!
In one of the Cisco Live hands on labs the guide states:
The device locates a DHCP server, bootstraps itself with its interface IP address, gateway, and DNS server IP address, and enables Guest Shell. The device then obtains the IP address or URL of a TFTP server and downloads a Python script to configure the device.
So that may very well be it. Unfortunately the 30 "new out of the box" switches all have their bootstrap configs now so I can't verify this and it does not really explain why entering the iox command would resolve this. As of yet, I've not updated the DHCP scope to provide a DNS server.
Thanks Stuart!
01-30-2021 01:16 PM
Hi @Claudia de Luna and @bigevilbeard you may be hitting CSCvw63161 with 16.12.4 ZTP/Guestshell, there is a workaround with DHCP AutoInstall (for "low" touch) and the manual workaround is:
mkdir flash:guest-share write erase reload
01-30-2021 01:21 PM
Hey @jcohoe,
Thanks for explaining this! Just so you know, we consoled into each switch and executed the iox command and then we did a write erase and reload. That allowed the ZTP process to actually execute.
I'll take a look at DHCP Autoinstall to just do the bootstrap on the mgmt port and aaa and then do the rest via SSH.
Thanks again!
10-12-2021 10:36 PM
I ran into basically the same issue. The 9300 series switch with the 'standard ios loaded', failed after a few days of testing.
I noticed that the script was downloaded and then immediately finishes. The guestshell wasn't loaded after DHCP and receiving the script.
I thought/ hoped that the ZTP would do the job, but if it doesn't then the objective is difficult to achieve.
Would you like to enter the initial configuration dialog? [yes/no]:
Loading 9K/python_script.py from 10.15.255.52 (via Vlan990): !
[OK - 8646 bytes]
Press RETURN to get started!
01-31-2021 01:52 AM
Thanks @jcohoe
12-07-2021 07:01 AM
Hi All,
I have been playing with this this week. I was using the script of Tim Dorssers https://github.com/tdorssers/ztp. It works like a charm now, he fixed some bugs for me. The issue is with IOS 16.12 it does not work for ZTP. Tim wrote an "auto install" version to by pass this problem. It is a TCL based script which you load via TFTP. Anyhow for you must start the switch normally first and apply the explanation of jcohoe. (make sure the switch has no config when you reload) After that it should work.
So, first you need to enable DHCP on your server. The DHCP server will provide the router an IP address for the TFTP server (option 150). The DHCP server will also provide the router with the name of a script (option 67) called "kickstart-conf". The script must be in your TFTP directory. This script can be found the autoinstall directory of the ZTP-tool of Tim. If you look at the details of this script you can see that it uses eventmanger to download another TCL-script "script.tcl" make sure you move this script one level up in the directory.
option ip-tftp-server code 150 = { ip-address }; subnet 10.1.1.0 netmask 255.255.255.0 { range 10.1.1.2 10.1.1.255; option routers 10.1.1.1; option domain-name-servers 10.1.1.1; option ip-tftp-server 10.1.1.1; option bootfile-name "kickstart-conf"; }
In the script.tcl you need to update the IP-address for: SYSLOG, LOGAPI and JSON to the IP of your machine. And of course you need to have a TFTP-server running.
You need to be patient, it takes some time for the router to download and install the new software. Like it does when you to it manually.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide