12-08-2016 07:29 AM - edited 03-01-2019 04:34 AM
Hi,
We are working on building out a proof of concept for zero touch provisioning and would like to focus on APIC-EM.
We have a temp server that we are using for a single APIC-EM VM. Its an ESXi 5.5 host. Along with that we have a MS DHCP server configured to provide option 43. We are running 1.3.1.9 as a standalone server.
Eth0 of the host goes to an external network with services (NTP etc.), Eth1 goes to the private "Provisioning Network". The APIC-EM VM is similarly configured so that it is accessible for scripting etc on its first interface and has access to the private network on the second interface.
Everything seems to be working as expected except that we can't seem to successfully use the PnP application. I've tried disabling the "external interface" so that only the private network is available thinking the network interfaces were more 'HA' /NIC Teaming but that did not make any difference.
We've tried a variety of devices which all meet the minimum requirements for hardware and software
2901 ISR (Gen2)
2960S Switch
3650 Switch
We've tried projects as well as just seeing they will be "discovered" without success.
The devices do get an IP address and they start the AutoInstall process and the APIC-EM never recognizes them, pre-provisioined or not.
Here is the log from the 2960S
```
*Mar 1 00:02:26.098: %SYS-5-RESTART: System restarted --
Cisco IOS Software, C2960S Software (C2960S-UNIVERSALK9-M), Version 12.2(58)SE2, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2011 by Cisco Systems, Inc.
Compiled Thu 21-Jul-11 02:22 by prod_rel_team
*Mar 1 00:02:27.634: %LINK-3-UPDOWN: Interface FastEthernet0, changed state to down
to up
*Mar 1 00:02:29.107: %USB_CONSOLE-6-MEDIA_RJ45: Console media-type is RJ45.
*Mar 1 00:02:30.444: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to up
*Mar 1 00:02:58.446: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to up
*Mar 1 00:03:09.215: AUTOINSTALL: Vlan1 is assigned 192.0.2.102 got vend id vend spec. info ret: succeed
*Mar 1 00:03:19.224: AUTOINSTALL: Obtain siaddr 192.0.2.100 (as config server) <--.100 is the DHCP server
%Error opening tftp://192.0.2.100/network-confg (Timed out)
%Error opening tftp://192.0.2.100/cisconet.cfg (Timed out)
%Error opening tftp://192.0.2.100/router-confg (Timed out)
%Error opening tftp://192.0.2.100/ciscortr.cfg (Timed out)
%Error opening tftp://192.0.2.100/network-confg (Timed out)
%Error opening tftp://192.0.2.100/cisconet.cfg (Timed out)
%Error opening tftp://192.0.2.100/router-confg (Timed out)
```
Any suggestions on where to look for issues would be very welcome!
Thanks!
Solved! Go to Solution.
12-08-2016 02:03 PM
Hi Claudia (and anyone else reading this thread),
this is the scenario i use every day in my lab.
Some troubleshooting tips (I will probably blogs these if there is interest).
1) Connect to the PnP switch and determine the following:
pnp profile manual-test
transport http ipv4 x.x.x.x port 80
you can then use the "debug pnp all" command to get great insight into what is going on
Adam
12-08-2016 09:56 AM
Hi
First off what application are you using? PnP? IWAN?
From what you posted it looks to me like post provisioning, the device does not have a route back to 192.0.2.100. Is that APIC-EM or a different tftp server? If different, did you verify that tftp is running and that no ACL or firewall rules drop the traffic?
Thanks
12-08-2016 10:10 AM
Hi Nicholas,
Sorry...should have mentioned...PnP.
12-08-2016 10:17 AM
I have taking ownership of the TAC case you have opened.
12-08-2016 02:03 PM
Hi Claudia (and anyone else reading this thread),
this is the scenario i use every day in my lab.
Some troubleshooting tips (I will probably blogs these if there is interest).
1) Connect to the PnP switch and determine the following:
pnp profile manual-test
transport http ipv4 x.x.x.x port 80
you can then use the "debug pnp all" command to get great insight into what is going on
Adam
12-08-2016 02:12 PM
Adam! The man himself...I've been listening to you alot lately! (Youtube and CiscoLive)
thanks for chiming in!
Nick and I just took a quick look.
On the 2960S switch, even though it running the minimum code it does not look like pnp is on there, so that may explain that.
On the ISR Gen2 2901, also running > recommended ios, it has ccp ans so Nick mentioned that that will cause it to fail. I'm deleting and retrying.
On the 3650s I may have had an incorrect SN so I'm about to verify and try that again.
I'll be using these commands shortly so thank you and Nick and I will be looking at this later on tonight.
Switch Ports Model SW Version SW Image
------ ----- ----- ---------- ----------
* 1 52 WS-C2960S-48LPS-L 12.2(58)SE2 C2960S-UNIVERSALK9-M
BTW:
12-08-2016 02:28 PM
OK..interesting:
On the Gen2 ISR 2901:
There is no pnp bootstrap and its trying to use DNS to go to the cloud server I believe....?
*Dec 8 22:10:29.439: %SYS-5-RESTART: System restarted --
Cisco IOS Software, C2900 Software (C2900-UNIVERSALK9-M), Version 15.4(3)M1, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2014 by Cisco Systems, Inc.
Compiled Sat 25-Oct-14 03:34 by prod_rel_team
*Dec 8 22:10:30.123: %SYS-6-BOOTTIME: Time taken to reboot after reload = 670 seconds
*Dec 8 22:10:30.735: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is OFF
*Dec 8 22:10:30.735: %CRYPTO-6-GDOI_ON_OFF: GDOI is OFF
*Dec 8 22:10:30.735: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is OFF
*Dec 8 22:10:30.735: %CRYPTO-6-GDOI_ON_OFF: GDOI is OFF
%Error opening tftp://192.0.2.100/router-confg (Timed out)
*Dec 8 22:10:49.551: %PNP-6-HTTP_CONNECTING: PnP Discovery trying to connect to PnP server http://devicehelper.cisco.com/pnp/HELLO
Router>en
Router#sh run | inc pnp
Router#
%Error opening tftp://192.0.2.100/ciscortr.cfg (Timed out)
*Dec 8 22:11:27.551: %PNP-6-HTTP_CONNECTING: PnP Discovery trying to connect to PnP server http://devicehelper.cisco.com/pnp/HELLO
%Error opening tftp://255.255.255.255/network-confg (Timed out)
12-08-2016 02:34 PM
There is a cloud fallback option too. That is in controlled availability today.
What discovery mechanism are you attempting to use for your testing? I just read your earlier post.... looks like option 43.
That should create the pnp profile.
"show pnp trace" is another useful command too.
Adam
12-08-2016 02:35 PM
PnP protocol will be using "time-pnp.cisco.com" and/or "pool.ntp.org" for time sync. Some lab networks block public NTP access, for such DNS mappings to local NTP server is needed.
Does this apply to your case CLaudia? If yes, can you do the needful in your setup?
12-08-2016 02:39 PM
We are going for zero touch. I have 150 3650s that are "fresh out of the box" and we are using them to as an APIC-EM proof of concept for new staging options. So we are using DHCP option 43 to point the devices on the private network to the APIC-EM private network interface.
On NTP, so yes, the provisioning network is totally isolated. I am including the local NTP server in the DHCP offer. Will that make a difference?
12-08-2016 02:41 PM
Thank you for the additional information, Chakrapani. Yes the network is totally isolated. If sending the NTP server in the DHCP offer does not work what would you suggest in the isolated lab. dummy DNS A records?
12-08-2016 02:42 PM
Yes, a dummy record of the following sort in your DNS server:
ip host time-pnp.cisco.com <IP of your choice>
Similar entry for pool.ntp.org I believe.
This way, you can resolve name queries for time-pnp.cisco.com and/or pool.ntp.org with an IP of your choice
12-08-2016 02:49 PM
OK..so on the 2901 ISR, if i use the manual bootstrap that Adam suggested it pops right up now to figure out how to get zero touch going!
Can someone confirm the DHCP string to use (we are using Microsoft 2012 Server)?
12-08-2016 02:58 PM
Hi Claudia,
option 43 needs to be
an ascii string - "5A1N;B2;K4;Iw.x.y.z.;J80"
where "w.x.y.z" is the IP address of your controller.
Adam
NOTE: Initial post had a typo with the "I" for the IP address
12-08-2016 04:42 PM
So a quick update:
root cause seems to be the Microsoft DHCP server.
I took the 2901 ISR router (which worked if I configured the pnp profile manually) and turned it into a DHCP server with the following configuration:
ip dhcp excluded-address 192.168.2.1 192.168.2.100
!
ip dhcp pool PNP_Pool
network 192.168.2.0 255.255.255.0
default-router 192.168.2.1
option 43 ascii "5A1D;B2;K4;I192.0.2.2;J80"
Note: Adam correct me if I am wrong but I believe your string above is missing the "I" before the APIC-EM IP and has a trailing period after the last octet of the IP. If that is true, might be good to correct for others who might find the info.
Has anyone run across instructions on how to set up the Option 43 on an MS DHCP server for PnP?
Test ISR G2 2901
Works with manually configured pnp profile after removing ccp files/directory and now in use as DHCP server so no longer a test client
Test 2960S WS-C2960S-48LPS-L 12.2(58)SE2
- Still not working even though it is running the right code and other devices are now getting the right info from the new IOS DHCP server
Test 3650 #2 "Out of the Box"
Now working in that on boot up I see it in the "Unplanned Devices" section
Note: Had to up the SNMP timeout to 30 seconds before it started working (after I fixed DHCP)
Test 3650 #1 "Out of the Box"
This one is interesting...I had a pre-provisioned record for the 2960S and I initially mistakenly put in the Serial Number of 3650#1. I then went back and fixed it but only after the APIC-EM PnP saw it and started talking to it. I disconnected once i saw that since I didn't want to load a 2960 image onto a 3650. So even now, with the correct SN of the 2960S when I bring up this 3650#1 APIC-EM still sees it as the 2960 and tries to provision it per the "state" I defined in the pre-provisioned tool. . That is, even though I now have a pre-privision record for a 2960S with the right SN when I bring up the 3650 whose SN I had mistakenly put in it tries to provision that 3650 per the 2960S rules I defined.
the APIC dashboard is "stuck" in "Deploying Image" and I can't delete that record from the APIC-EM.
Test2060S | FOC1534Y4ES | WS-C2960S-48LPS | c2960s-universalk9-mz.122-58.SE2.bin | 2016-12-08 16:16:12 (Pacific Standard Time) | Deploying Image |
Note: after it failed, i was able to remove the entire project.
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:
This community is intended for developer topics around Data Center technology and products. If you are looking for a non-developer topic about Data Center, you might find additional information in the Data Center and Cloud community