cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
5305
Views
26
Helpful
21
Replies

Design Recommendation for APIC-EM Network

Claudia de Luna
Spotlight
Spotlight

Hi,

We are working on building out a proof of concept for zero touch provisioning and would like to focus on APIC-EM.

We have a temp server that we are using for a single APIC-EM VM.  Its an ESXi 5.5 host. Along with that we have a MS DHCP server configured to provide option 43.  We are running 1.3.1.9 as a standalone server.

Eth0 of the host goes to an external network with services (NTP etc.), Eth1 goes to the private "Provisioning Network".  The APIC-EM VM is similarly configured so that it is accessible for scripting etc on its first interface and has access to the private network on the second interface.

Everything seems to be working as expected except that we can't seem to successfully use the PnP application.  I've tried disabling the "external interface" so that only the private network is available thinking the network interfaces were more 'HA' /NIC Teaming but that did not make any difference.

We've tried a variety of devices which all meet the minimum requirements for hardware and software

2901 ISR (Gen2)

2960S Switch

3650 Switch

We've tried projects as well as just seeing they will be "discovered" without success.

The devices do get an IP address and they start the AutoInstall process and the APIC-EM never recognizes them, pre-provisioined or not.

Here is the log from the 2960S

```

*Mar  1 00:02:26.098: %SYS-5-RESTART: System restarted --

Cisco IOS Software, C2960S Software (C2960S-UNIVERSALK9-M), Version 12.2(58)SE2, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2011 by Cisco Systems, Inc.

Compiled Thu 21-Jul-11 02:22 by prod_rel_team

*Mar  1 00:02:27.634: %LINK-3-UPDOWN: Interface FastEthernet0, changed state to down

to up

*Mar  1 00:02:29.107: %USB_CONSOLE-6-MEDIA_RJ45: Console media-type is RJ45.

*Mar  1 00:02:30.444: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to up

*Mar  1 00:02:58.446: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to up

*Mar  1 00:03:09.215: AUTOINSTALL: Vlan1 is assigned 192.0.2.102 got vend id vend spec. info ret: succeed

*Mar  1 00:03:19.224: AUTOINSTALL: Obtain siaddr 192.0.2.100 (as config server) <--.100 is the DHCP server

%Error opening tftp://192.0.2.100/network-confg (Timed out)

%Error opening tftp://192.0.2.100/cisconet.cfg (Timed out)

%Error opening tftp://192.0.2.100/router-confg (Timed out)

%Error opening tftp://192.0.2.100/ciscortr.cfg (Timed out)

%Error opening tftp://192.0.2.100/network-confg (Timed out)

%Error opening tftp://192.0.2.100/cisconet.cfg (Timed out)

%Error opening tftp://192.0.2.100/router-confg (Timed out)

```

Any suggestions on where to look for issues would be very welcome!

Thanks!

1 Accepted Solution

Accepted Solutions

aradford
Cisco Employee
Cisco Employee

Hi Claudia (and anyone else reading this thread),

this is the scenario i use every day in my lab.

Some troubleshooting tips (I will probably blogs these if there is interest).

1) Connect to the PnP switch and determine the following:

  • the switch has a valid IP address
  • the switch can ping the controller (by the appropriate IP address -- internal vs external)
  • that the PnP profile has been established on the switch (which discovery mechanism are you using) - "show run | inc pnp" should show a "pnp profile pnp-zero-touch"
  • If there is an issue with discovery, you can create the pap-profile manually to test the rest of the process

pnp profile manual-test

transport http ipv4 x.x.x.x port 80

you can then use the "debug pnp all" command to get great insight into what is going on

Adam

View solution in original post

21 Replies 21

ngoldwat
Level 4
Level 4

Hi

First off what application are you using? PnP? IWAN?

From what you posted it looks to me like post provisioning, the device does not have a route back to 192.0.2.100. Is that APIC-EM or a different tftp server?  If different, did you verify that tftp is running and that no ACL or firewall rules drop the traffic?

Thanks

Hi Nicholas,

Sorry...should have mentioned...PnP.

I have taking ownership of the TAC case you have opened.

aradford
Cisco Employee
Cisco Employee

Hi Claudia (and anyone else reading this thread),

this is the scenario i use every day in my lab.

Some troubleshooting tips (I will probably blogs these if there is interest).

1) Connect to the PnP switch and determine the following:

  • the switch has a valid IP address
  • the switch can ping the controller (by the appropriate IP address -- internal vs external)
  • that the PnP profile has been established on the switch (which discovery mechanism are you using) - "show run | inc pnp" should show a "pnp profile pnp-zero-touch"
  • If there is an issue with discovery, you can create the pap-profile manually to test the rest of the process

pnp profile manual-test

transport http ipv4 x.x.x.x port 80

you can then use the "debug pnp all" command to get great insight into what is going on

Adam

Adam!  The man himself...I've been listening to you alot lately! (Youtube and CiscoLive)

thanks for chiming in!

Nick and I just took a quick look.

On the 2960S switch, even though it running the minimum code it does not look like pnp is on there, so that may explain that.

On the ISR Gen2 2901, also running > recommended ios, it has ccp ans so Nick mentioned that that will cause it to fail.  I'm deleting and retrying.

On the 3650s I may have had an incorrect SN so I'm about to verify and try that again.

I'll be using these commands shortly so thank you and Nick and I will be looking at this later on tonight.

Switch Ports Model              SW Version            SW Image               

------ ----- -----              ----------            ----------             

*    1 52    WS-C2960S-48LPS-L  12.2(58)SE2           C2960S-UNIVERSALK9-M

BTW:

  • the switch has a valid IP address - Yes it does so DHCP is working TBD on the Opt 43
  • the switch can ping the controller (by the appropriate IP address -- internal vs external)  It can on the internal

  • I will check these in a few:
  • that the PnP profile has been established on the switch (which discovery mechanism are you using) - "show run | inc pnp" should show a "pnp profile pnp-zero-touch"
  • If there is an issue with discovery, you can create the pap-profile manually to test the rest of the process

OK..interesting:

On the Gen2 ISR 2901:

There is no pnp bootstrap and its trying to use DNS to go to the cloud server I believe....?

*Dec  8 22:10:29.439: %SYS-5-RESTART: System restarted --

Cisco IOS Software, C2900 Software (C2900-UNIVERSALK9-M), Version 15.4(3)M1, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2014 by Cisco Systems, Inc.

Compiled Sat 25-Oct-14 03:34 by prod_rel_team

*Dec  8 22:10:30.123: %SYS-6-BOOTTIME: Time taken to reboot after reload =  670 seconds

*Dec  8 22:10:30.735: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is OFF

*Dec  8 22:10:30.735: %CRYPTO-6-GDOI_ON_OFF: GDOI is OFF

*Dec  8 22:10:30.735: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is OFF

*Dec  8 22:10:30.735: %CRYPTO-6-GDOI_ON_OFF: GDOI is OFF

%Error opening tftp://192.0.2.100/router-confg (Timed out)

*Dec  8 22:10:49.551: %PNP-6-HTTP_CONNECTING: PnP Discovery trying to connect to PnP server http://devicehelper.cisco.com/pnp/HELLO

Router>en

Router#sh run | inc pnp

Router#

%Error opening tftp://192.0.2.100/ciscortr.cfg (Timed out)

*Dec  8 22:11:27.551: %PNP-6-HTTP_CONNECTING: PnP Discovery trying to connect to PnP server http://devicehelper.cisco.com/pnp/HELLO

%Error opening tftp://255.255.255.255/network-confg (Timed out)

There is a cloud fallback option too.  That is in controlled availability today.

What discovery mechanism are you attempting to use for your testing?  I just read your earlier post.... looks like option 43.

That should create the pnp profile.

"show  pnp trace" is another useful command too.

Adam

PnP protocol will be using "time-pnp.cisco.com" and/or "pool.ntp.org"  for time sync.   Some lab networks block public NTP access, for such DNS mappings to local NTP server is needed.

Does this apply to your case CLaudia? If yes, can you do the needful in your setup?

We are going for zero touch.  I have 150 3650s that are "fresh out of the box" and we are using them to as an APIC-EM proof of concept for new staging options.   So we are using DHCP option 43 to point the devices on the private network to the APIC-EM private network interface.

On NTP, so yes, the provisioning network is totally isolated.  I am including the local NTP server in the DHCP offer. Will that make a difference?

Thank you for the additional information, Chakrapani.  Yes the network is totally isolated.  If sending the NTP server in the DHCP offer does not work what would you suggest in the isolated lab.  dummy DNS A records?

Yes, a dummy record of the following sort in your DNS server:

ip host time-pnp.cisco.com <IP of your choice>

Similar entry for pool.ntp.org I believe.

This way, you can resolve name queries for time-pnp.cisco.com and/or pool.ntp.org with an IP of your choice

OK..so on the 2901 ISR, if i use the manual bootstrap that Adam suggested it pops right up now to figure out how to get zero touch going!

Can someone confirm the DHCP string to use (we are using Microsoft 2012 Server)?

Hi Claudia,

option 43 needs to be

an ascii string - "5A1N;B2;K4;Iw.x.y.z.;J80"


where "w.x.y.z" is the IP address of your controller.


Adam


NOTE:  Initial post had a typo with the "I" for the IP address

So a quick update:

root cause seems to be the Microsoft DHCP server.

I took the 2901 ISR router (which worked if I configured the pnp profile manually) and turned it into a DHCP server with the following configuration:

ip dhcp excluded-address 192.168.2.1 192.168.2.100

!

ip dhcp pool PNP_Pool

network 192.168.2.0 255.255.255.0

default-router 192.168.2.1

option 43 ascii "5A1D;B2;K4;I192.0.2.2;J80"

Note: Adam correct me if I am wrong but I believe your string above is missing the "I" before the APIC-EM IP and has a trailing period after the last octet of the IP.   If that is true, might be good to correct for others who might find the info.

Has anyone run across instructions on how to set up the Option 43 on an MS DHCP server for PnP?

Test ISR G2 2901

Works with manually configured pnp profile after removing ccp files/directory and now in use as DHCP server so no longer a test client

Test 2960S  WS-C2960S-48LPS-L  12.2(58)SE2

- Still not working even though it is running the right code and other devices are now getting the right info from the new IOS DHCP server

Test 3650 #2 "Out of the Box"

Now working in that on boot up I see it in the "Unplanned Devices" section

Note: Had to up the SNMP timeout to 30 seconds before it started working (after I fixed DHCP)

Test 3650 #1 "Out of the Box"

This one is interesting...I had a pre-provisioned record for the 2960S and I initially mistakenly put in the Serial Number of 3650#1.  I then went back and fixed it but only after the APIC-EM PnP saw it and started talking to it.  I disconnected once i saw that since I didn't want to load a 2960 image onto a 3650.  So even now, with the correct SN of the 2960S when I bring up this 3650#1 APIC-EM still sees it as the 2960 and tries to provision it per the "state" I defined in the pre-provisioned tool. . That is, even though I now have a pre-privision record for a 2960S with the right SN when I bring up the 3650 whose SN I had mistakenly put in it tries to provision that 3650 per the 2960S rules I defined.

the APIC dashboard is "stuck"  in "Deploying Image" and I can't delete that record from the APIC-EM.

Test2060S

FOC1534Y4ESWS-C2960S-48LPSc2960s-universalk9-mz.122-58.SE2.bin2016-12-08 16:16:12 (Pacific Standard Time)Deploying Image

Note: after it failed, i was able to remove the entire project.

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:


This community is intended for developer topics around Data Center technology and products. If you are looking for a non-developer topic about Data Center, you might find additional information in the Data Center and Cloud community