topic Re: Design Recommendation for APIC-EM Network in Controllers

Design Recommendation for APIC-EM Network

Claudia de Luna — Fri, 01 Mar 2019 12:34:16 GMT

Hi,

We are working on building out a proof of concept for zero touch provisioning and would like to focus on APIC-EM.

We have a temp server that we are using for a single APIC-EM VM. Its an ESXi 5.5 host. Along with that we have a MS DHCP server configured to provide option 43. We are running 1.3.1.9 as a standalone server.

Eth0 of the host goes to an external network with services (NTP etc.), Eth1 goes to the private "Provisioning Network". The APIC-EM VM is similarly configured so that it is accessible for scripting etc on its first interface and has access to the private network on the second interface.

Everything seems to be working as expected except that we can't seem to successfully use the PnP application. I've tried disabling the "external interface" so that only the private network is available thinking the network interfaces were more 'HA' /NIC Teaming but that did not make any difference.

We've tried a variety of devices which all meet the minimum requirements for hardware and software

2901 ISR (Gen2)

2960S Switch

3650 Switch

We've tried projects as well as just seeing they will be "discovered" without success.

The devices do get an IP address and they start the AutoInstall process and the APIC-EM never recognizes them, pre-provisioined or not.

Here is the log from the 2960S

```

*Mar 1 00:02:26.098: %SYS-5-RESTART: System restarted --

Cisco IOS Software, C2960S Software (C2960S-UNIVERSALK9-M), Version 12.2(58)SE2, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Compiled Thu 21-Jul-11 02:22 by prod_rel_team

*Mar 1 00:02:27.634: %LINK-3-UPDOWN: Interface FastEthernet0, changed state to down

to up

*Mar 1 00:02:29.107: %USB_CONSOLE-6-MEDIA_RJ45: Console media-type is RJ45.

*Mar 1 00:02:30.444: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to up

*Mar 1 00:02:58.446: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to up

*Mar 1 00:03:09.215: AUTOINSTALL: Vlan1 is assigned 192.0.2.102 got vend id vend spec. info ret: succeed

*Mar 1 00:03:19.224: AUTOINSTALL: Obtain siaddr 192.0.2.100 (as config server) <--.100 is the DHCP server

%Error opening tftp://192.0.2.100/network-confg (Timed out)

%Error opening tftp://192.0.2.100/cisconet.cfg (Timed out)

%Error opening tftp://192.0.2.100/router-confg (Timed out)

%Error opening tftp://192.0.2.100/ciscortr.cfg (Timed out)

%Error opening tftp://192.0.2.100/network-confg (Timed out)

%Error opening tftp://192.0.2.100/cisconet.cfg (Timed out)

%Error opening tftp://192.0.2.100/router-confg (Timed out)

```

Any suggestions on where to look for issues would be very welcome!

Thanks!

Re: Design Recommendation for APIC-EM Network

ngoldwat — Thu, 08 Dec 2016 17:56:40 GMT

First off what application are you using? PnP? IWAN?

From what you posted it looks to me like post provisioning, the device does not have a route back to 192.0.2.100. Is that APIC-EM or a different tftp server? If different, did you verify that tftp is running and that no ACL or firewall rules drop the traffic?

Thanks

Re: Design Recommendation for APIC-EM Network

Claudia de Luna — Thu, 08 Dec 2016 18:10:21 GMT

Hi Nicholas,

Sorry...should have mentioned...PnP.

Re: Design Recommendation for APIC-EM Network

ngoldwat — Thu, 08 Dec 2016 18:17:47 GMT

I have taking ownership of the TAC case you have opened.

Re: Design Recommendation for APIC-EM Network

aradford — Thu, 08 Dec 2016 22:03:10 GMT

Hi Claudia (and anyone else reading this thread),

this is the scenario i use every day in my lab.

Some troubleshooting tips (I will probably blogs these if there is interest).

1) Connect to the PnP switch and determine the following:

the switch has a valid IP address
the switch can ping the controller (by the appropriate IP address -- internal vs external)
that the PnP profile has been established on the switch (which discovery mechanism are you using) - "show run | inc pnp" should show a "pnp profile pnp-zero-touch"
If there is an issue with discovery, you can create the pap-profile manually to test the rest of the process

pnp profile manual-test

transport http ipv4 x.x.x.x port 80

you can then use the "debug pnp all" command to get great insight into what is going on

Adam

Re: Design Recommendation for APIC-EM Network

Claudia de Luna — Thu, 08 Dec 2016 22:12:59 GMT

Adam! The man himself...I've been listening to you alot lately! (Youtube and CiscoLive)

thanks for chiming in!

Nick and I just took a quick look.

On the 2960S switch, even though it running the minimum code it does not look like pnp is on there, so that may explain that.

On the ISR Gen2 2901, also running > recommended ios, it has ccp ans so Nick mentioned that that will cause it to fail. I'm deleting and retrying.

On the 3650s I may have had an incorrect SN so I'm about to verify and try that again.

I'll be using these commands shortly so thank you and Nick and I will be looking at this later on tonight.

Switch Ports Model SW Version SW Image

------ ----- ----- ---------- ----------

* 1 52 WS-C2960S-48LPS-L 12.2(58)SE2 C2960S-UNIVERSALK9-M

BTW:

the switch has a valid IP address - Yes it does so DHCP is working TBD on the Opt 43
the switch can ping the controller (by the appropriate IP address -- internal vs external) It can on the internal

I will check these in a few:
that the PnP profile has been established on the switch (which discovery mechanism are you using) - "show run | inc pnp" should show a "pnp profile pnp-zero-touch"
If there is an issue with discovery, you can create the pap-profile manually to test the rest of the process

Re: Design Recommendation for APIC-EM Network

Claudia de Luna — Thu, 08 Dec 2016 22:28:23 GMT

OK..interesting:

On the Gen2 ISR 2901:

There is no pnp bootstrap and its trying to use DNS to go to the cloud server I believe....?

*Dec 8 22:10:29.439: %SYS-5-RESTART: System restarted --

Cisco IOS Software, C2900 Software (C2900-UNIVERSALK9-M), Version 15.4(3)M1, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Compiled Sat 25-Oct-14 03:34 by prod_rel_team

*Dec 8 22:10:30.123: %SYS-6-BOOTTIME: Time taken to reboot after reload = 670 seconds

*Dec 8 22:10:30.735: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is OFF

*Dec 8 22:10:30.735: %CRYPTO-6-GDOI_ON_OFF: GDOI is OFF

*Dec 8 22:10:30.735: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is OFF

*Dec 8 22:10:30.735: %CRYPTO-6-GDOI_ON_OFF: GDOI is OFF

%Error opening tftp://192.0.2.100/router-confg (Timed out)

*Dec 8 22:10:49.551: %PNP-6-HTTP_CONNECTING: PnP Discovery trying to connect to PnP server http://devicehelper.cisco.com/pnp/HELLO

Router>en

Router#sh run | inc pnp

Router#

%Error opening tftp://192.0.2.100/ciscortr.cfg (Timed out)

*Dec 8 22:11:27.551: %PNP-6-HTTP_CONNECTING: PnP Discovery trying to connect to PnP server http://devicehelper.cisco.com/pnp/HELLO

%Error opening tftp://255.255.255.255/network-confg (Timed out)

Re: Design Recommendation for APIC-EM Network

aradford — Thu, 08 Dec 2016 22:31:55 GMT

Thanks Claudia :-),

glad it is under control. Good catch on three common issues. CCP is also a common gotcha.

Let us know if there is anything else we can help with?

Not sure if you have seen my blogs, but sometimes that can also help. There are quite a few on different pnp deployment models

My Blog Index

Adam

Re: Design Recommendation for APIC-EM Network

aradford — Thu, 08 Dec 2016 22:34:06 GMT

There is a cloud fallback option too. That is in controlled availability today.

What discovery mechanism are you attempting to use for your testing? I just read your earlier post.... looks like option 43.

That should create the pnp profile.

"show pnp trace" is another useful command too.

Adam

Re: Design Recommendation for APIC-EM Network

cchitnis — Thu, 08 Dec 2016 22:35:47 GMT

PnP protocol will be using "time-pnp.cisco.com" and/or "pool.ntp.org" for time sync. Some lab networks block public NTP access, for such DNS mappings to local NTP server is needed.

Does this apply to your case CLaudia? If yes, can you do the needful in your setup?

Re: Design Recommendation for APIC-EM Network

Claudia de Luna — Thu, 08 Dec 2016 22:39:06 GMT

We are going for zero touch. I have 150 3650s that are "fresh out of the box" and we are using them to as an APIC-EM proof of concept for new staging options. So we are using DHCP option 43 to point the devices on the private network to the APIC-EM private network interface.

On NTP, so yes, the provisioning network is totally isolated. I am including the local NTP server in the DHCP offer. Will that make a difference?

Re: Design Recommendation for APIC-EM Network

Claudia de Luna — Thu, 08 Dec 2016 22:41:03 GMT

Thank you for the additional information, Chakrapani. Yes the network is totally isolated. If sending the NTP server in the DHCP offer does not work what would you suggest in the isolated lab. dummy DNS A records?

Re: Design Recommendation for APIC-EM Network

cchitnis — Thu, 08 Dec 2016 22:42:55 GMT

Yes, a dummy record of the following sort in your DNS server:

ip host time-pnp.cisco.com <IP of your choice>

Similar entry for pool.ntp.org I believe.

This way, you can resolve name queries for time-pnp.cisco.com and/or pool.ntp.org with an IP of your choice

Re: Design Recommendation for APIC-EM Network

Claudia de Luna — Thu, 08 Dec 2016 22:49:16 GMT

OK..so on the 2901 ISR, if i use the manual bootstrap that Adam suggested it pops right up now to figure out how to get zero touch going!

Can someone confirm the DHCP string to use (we are using Microsoft 2012 Server)?

Re: Design Recommendation for APIC-EM Network

aradford — Thu, 08 Dec 2016 22:58:23 GMT

Hi Claudia,

option 43 needs to be

an ascii string - "5A1N;B2;K4;Iw.x.y.z.;J80"

where "w.x.y.z" is the IP address of your controller.

Adam

NOTE: Initial post had a typo with the "I" for the IP address

Re: Design Recommendation for APIC-EM Network

Claudia de Luna — Fri, 09 Dec 2016 00:42:33 GMT

So a quick update:

root cause seems to be the Microsoft DHCP server.

I took the 2901 ISR router (which worked if I configured the pnp profile manually) and turned it into a DHCP server with the following configuration:

ip dhcp excluded-address 192.168.2.1 192.168.2.100

ip dhcp pool PNP_Pool

network 192.168.2.0 255.255.255.0

default-router 192.168.2.1

option 43 ascii "5A1D;B2;K4;I192.0.2.2;J80"

Note: Adam correct me if I am wrong but I believe your string above is missing the "I" before the APIC-EM IP and has a trailing period after the last octet of the IP. If that is true, might be good to correct for others who might find the info.

Has anyone run across instructions on how to set up the Option 43 on an MS DHCP server for PnP?

Test ISR G2 2901

Works with manually configured pnp profile after removing ccp files/directory and now in use as DHCP server so no longer a test client

Test 2960S WS-C2960S-48LPS-L 12.2(58)SE2

- Still not working even though it is running the right code and other devices are now getting the right info from the new IOS DHCP server

Test 3650 #2 "Out of the Box"

Now working in that on boot up I see it in the "Unplanned Devices" section

Note: Had to up the SNMP timeout to 30 seconds before it started working (after I fixed DHCP)

Test 3650 #1 "Out of the Box"

This one is interesting...I had a pre-provisioned record for the 2960S and I initially mistakenly put in the Serial Number of 3650#1. I then went back and fixed it but only after the APIC-EM PnP saw it and started talking to it. I disconnected once i saw that since I didn't want to load a 2960 image onto a 3650. So even now, with the correct SN of the 2960S when I bring up this 3650#1 APIC-EM still sees it as the 2960 and tries to provision it per the "state" I defined in the pre-provisioned tool. . That is, even though I now have a pre-privision record for a 2960S with the right SN when I bring up the 3650 whose SN I had mistakenly put in it tries to provision that 3650 per the 2960S rules I defined.

the APIC dashboard is "stuck" in "Deploying Image" and I can't delete that record from the APIC-EM.

Test2060S

FOC1534Y4ES

WS-C2960S-48LPS

c2960s-universalk9-mz.122-58.SE2.bin

2016-12-08 16:16:12 (Pacific Standard Time)

Deploying Image

Note: after it failed, i was able to remove the entire project.

Re: Design Recommendation for APIC-EM Network

aradford — Fri, 09 Dec 2016 01:40:03 GMT

Sorry about typo.. i have corrected that now.

a couple more questions:

1) for the 2960S, can you manually create a pnp profile? does the option 43 get received by the device?

2) 3650#2. SNMP should not be required until after the PnP process is complete (the controller adds it to the inventory). Are you using VLAN 1 for management, and what version of code on the 3650.

3) 3650#1. A rule binds a serial number to an image/config file. if the deployment fails you need to wait for an error timeout (16mins for config). Once this occurs you can remove the rule and start again.

Adam

Re: Design Recommendation for APIC-EM Network

Claudia de Luna — Fri, 09 Dec 2016 01:53:00 GMT

going from bottom to top starting with your 3)

3) Thanks for the details on the timeout. I think that is consistent with what I saw.

2) I am using Vlan1 - trying to keep it very simple.

Not sure why changing the SNMP to 30sec should have made a difference. Possible I didn't wait long enough....I tried it with the default value and it never came up in APIC-EM. Once I updated that it seemed to come right up.

1) So this one is still not working. I've upgraded it to 15 code and now the pnp commands are there (no debug) but when I manually configure it just keeps trying the APIC-EM but never gets a response. It does get an IP from the IOS DHCP server so I'm assuming it gets DHCP option 43 as the other devices seem to be now but I've not done any packet captures.

Switch(config)#

Switch(config)#pnp profile manual-test

Switch(config-pnp-init)#transport http ipv4 192.0.2.2 port 80

Switch(config-pnp-init)#end

Switch#d

Mar 30 01:35:35.988: %SYS-5-CONFIG_I: Configured from console by consolee

% Ambiguous command: "de"

Switch#debug pnp ?

% Unrecognized command

Switch#debug pnp

% Invalid input detected at '^' marker.

Switch#show pnp profile

Initiator Profile manual-test: 0 open connections: 0 closing connections

Encap: pnp

WSSE header is not required. Configured authorization level is 1

Max message (RX) is 50 Kbytes

XEP Faults are sent

Idle timeout infinite

Keepalive not configured

Reconnect time 60 seconds

Primary transport: http to host 192.0.2.2, port 80, URL onplusops/WORK-REQUEST

Not connected, next reconnect attempt in 41 seconds

Switch#

Re: Design Recommendation for APIC-EM Network

aradford — Fri, 09 Dec 2016 02:16:30 GMT

I just took a closer look at the 2960s.

that version of code is very old.c2960s-universalk9-mz.122-58.SE2.bin (27-JUL-2011). There is no way that will work.

PnP release notes require the following min version of code for 2960S 15.2.2E3, 15.2.3E2, 15.2.4E1

There is a c2960s-universalk9-mz.152-2.E5a.bin (Oct 2016) that should work well (and have debugging)!

unfortunately, you have a "bootstrap" issue. you need to upgrade to 15.2(2)E5a before you can test pnp.

Re: Design Recommendation for APIC-EM Network

Claudia de Luna — Fri, 09 Dec 2016 02:26:59 GMT

Thank you for that!

So I am checking for support info here:

Supported Platforms for the Cisco Application Policy Infrastructure Controller Enterprise Module, Release 1.3.0.x - Cis…

am I reading this wrong? I assumed that Base Apps support included PnP? but maybe not since the other column headings could be included in the "base apps"....

Supported Switches	Minimum Software Version 1	Recommended Software Version	Base Apps Support	Path Trace Support	EasyQoS Support	Path Stats Interface	Path Stats QoS
Catalyst 2960-S Series switches, including stacks	>=12.1	Cisco IOS 15.2(1)E1, 12.2(58)SE2	Yes	Yes	Yes	Yes	No

Supported Switches

Minimum Software Version

Recommended Software Version

Base Apps

Support

Path Trace

Support

EasyQoS

Support

Path Stats Interface

Path Stats QoS

Catalyst 2960-S Series switches, including stacks

>=12.1

Cisco IOS 15.2(1)E1, 12.2(58)SE2

Yes

Understand the bootstrapping issue...NP...this is for a demo so I'll upgrade to your suggested version.

Thx!