topic Re: Need suggestion for PNP solution in Controllers

Need suggestion for PNP solution

JPvisltgb — Fri, 01 Mar 2019 12:39:29 GMT

Hi,

I need some suggestions for PnP solution. I have many branch offices.

Branch office it is simple topology Router Gi0/1 ---> Gi0/49 Switch.

Router is connected to HQ through MPLS network. DHCP server is in the HQ.

So we are planning to start PnP solution using Apic-EM for all network hardware in branch office.

Scenario:

First part:

Empty router is connected to ISP CPE router. CPE router is relaying DHCP requests from router to central DHCP. Router getting DHCP configuration. After that it is connecting to APIC-EM using url pnpserver.domain.com and getting all config.

Second part:

After router succesfully provisioned it become "router-on-stick". It have several subinterfaces with tagged traffic (Management, Users and Phones)

And here we have a problem:

Empty switch which is connected to router by default is trying to get DHCP to Vlan1 but router doesn't have untagged vlan on its link anymore.

We came up with one solution:

We created additional network (and new DHCP scope) for switch deployment. On Branch router on Gi0/1 we configured IP address and ip helper to our DHCP for untagged traffic. So switch can get ip and other config from newly created deployment DHCP scope and then it connects to pnpserver.

In total we are using additional subnet and configuration on router and DHCP server.

So maybe there is more elegant solution for that?

One more question:

After provisioning completed I have many messages in router :

Jul 12 09:47:08.305 CEST: %XML-UPDOWN: pnp-zero-touch XML Interface(101) UP. PID=218

Jul 12 09:47:08.306 CEST: %XML-UPDOWN: pnp-zero-touch XML Interface(101) DOWN(502).

What should be done to remove these messages ?

Re: Need suggestion for PNP solution

Seb Rupik — Wed, 12 Jul 2017 13:04:18 GMT

Hi Jegor,

Have you tired configuring the command 'pnp startup vlan X' on the router? In your case X will represent your management VLAN ID. Obviously your router needs to support the Open Plug-n-Play agent for this to work:

Cisco Open Plug-n-Play Agent Configuration Guide, Cisco IOS XE Release 3E - Cisco

Not sure about those other messages you are seeing.

cheers,

Seb.

Re: Need suggestion for PNP solution

aradford — Wed, 12 Jul 2017 15:07:15 GMT

Nice suggestion Seb. "pnp startup-vlan" should work, unless it is not supported.

In terms of the messages, do you have pnp debug enabled?

What is the dhcp string you are using for option 43?

You have two options,

1) turn off pnp debug, probably by changing option 43, depending on what you put in it.

2) turn off pnp agent. You can do put "no pnp profile XXX" where XXX= the pnp profile name

Adam

Re: Need suggestion for PNP solution

ngoldwat — Wed, 12 Jul 2017 16:53:45 GMT

What are you actually using for DHCP? Windows? Blue Coat?

Re: Need suggestion for PNP solution

JPvisltgb — Thu, 13 Jul 2017 07:04:57 GMT

We are using Microsoft DHCP.

I don't use option 43 in DHCP. I'm using DNS method.

I found out that messages in console log while deployment config:

.Jul 13 07:58:48.881 CEST: %XML-SRVC: urn:cisco:pnp:config-upgrade XML Service(212) FAILURE(712). PID=609

Jul 13 08:54:32.598 CEST: %XML-UPDOWN: pnp-zero-touch XML Interface(101) DOWN(502).

Jul 13 08:55:32.598 CEST: %XML-UPDOWN: pnp-zero-touch XML Interface(101) UP. PID=609

Jul 13 08:55:32.600 CEST: %XML-UPDOWN: pnp-zero-touch XML Interface(101) DOWN(502).

Jul 13 08:56:32.601 CEST: %XML-UPDOWN: pnp-zero-touch XML Interface(101) UP. PID=609

Jul 13 08:56:32.605 CEST: %XML-UPDOWN: pnp-zero-touch XML Interface(101) DOWN(502).

Jul 13 08:57:32.606 CEST: %XML-UPDOWN: pnp-zero-touch XML Interface(101) UP. PID=609

Jul 13 08:57:32.608 CEST: %XML-UPDOWN: pnp-zero-touch XML Interface(101) DOWN(502).

Jul 13 08:58:32.610 CEST: %XML-UPDOWN: pnp-zero-touch XML Interface(101) UP. PID=609

Jul 13 08:58:32.612 CEST: %XML-UPDOWN: pnp-zero-touch XML Interface(101) DOWN(502).

And Config deployment process stuck till time out.

It is happened only on router ISR4321 with IOS-XE 03.16.05.

Re: Need suggestion for PNP solution

aradford — Thu, 13 Jul 2017 07:36:48 GMT

That looks like an issue with the configuration file you are pushing?

Jul 13 07:58:48.881 CEST: %XML-SRVC: urn:cisco:pnp:config-upgrade XML Service(212) FAILURE(712). PID=609

If you click on the serial number in the PnP app, what does the logging history show?

Adam

Re: Need suggestion for PNP solution

JPvisltgb — Thu, 13 Jul 2017 09:25:46 GMT

I double checked config and found one mistake.

But situation the same.

Now I'm getting

.Jul 13 10:24:35.012 CEST: %XML-SRVC: urn:cisco:pnp:config-upgrade XML Service(212) OK. PID=364

.Jul 13 10:24:42.355 CEST: %XML-UPDOWN: pnp-zero-touch XML Interface(101) DOWN(502).

Jul 13 10:25:42.356 CEST: %XML-UPDOWN: pnp-zero-touch XML Interface(101) UP. PID=364

Jul 13 10:25:47.358 CEST: %XML-UPDOWN: pnp-zero-touch XML Interface(101) DOWN(502).

Apic-EM after time out shows me

ERROR_HEALTH_CHECK_TIMER_EXPIRED

Failed health check since device is stuck in non-terminal state PROVISIONING_CONFIG for more than threshold time: 0 hours, 10 minutes, 0 seconds

APIC-EM Version 1.4.1.1159.

Re: Need suggestion for PNP solution

aradford — Thu, 13 Jul 2017 09:30:23 GMT

do you have "aaa authorisation" commands in your configuration?

Re: Need suggestion for PNP solution

JPvisltgb — Thu, 13 Jul 2017 09:32:44 GMT

Yes. I saw issue that old APIC-EM had. But I'm using newer version of Apic-em and router.

Re: Need suggestion for PNP solution

aradford — Thu, 13 Jul 2017 10:02:50 GMT

Yes, it is addressed, but you need IOS-XE 16.3.2 (or later)

You can also address it with an EEM script work around I have documented Network Automation with Plug and Play (PnP) – Part 7

Re: Need suggestion for PNP solution

JPvisltgb — Thu, 13 Jul 2017 11:02:04 GMT

Seems I found problem.

Providers CPE router have ip helper on it interface with untagged vlan to which our router is connected. Also there is tagged vlan on CPE interface.

So when empty router is connected to CPE it is getting dhcp configuration using untagged interface, let's say 192.168.1.2 255.255.255.0 GW 192.168.1.1 and etc.

But router production config is removing ip address dhcp from Gi0/0 and creating subinterface Gi0/0.99 with static address, let's say 10.0.0.2 255.255.255.0 GW 10.0.0.1 an etc.

As I understand after APIC-EM applied this configurations it is loosing connection with 192.168.1.2 because router doesn't have this IP anymore and config deployment process get stuck.

Is there any workaround ?

Re: Need suggestion for PNP solution

aradford — Thu, 13 Jul 2017 11:17:37 GMT

changing IP address during deployment is fine.

Device just needs to be able to communicate to APIC-EM.

Can the device communicate to APIC-EM once the new IP address is assigned?

Re: Need suggestion for PNP solution

JPvisltgb — Thu, 13 Jul 2017 11:26:02 GMT

That is the problem. From new IP address router can't communicate with APIC-EM. After configuration it will have several VRF's. One VRF will be management and from this vrf router will be able to communicate to APIC-EM

Re: Need suggestion for PNP solution

aradford — Thu, 13 Jul 2017 11:29:56 GMT

for deployment to succeed you will need to have some sort of IP connectivity post the config push. This is used as a sanity check to make sure the provisioning succeeded.

Is the management vrf being provisioned by APIC-EM?

If so, you need to update the pnp-profile to use the vrf. I can show you how to do this...

If the management vrf is not being configured, you will have a problem,.

Adam

Re: Need suggestion for PNP solution

JPvisltgb — Thu, 13 Jul 2017 11:32:57 GMT

All vrf are provisioned by APIC-Em, so please can you show how should pnp-profile look like in configuration?

Re: Need suggestion for PNP solution

aradford — Thu, 13 Jul 2017 11:41:54 GMT

You need the following commands:

! change the PnP profile to use the management VRF

pnp profile network-pnp

transport https ipv4 x.x.x.x port 443 vrf MGMT

! change HTTP request to come from from management interface

ip http client source-interface <interface-name>

I am pretty sure you need to set the source interface as well for the HTTP request initiating from the router. Remember, all PnP connections are initiated from the router.

Adam

Re: Need suggestion for PNP solution

JPvisltgb — Thu, 13 Jul 2017 12:10:00 GMT

I've made changs using these commands.

Seems system crashed.

chunk_diagnose, code = 3

chunk name is MallocLite

current chunk header = 0x7FBF83AB2DA0

data check, ptr = 0x7FBF83AB2DB8

next chunk header = 0x7FBF83AB2DF0

data check, ptr = 0x7FBF83AB2E08

previous chunk header = 0x7FBF83AB2D50

data check, ptr = 0x7FBF83AB2D68

========= Dump chunk = 0x7FBF83AB2DA0 ======================

7FBF83AB2CB8: 7FBF EB8F1DA1 7FBF 83AA79E0 0 0 7FBF FE92A0A0

7FBF83AB2CD8: 0 0 7FBF 831D8B20 7FBF EB88EF40 0 0

7FBF83AB2CF8: 0 0 15A3C78B 1 7FBF EB77E12F 7FBF 83AA79E0

7FBF83AB2D18: 0 0 0 0 0 0 0 3

7FBF83AB2D38: 7FBF 831D8300 9 0 0 0 15A3C78B 1

7FBF83AB2D58: 7FBF EB77E12F 7FBF 83AA79E0 0 0 0 0

7FBF83AB2D78: 0 0 0 3 7FBF 831D8788 9 0

7FBF83AB2D98: 0 0 15A3C78B FFFFFFFF 7FBF EB8F1DA1 7FBF 83AA79E0

7FBF83AB2DB8: EF4321CD 0 7FBF EB8F1DBE B0D0B0D B0D0B0D B0D0B0D B0D0B0D

7FBF83AB2DD8: B0D0B0D B0D0B0D B0D0B0D B0D0B0D B0D0B0D 0 15A3C78B 1

7FBF83AB2DF8: 7FBF EB41B72B 7FBF 83AA79E0 0 0 DADA2012 0

7FBF83AB2E18: 7FBF EB401AF0 7FBF EB400830 7FBF 835796F0 0 0

7FBF83AB2E38: 0 0 15A3C78B 1 7FBF EB65740A 7FBF 83AA79E0

7FBF83AB2E58: 0 0 301F311D 301B0603 5504030C 1473646E 2D6E6574 776F726B

7FBF83AB2E78: 2D696E66 72612D63 61000000 0 0 0 15A3C78B 1

7FBF83AB2E98: 7FBF EB77E12F 7FBF 83AA79E0 0 0 7FBF 83AB08E0

========= Dump chunk_next = 0x7FBF83AB2DF0 ======================

7FBF83AB2CF0: 0 0 0 0 15A3C78B 1 7FBF EB77E12F

7FBF83AB2D10: 7FBF 83AA79E0 0 0 0 0 0 0

7FBF83AB2D30: 0 3 7FBF 831D8300 9 0 0 0

7FBF83AB2D50: 15A3C78B 1 7FBF EB77E12F 7FBF 83AA79E0 0 0

7FBF83AB2D70: 0 0 0 0 0 3 7FBF 831D8788

7FBF83AB2D90: 9 0 0 0 15A3C78B FFFFFFFF 7FBF EB8F1DA1

7FBF83AB2DB0: 7FBF 83AA79E0 EF4321CD 0 7FBF EB8F1DBE B0D0B0D B0D0B0D

7FBF83AB2DD0: B0D0B0D B0D0B0D B0D0B0D B0D0B0D B0D0B0D B0D0B0D B0D0B0D 0

7FBF83AB2DF0: 15A3C78B 1 7FBF EB41B72B 7FBF 83AA79E0 0 0

7FBF83AB2E10: DADA2012 0 7FBF EB401AF0 7FBF EB400830 7FBF 835796F0

7FBF83AB2E30: 0 0 0 0 15A3C78B 1 7FBF EB65740A

7FBF83AB2E50: 7FBF 83AA79E0 0 0 301F311D 301B0603 5504030C 1473646E

7FBF83AB2E70: 2D6E6574 776F726B 2D696E66 72612D63 61000000 0 0 0

7FBF83AB2E90: 15A3C78B 1 7FBF EB77E12F 7FBF 83AA79E0 0 0

7FBF83AB2EB0: 7FBF 83AB08E0 0 0 7FBF 83AB33B0 7FBF 79C7A798

7FBF83AB2ED0: 88 0 0 0 15A3C78B 1 7FBF EB77E12F

========== Dump chunk_previous = 0x7FBF83AB2D50 =====================

7FBF83AB2C50: 9 0 0 0 15A3C78B 1 7FBF EB77E12F

7FBF83AB2C70: 7FBF 83AA79E0 0 0 0 0 0 0

7FBF83AB2C90: 0 3 7FBF 831D7FE0 9 0 0 0

7FBF83AB2CB0: 15A3C78B 1 7FBF EB8F1DA1 7FBF 83AA79E0 0 0

7FBF83AB2CD0: 7FBF FE92A0A0 0 0 7FBF 831D8B20 7FBF EB88EF40

7FBF83AB2CF0: 0 0 0 0 15A3C78B 1 7FBF EB77E12F

7FBF83AB2D10: 7FBF 83AA79E0 0 0 0 0 0 0

7FBF83AB2D30: 0 3 7FBF 831D8300 9 0 0 0

7FBF83AB2D50: 15A3C78B 1 7FBF EB77E12F 7FBF 83AA79E0 0 0

7FBF83AB2D70: 0 0 0 0 0 3 7FBF 831D8788

7FBF83AB2D90: 9 0 0 0 15A3C78B FFFFFFFF 7FBF EB8F1DA1

7FBF83AB2DB0: 7FBF 83AA79E0 EF4321CD 0 7FBF EB8F1DBE B0D0B0D B0D0B0D

7FBF83AB2DD0: B0D0B0D B0D0B0D B0D0B0D B0D0B0D B0D0B0D B0D0B0D B0D0B0D 0

7FBF83AB2DF0: 15A3C78B 1 7FBF EB41B72B 7FBF 83AA79E0 0 0

7FBF83AB2E10: DADA2012 0 7FBF EB401AF0 7FBF EB400830 7FBF 835796F0

7FBF83AB2E30: 0 0 0 0 15A3C78B 1 7FBF EB65740A

============================================

Refcount on freed chunk is non-zero

%Software-forced reload

Frame pointer 0x7FBF83C4ED00, PC = 0x7FBFAB4791A9

-Traceback= 1#e7b171113d9d8fef543d3319f9a4e292 c:7FBFAB448000+311A9 c:7FBFAB448000+32659 :7FBFE1306000+AFFAF5E :7FBFE1306000+9F61399 :7FBFE1306000+9F43AAD :7FBFE1306000+7470804 :7FBFE1306000+7470418 :7FBFE1306000+A0E3EBA :7FBFE1306000+9536984 :7FBFE1306000+A0FABB5 :7FBFE1306000+953C5C9 :7FBFE1306000+A0C988B :7FBFE1306000+A0C94AB

Fastpath Thread backtrace:

-Traceback= 1#e7b171113d9d8fef543d3319f9a4e292 c:7FBFAB448000+BDF12

Auxiliary Thread backtrace:

-Traceback= 1#e7b171113d9d8fef543d3319f9a4e292 pthread:7FBFA569D000+A7C9

RAX = 0000000000000000 RBX = 00007FBF83C4EDB0

RCX = FFFFFFFFFFFFFFFF RDX = 0000000000000006

RSP = 00007FBF83C4ED00 RBP = 00007FBF83C4ED00

RSI = 00000000000067D8 RDI = 00000000000067D8

R8 = 0000000000000080 R9 = 0101010101010101

R10 = 0000000000000008 R11 = 0000000000000206

R12 = 00007FBFE1219160 R13 = 046877E8BF7F0000

R14 = 0000000000000000 R15 = 00007FBF83AA79E0

RFL = 0000000000000206 RIP = 00007FBFAB4791A9

CS = 0033 FS = 0000 GS = 0000

ST0 = 0000 0000000000000000 ST1 = 0000 0000000000000000

ST2 = 0000 0000000000000000 ST3 = 0000 0000000000000000

ST4 = 0000 0000000000000000 ST5 = 0000 0000000000000000

ST6 = 0000 0000000000000000 ST7 = 0000 0000000000000000

X87CW = 037F X87SW = 0000 X87TG = 0000 X87OP = 0000

X87IP = 0000000000000000 X87DP = 0000000000000000

XMM0 = 00000000000000000000000000000000

XMM1 = 00000000000000000000000000000000

XMM2 = 00000000000000000000000000000000

XMM3 = 00000000000000000000000000000000

XMM4 = 00000000000000000000000000000000

XMM5 = 00000000000000000000000000000000

XMM6 = 00000000000000000000000000000000

XMM7 = 00000000000000000000000000000000

XMM8 = 00000000000000000000000000000000

XMM9 = 00000000000000000000000000000000

XMM10 = FFFFFFFF000000000000000000000000

XMM11 = 00000000000000000000000000000A0A

XMM12 = 00000004000000040000000400000004

XMM13 = 0000000E000000090000000E000000D2

XMM14 = 00000000000000000000000000000000

XMM15 = FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF

MXCSR = 00001FA0

Writing crashinfo to bootflash:crashinfo_RP_00_00_20170713-135922-CEST

Buffered messages:

Jul 13 13:59:18.970 CEST: %XML-UPDOWN: pnp-zero-touch XML Interface(101) DOWN(502).

Jul 13 13:59:18.970 CEST: %XML-UPDOWN: pnp-zero-touch XML Interface(101) UP. PID=364

Jul 13 13:59:18.973 CEST: %SYS-2-CHUNKBADREFCOUNT: Bad chunk reference count, chunk 7FBF83AA79E0 data 7FBF83AB2DB8 refcount FFFFFFFFFFFFFFFF alloc pc 7FBFEB8F1DA1. -Process= "XEP_pnp-zero-touch", ipl= 0, pid= 364

-Traceback= 1#e7b171113d9d8fef543d3319f9a4e292 :7FBFE1306000+9F61382 :7FBFE1306000+9F43AAD :7FBFE1306000+7470804 :7FBFE1306000+7470418 :7FBFE1306000+A0E3EBA :7FBFE1306000+9536984 :7FBFE1306000+A0FABB5 :7FBFE1306000+953C5C9 :7FBFE1306000+A0C988B :7FBFE1306000+A0C94ABJul 13 14:00:18.664 R0/0: %PMAN-3-PROCHOLDDOWN: The process linux_iosd-image has been helddown (rc 134)

Jul 13 14:00:19.028 R0/0: %PMAN-0-PROCFAILCRIT: A critical process linux_iosd_image has failed (rc 134)

Jul 13 14:00:26.885 R0/0: %PMAN-5-EXITACTION: Process manager is exiting: critical process fault, linux_iosd_image, rp_0_0, rc=134

Initializing Hardware ...

and rebooted

Re: Need suggestion for PNP solution

aradford — Thu, 13 Jul 2017 12:15:20 GMT

That is not good.

can you try without the http command?

Re: Need suggestion for PNP solution

JPvisltgb — Thu, 13 Jul 2017 12:46:23 GMT

I removed ip http client source-interface <interface-name> - System crashed.

Then I removed from pnp profile line vrf MGMT - System crashed.

Removed

pnp profile pnp-zero-touch

transport https ipv4 x.x.x.x port 443

pnp startup-vlan 99

Everything is ok

So seems that IOS don't like pnp profile overwritting.

Re: Need suggestion for PNP solution

JPvisltgb — Mon, 31 Jul 2017 07:40:34 GMT

I found solution.

Apic-em is creating pnp profile - pnp-zero-touch.

So i just added

backup transport https host XXX.XXX.XXX.XXX port 443 vrf VRF-NAME

to that profile.

and it works.