ā11-07-2016 02:30 PM - edited ā03-01-2019 04:33 AM
I wonder if anybody else is having this issue, I am trying to perfect my PNP config's and I have been added features as I go along to introduce complexity to my project to help me prepare for a real deployment.
I have 8 devices in my lab, 3 routers and 5 switches (including a 4 member switch stack). I have been successfully able to deploy image upgrades and basic configuration templates. I am now moving on to assigning ip addresses' and non-vlan1 configurations + etherchannel configurations.
I am now running into this error now that I have changed my switching configuration to include management vlan with basic default route. I have the pnp startup-vlan XX command on the upstream switch. One of my switches get configured correctly with the static management vlan and ip. The other ones come back with and error in APIC-EM controller about not being able to manage the device due to a time out (see error below). I found a bug ID with this error but I would suggest that it doesn't fit the problem I am seeing.
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuy57714/?referring_site=bugquickviewredir
Error message in the project page:
#######################################
Failed health check since device is stuck in non-terminal state PROVISIONING_CONFIG for more than threshold time: 0 hours, 16 minutes, 0 seconds
#######################################
After waiting for the configuration to deploy, if I log in the console I see this message and the job error's out in APIC-EM. If I enter no to the prompt I can see the configuration was delivered to the switch with the exception of enabling SSH.
#######################################
--- System Configuration Dialog ---
Enable secret warning
----------------------------------
In order to access the device manager, an enable secret is required
If you enter the initial configuration dialog, you will be prompted for the enable secret
If you choose not to enter the intial configuration dialog, or if you exit setup without setting the enable secret,
please set an enable secret using the following CLI in configuration mode-
enable secret 0 <cleartext password>
----------------------------------
Would you like to enter the initial configuration dialog? [yes/no]:
Has anybody else noticed this behaviour?
JP
Solved! Go to Solution.
ā11-07-2016 05:47 PM
ok. sounds like a config file issue.
you should be able to ping the default gateway from the switch as it is in the same vlan? does that work?
If not, need to check that the vlan is being trunked between the two switches.
ā11-07-2016 02:42 PM
Hi JP,
there are a couple of reasons this might occur.
What model switch are you using and which version of code is it booting with?
Also, are you able to share a sanitised version of the config please?
Also, when doing testing, it is important that you "clean up" the device properly...
Here is a "full clean up" (as well as wr er) for a stack. I am going to post a PnP stacking blog later today.
# remove the certificates on active and standby
delete /force nvram:*.cer
delete /force stby-nvram:*.cer
# remove vlan data based from active and standby
delete /force flash-1:vlan.dat
delete /force flash-2:vlan.dat
# remove certificates from memory. NOTE: you will not be able to SSH after this
conf t
crypto key zeroize
yes
end
ā11-07-2016 04:05 PM
Hello Adam, after testing your clean up suggestions I am still getting to the same point in the deployment. It seems that during the cut over to the vlan the pnp server gets disconnected and the the pnp process stops. Can you please review the console logs and tell me what you see?
The hardware: WS-C3650-48PD - cat3k_caa-universalk9.SPA.03.07.04.E.152-3.E4.bin
Nov 7 23:40:21.884: %PNPA-DHCP Op-43 Msg: OK to process message
Nov 7 23:40:21.884: XML-UPDOWN: PNPA_DHCP_OP43 XML Interface(102) UP. PID=399
Nov 7 23:40:21.884: %PNPA-DHCP Op-43 Msg: _pdoon.1.ntf.don=399
Nov 7 23:40:21.891: %PNPA-DHCP Op-43 Msg: _pdoop.1.org=[A1D;B2;K4;I172.X.X.X;J80]
Nov 7 23:40:21.891: %PNPA-DHCP Op-43 Msg: _pdgfa.1.inp=[B2;K4;I172.X.X.X;J80]
Nov 7 23:40:21.891: %PNPA-DHCP Op-43 Msg: _pdgfa.1.B2.s12=[ ipv4 ]
Nov 7 23:40:21.891: %PNPA-DHCP Op-43 Msg: _pdgfa.1.K4.htp=[ transport http ]
Nov 7 23:40:21.891: %PNPA-DHCP Op-43 Msg: _pdgfa.1.Ix.srv.ip.rm=[ 172.X.X.X ]
Nov 7 23:40:21.891: %PNPA-DHCP Op-43 Msg: _pdgfa.1.Jx.srv.rt.rm=[ port 80 ]
Nov 7 23:40:21.891: %PNPA-DHCP Op-43 Msg: _pdoop.1.ztp=[pnp-zero-touch] host=[] ipad=[172.X.X.X] port=80
Nov 7 23:40:21.891: %PNPA-DHCP Op-43 Msg: _pors.done=1
Nov 7 23:40:21.891: %PNPA-DHCP Op-43 Msg: _pdokp.1.now=PNPA_DHCP_OP43 pid=399
Nov 7 23:40:21.891: XML-UPDOWN: PNPA_DHCP_OP43 XML Interface(102) SHUTDOWN(101). PID=399
Nov 7 23:40:21.936: %DHCP-6-ADDRESS_ASSIGN: Interface Vlan107 assigned DHCP address 172.X.X.X, mask 255.255.255.0, hostname
Nov 7 23:40:37.689: AUTOINSTALL: Obtain siaddr 172.16.X.X (as config server)
Nov 7 23:40:47.696: %PNP-6-HTTP_CONNECTING: PnP Discovery trying to connect to PnP server http://172.X.X.X:80/pnp/HELLO
Nov 7 23:40:47.707: %PNP-6-HTTP_CONNECTED: PnP Discovery connected to PnP server http://172.X.X.X:80/pnp/HELLO
Nov 7 23:40:48.724: %PNP-6-PROFILE_CONFIG: PnP Discovery profile pnp-zero-touch configured
%Error opening tftp://172.16.33.139/network-confg (Timed out)
Nov 7 23:41:23.212: %SYS-6-CLOCKUPDATE: System clock has been updated from 23:41:19 UTC Mon Nov 7 2016 to 23:41:23 UTC Mon Nov 7 2016, configured from console by console.
Nov 7 23:41:31.992: %PKI-4-NOCONFIGAUTOSAVE: Configuration was modified. Issue "write memory" to save new IOS PKI configuration
%Error opening tftp://172.16.33.139/cisconet.cfg (Timed out)
%Error opening tftp://172.16.33.139/router-confg (Timed out)
%Error opening tftp://172.16.33.139/ciscortr.cfg (Timed out)
000038: Nov 7 23:44:14 UTC: %SYS-5-LOG_CONFIG_CHANGE: Buffer logging: level informational, xml disabled, filtering disabled, size (65000)
000039: Nov 7 23:44:14 UTC: %SYS-5-LOG_CONFIG_CHANGE: Monitor logging: level informational, xml disabled, filtering disabled
000040: Nov 7 15:44:14 PST: %SYS-6-CLOCKUPDATE: System clock has been updated from 23:44:14 UTC Mon Nov 7 2016 to 15:44:14 PST Mon Nov 7 2016, configured from console by vty1.
000041: Nov 7 15:44:14 PST: %SYS-6-CLOCKUPDATE: System clock has been updated from 15:44:14 PST Mon Nov 7 2016 to 15:44:14 PST Mon Nov 7 2016, configured from console by vty1.
000042: Nov 7 15:44:14 PST: %SW_VLAN-6-VTP_DOMAIN_NAME_CHG: VTP domain name changed to company.corp.
000043: Nov 7 15:44:15 PST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan107, changed state to down
000047: Nov 7 15:44:20 PST: %PARSER-4-BADCFG: Unexpected end of configuration file.
000048: Nov 7 15:44:20 PST: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 172.1X.X.X port 514 started - CLI initiated
000049: Nov 7 15:44:20 PST: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 172.X.X.X port 514 started - CLI initiated
000050: Nov 7 15:44:23 PST: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 172.X.X.X port 514 started - CLI initiated
000051: Nov 7 15:44:45 PST: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan107, changed state to up
000052: Nov 7 15:44:49 PST: %PNPA-DHCP Op-43 Msg: _papdo.2.eRr.ena
000053: Nov 7 15:44:49 PST: %PNPA-DHCP Op-43 Msg: _pdoon.2.eRr.pdo=-1
Here is the config:
service nagle
no service config
no service pad
service tcp-keepalives-in
service timestamps debug datetime localtime show-timezone
service timestamps log datetime localtime show-timezone
service password-encryption
service sequence-numbers
!
hostname STG-BR-S-01
!
boot-start-marker
boot-end-marker
!
!
vrf definition Mgmt-vrf
!
address-family ipv4
exit-address-family
!
address-family ipv6
exit-address-family
!
logging buffered 65000 informational
logging monitor informational
enable secret 5 XXXXXXX
!
username swadmin privilege 15 secret 5 XXXXXXXXXX
aaa new-model
!
!
aaa authentication login default group tacacs+ local
aaa accounting update periodic 1
aaa accounting exec default start-stop group tacacs+
aaa accounting commands 15 default start-stop group tacacs+
aaa accounting connection default start-stop group tacacs+
no aaa accounting system guarantee-first
!
!
!
aaa session-id common
clock timezone PST -8 0
clock summer-time PDT recurring
no ip domain-lookup
ip domain-name bcferries.corp
!
!
login on-failure log
vtp domain XXXXXXX.corp
vtp mode transparent
authentication mac-move permit
port-channel load-balance src-ip
spanning-tree mode pvst
spanning-tree extend system-id
!
redundancy
mode sso
interface Vlan107
ip address X.X.X.X 255.255.255.0
ip helper-address X.X.X.X
no ip redirects
no ip route-cache
ip default-gateway X.X.X.X
ip forward-protocol nd
no ip http server
no ip http secure-server
ip ssh version 2
ip scp server enable
!
!
ip sla enable reaction-alerts
logging source-interface Vlan1
logging host X.X.X.X
!
!
snmp ifmib ifindex persist
tacacs server ACS-01
address ipv4 X.X.X.X
key 7 XXXXXXXX
tacacs server ACS-02
address ipv4 X.X.X.X
key 7 XXXXXXXXX
!
!
!
line con 0
exec-timeout 30 0
privilege level 15
stopbits 1
line vty 0 4
exec-timeout 30 0
privilege level 15
transport input ssh
line vty 5 15
exec-timeout 30 0
privilege level 15
transport input ssh
!
ntp server X.X.X.X
ā11-07-2016 04:10 PM
Hi JP
you need to have "end" as the last line of the config file :-)
this error message is an indication
000047: Nov 7 15:44:20 PST: %PARSER-4-BADCFG: Unexpected end of configuration file.
Let me know if that resolves the problem?
Adam
ā11-07-2016 04:47 PM
Thanks Adam.
Putting end has corrected the error message: 000047: Nov 7 15:44:20 PST: %PARSER-4-BADCFG: Unexpected end of configuration file.
But I am still left with the original error, the deploy times out after 15 mins.
Any other suggestions?
ā11-07-2016 05:02 PM
Is the credential in the config file the same one as APIC-EM would use to discover the device?
username swadmin privilege 15 secret 5 XXXXXXXXXX
ā11-07-2016 05:06 PM
Also, do you have vlan 1 configured on this device?
I see you are using it as the logging source?
logging source-interface Vlan1
If pnp startup-vlan does the right thing, you should have vlan 1 shutdown, vlan 107 enabled, and any active interfaces in vlan 107?
I can see vlan 107 being created in the pnp logs.
Adam
ā11-07-2016 05:06 PM
Also, do you have vlan 1 configured on this device?
I see you are using it as the logging source?
logging source-interface Vlan1
If pnp startup-vlan does the right thing, you should have vlan 1 shutdown, vlan 107 enabled, and any active interfaces in vlan 107?
I can see vlan 107 being created in the pnp logs.
Adam
ā11-07-2016 05:35 PM
In my testing, I keep the configuration the same but only changed the section under vlan 107 to ip address DHCP instead of a static. This change completes with a state of "provisioned".
The thing I am noticing is that my switches with L3 code are working perfectly with this configuration but my L2 ones are all failing. I have tired testing the reach-ability of the device and I am missing something. I can't route off this device even with the ip route 0.0.0.0 0.0.0.0 X.X.X.X command entered, I have also configured the default gateway command.
The vlan1 as a logging source I think is a mistake and I will remove that line.
JP
ā11-07-2016 05:47 PM
ok. sounds like a config file issue.
you should be able to ping the default gateway from the switch as it is in the same vlan? does that work?
If not, need to check that the vlan is being trunked between the two switches.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide