Prime APIC-EM PnP issue

[parag waghmare]

Hi Experts,

I need some help.

I have a situation. I am configuring a 3850 switch with PnP using Prime and APIC-EM.

I am using PnP profile on Prime and activating it.

The PnP profile is activated and the configuration is send to APIC-EM.

APIC-EM shows the device as provisioned, but Prime still shows current status as pending and thus my Post-pnp script does not run.

Why would APIC-EM show as device provisioned and Prime cant see it.

Is there any setting on APIC-EM where I need to point to Prime ?

Prime talks to APIC-EM on https.

Can anyone help on this ?

Thank you.

[aradford]

Hi Parag,

it would be helpful to know what version of APIC-EM and PI you are using.

there is only a unidirectional connection (PI -> APIC-EM) required.

Just out of interest, what version of code on 3850?

Adam

[parag waghmare]

Hi Adam,

Thanks for responding.

This is where I am at at the moment.

Deployment Environment Information :

• - Cisco Prime 3.1.5
• - APIC-EM 1.4.2
• - APIC-EM / Cisco Prime 3.1 are both deployed in VM Environment using VMWARE ESXI.
• - Both are in the same subnet ( no firewall involved )
• - Upstream device is 6509-E ( Running 15.2 IOS )  SUP2-T
• - Downstream device is 3850-48U ( Running 3.6.6E IOS XE)

Symptoms :

1. New PnP profile was created using Cisco Prime with devices added successfully and pushed to APIC-EM. Post PnP radius configuration also applied to the profile.
2. APIC-EM discovers the device using DHCP Option 43 and successfully goes through each step ( as seen on the status ) and provisions the device.
3. 3.       Prime still displays the provisioning status as “PENDING”.
4. Prime will not push the Post PnP configuration because device is still maintaining the “PENDING” status.
5. Another symptom : APIC-EM Reachability History in Cisco Prime APIC-EM Controller dashboard does not show any “ RUN ID” states “No Data available”

Troubleshooting steps performed:

1. Prime has successfully been configured using APIC-EM Credentials which has provided status of “ Success “ for reachability
2. APIC-EM has successfully been configured using Prime root credentials 
3. Sync is completed using port 443 ( HTTPS )
4. NTP servers are both the same on both VM’s, and have reported the correct times synced ( may be necessary for HTTPS protocol )
5. Dashboard Job tasks in Cisco Prime have been checked ( APIC-EM Sync Site ) and manually pushed and have reported to have run successfully.

We pulled up the IFM_APIC.log file as well on prime and it shows me this error, which could be the cause. Dont know why this is happening.

[2017-07-13 10:16:41,300] [pool-99-thread-1] [apic] [INFO ] - Thread Id : [23,272,302] : IFM_APIC_INFO: [ApicPollingService: Polling APIC controller for device ]

[2017-07-13 10:16:41,314] [pool-99-thread-1] [apic] [INFO ] - Thread Id : [23,272,302] : IFM_APIC_INFO: [APIC-REQ=  requestURL -/api/v1/pnp-project/2eab8c62-2d9a-410d-bb12-644893c24a8a/device?offset=1&limit=100 httpMethod -GET requestData - {} requestID - 3eff8533-8691-4b3f-9c31-17d27795d5be]

[2017-07-13 10:16:41,340] [pool-99-thread-1] [apic] [ERROR] - Thread Id : [23,272,302] : IFM_APIC_ERROR: [Exception [Ljava.lang.StackTraceElement;@3ce4ba6]

[2017-07-13 10:16:41,341] [pool-99-thread-1] [apic] [ERROR] - Thread Id : [23,272,302] : IFM_APIC_ERROR_DETAILS: [Unable to establish the connection with the Server.] : IFM_APIC_ERROR: [com.cisco.nm.expression.function.FunctionException: Unable to establish the connection with the Server.]

[2017-07-13 10:16:41,341] [pool-99-thread-1] [apic] [ERROR] - IFM_TRUSTSEC_EXCEPTION

1. com.cisco.nm.expression.function.FunctionException: Unable to establish the connection with the Server.

at XDE_ENGINE.[apicInteractor]apicConfigPush.xde@root:141204135019494.catch.root:150207173738561.steps.root:150207174605273(apicConfigPush.xde)

at com.cisco.nm.workflow.steps.ThrowStep$1.executeExpressionResult(ThrowStep.java:56)

at com.cisco.nm.workflow.steps.StepExpressionJavaScript$2.executeExpressionResult(StepExpressionJavaScript.java:121)

at com.cisco.nm.expression.js.JavaScriptCompiler$JavaScriptExecutor.callbackSuccess(JavaScriptCompiler.java:315)

Thank you.

[aradford]

Thanks for debugs.

Was the project name already defined in APIC-EM before using PI?

I have seen this issue where a project was defined in both places.

BTW, have you tested the API call with POSTMAN or some sort of similar tool?

/api/v1/pnp-project/2eab8c62-2d9a-410d-bb12-644893c24a8a/device?offset=1&limit=100

Adam

[parag waghmare]

Hi Adam,

Thanks for your response.

Was the project name already defined in APIC-EM before using PI?

No, we created them in prime and it automatically pushes it to prime.

We tried few things in the mean time :

We tried using another VM Cisco Prime 3.1.5. This Cisco prime deployment was exactly same as other prime instance, but this time around, the PnP provisioning status went to 100% SUCCESS, but our Post PnP is not kicking in.

We are trying to deploy the aaa config using post PnP config. I had couple of questions related to it.

1) How does the credential profile come into picture for post PnP to kick in.

2) We configured a local username password on the switch as you do in your APIC-EM demo video, and entered same credential details in the credential profile attached to the PnP profile on prime. But still prime does not login with those credentials. Are we missing any step ?

3) Once the PnP initial config is completed, does prime login again to kick start the post-PnP process ?

On the reachability front between Prime and APIC-EM :

Even on the new Prime Instance , the reachability log still does not show anything. There is no apic polling happening every 5 mins. What could be the issue? Is it Any bug with prime 3.1.5 which could be causing issue to talk to APIC-EM ? Which direction should we look into  to troubleshoot this issue ?

My understanding was that as we don't see an reachability log on Prime, that could be the reason why the PnP provisioning status does not show up on PnP device status. But now, we do see the PnP provisioning status moving to 100% SUCCESS, but still cant see the reachability log.

BTW, have you tested the API call with POSTMAN or some sort of similar tool?

/api/v1/pnp-project/2eab8c62-2d9a-410d-bb12-644893c24a8a/device?offset=1&limit=100

No, Do you have any testing document for POSTMAN on how to test the api call ??

Appreciate your help.

[ngoldwat]

My guess is you most likely have a problematic entry in your database.  I'd open a TAC case on this.

[ngoldwat]

Re: Postman

Coding 102 REST API Basics using Cisco Spark - YouTube  Around the 28 minute mark.

Cisco Network Programmability Using APIC-EM APIs with Saurav Prasad - YouTube Around the 11 minute mark.

[parag waghmare]

Thanks for the links Nic. With your comment on database, can you further elaborate please? Could this be the cause for the reachability issue between prime and apic-em or the pop-pnp config issue.

Adam, if you can please provide your inputs that would be helpful too.

Really need to get this going.

Really appreciate all the experts inputs.

Thank you.

[rune.jon]

Hi

I have the same issue. PI 3.1.4 and APIC-EM 1.4.

Used 3650 and 3850 on 16.3.2.

Post-Pnp takes time after successfull provisioning.

But it runs but takes an hour before it gets success.

1 hour for status success and I am on the console and see that the post-pnp was done within 2 minutes.

[Per Tenggren]

Hi,

Were you ever able to solve this?

/Per