topic Can't access UI on APIC-EM in Controllers

Can't access UI on APIC-EM

R0n1n — Fri, 01 Mar 2019 12:35:35 GMT

Hello all -

I stood up an APIC-EM last week and have not been able to access the UI of the site. The grapevine services are running...

but am still unable to access. I tried by recommended URL: https://10.0.11.225 as well as the grapevine developer console: https://10.0.11.225:14141.

I get no response when attempting the first URL and this when access the dev console:

Any help would be appreciated.

Thank you,

Daniel

Re: Can't access UI on APIC-EM

aradford — Wed, 25 Jan 2017 14:30:58 GMT

Hi Daniel,

I would log into the console and do a "reset_grapevine".

This should restart all of the services. Given you have not used the controller, you can say "y" to the questions you get asked.

It will also show all of the services starting up (or why they failed).

Port 14141 is no longer used in 1.3 and later and there is just a redirect in place now.

Adam

Re: Can't access UI on APIC-EM

R0n1n — Wed, 25 Jan 2017 15:29:09 GMT

Thanks Adam for the reply.

I did what you suggested. Afterwards, when attempting to access the site, this is what I get:

We don't have any proxies in place, but since I was accessing this server from across the WAN, I attempted to access the server from within the same site and received the same message.

Capturing the traffic in Wireshark shows the server resetting the connection:

Any additional ideas?

We only added one NIC (which we gave the internal address) and the grape-br0 interface has the address 169.254.0.1. Does this interface need a usable IP? Is this what the UI uses?

Daniel

Re: Can't access UI on APIC-EM

aradford — Wed, 25 Jan 2017 15:41:08 GMT

really strange. Do you have a firewall protecting the APIC-EM subnet?

The grape-bro interface is internal to the device. You do not want to reach that externally. it is for E-W message bus communication between the grapevine services.

Can you try the following on the APIC-EM console?

$ curl -k https://localhost

You should see something like this

<!doctype html>

<html>

<head>

    <meta http-equiv="content-type" content="text/html; charset=utf-8">

    <meta charset="utf-8" />

    <title>Login - APIC - Enterprise Module</title>

I assume all the services were up after the "reset_grapevine"?

you should see a screen like the following?

Re: Can't access UI on APIC-EM

R0n1n — Wed, 25 Jan 2017 15:52:09 GMT

Here is the result and clearly where the problem lies:

Wintel only gave me 200 Gb for storage. Any chance that could be impacting the server?

Daniel

Re: Can't access UI on APIC-EM

aradford — Wed, 25 Jan 2017 15:57:41 GMT

That should be ok, at least to get started.

Did you allocate enough RAM / cpu to the virtual machine?

You need 32G RAM and 8vcpu

Re: Can't access UI on APIC-EM

R0n1n — Wed, 25 Jan 2017 16:05:16 GMT

There is 64 Gb or RAM allocated to this server, but only 6 vCPU. The CPU has only peaked at 27% the past week.

Re: Can't access UI on APIC-EM

aradford — Wed, 25 Jan 2017 16:11:19 GMT

that should be fine.

Disk IO?

Can you run

$ dd if=/dev/zero of=/tmp/foo bs=1M count=512 conv=fdatasync
512+0 records in
512+0 records out

536870912 bytes (537 MB) copied, 23.7006 s, 22.7 MB/s

I am interested in the MB/s

Re: Can't access UI on APIC-EM

R0n1n — Wed, 25 Jan 2017 16:15:23 GMT

Here are those results:

Thanks.

Re: Can't access UI on APIC-EM

aradford — Wed, 25 Jan 2017 16:20:37 GMT

I think we have a winner...

Disk IO is meant to be 200MB/s.

You can get away with less if you up cpu in a lab.

Once you drop below 30MB/s bad things happen.

Are you sure all of the services started? It is likely taking forever to start them up after a reset_grapevine.

Maybe try the command a couple more times to see if it is consistently bad.

Re: Can't access UI on APIC-EM

R0n1n — Wed, 25 Jan 2017 16:24:25 GMT

Yes, the reset took quite a while. How would I go about increasing the disk IO?

Re: Can't access UI on APIC-EM

aradford — Wed, 25 Jan 2017 16:28:17 GMT

Depends... is the storage local to the server?

Is the raid controller working correctly?

Are there other high I/O workloads running on the same server?

It would be interesting to know why the IO is so slow... is it your server or is there a server admin?

Re: Can't access UI on APIC-EM

R0n1n — Wed, 25 Jan 2017 17:01:39 GMT

Adam - I asked one of the server guys to take another look at it. He found an issue that he wouldn't clarify completely (related to memory though). He corrected it and I am restarting the server to test it.

Thanks,

Daniel

Re: Can't access UI on APIC-EM

aradford — Wed, 25 Jan 2017 17:03:43 GMT

Thanks Daniel,

please keep us updated.

Re: Can't access UI on APIC-EM

R0n1n — Wed, 25 Jan 2017 17:30:42 GMT

Same issues unfortunately.

Re: Can't access UI on APIC-EM

R0n1n — Wed, 25 Jan 2017 21:57:07 GMT

I did another reset_grapevine and the came up with the following:

I asked the Wintel team to up the resources to see if that resolves my issue.

Re: Can't access UI on APIC-EM

aradford — Thu, 26 Jan 2017 05:02:50 GMT

Hi Daniel,

one more thing..

I do not suppose there is anything interesting in the following logfile

/var/log/grapevine/grapevine_capacity_manager.log

Re: Can't access UI on APIC-EM

R0n1n — Thu, 26 Jan 2017 14:19:32 GMT

Adam - this is the one thing of note in the log file:

Re: Can't access UI on APIC-EM

R0n1n — Thu, 26 Jan 2017 14:56:29 GMT

Is there any way to reset the VM back to factory default? I can try and rebuild it.