Re: Can't access UI on APIC-EM

R0n1n · ‎01-23-2017

Hello all -

I stood up an APIC-EM last week and have not been able to access the UI of the site. The grapevine services are running...

but am still unable to access. I tried by recommended URL: https://10.0.11.225 as well as the grapevine developer console: https://10.0.11.225:14141.

I get no response when attempting the first URL and this when access the dev console:

Any help would be appreciated.

Thank you,

Daniel

aradford · ‎01-25-2017

Hi Daniel,

I would log into the console and do a "reset_grapevine".

This should restart all of the services. Given you have not used the controller, you can say "y" to the questions you get asked.

It will also show all of the services starting up (or why they failed).

Port 14141 is no longer used in 1.3 and later and there is just a redirect in place now.

Adam

R0n1n · ‎01-25-2017

Thanks Adam for the reply.

I did what you suggested. Afterwards, when attempting to access the site, this is what I get:

We don't have any proxies in place, but since I was accessing this server from across the WAN, I attempted to access the server from within the same site and received the same message.

Capturing the traffic in Wireshark shows the server resetting the connection:

Any additional ideas?

We only added one NIC (which we gave the internal address) and the grape-br0 interface has the address 169.254.0.1. Does this interface need a usable IP? Is this what the UI uses?

Daniel

aradford · ‎01-25-2017

really strange. Do you have a firewall protecting the APIC-EM subnet?

The grape-bro interface is internal to the device. You do not want to reach that externally. it is for E-W message bus communication between the grapevine services.

Can you try the following on the APIC-EM console?

$ curl -k https://localhost

You should see something like this

<!doctype html>

<html>

<head>

    <meta http-equiv="content-type" content="text/html; charset=utf-8">

    <meta charset="utf-8" />

    <title>Login - APIC - Enterprise Module</title>

I assume all the services were up after the "reset_grapevine"?

you should see a screen like the following?

R0n1n · ‎01-25-2017

Here is the result and clearly where the problem lies:

Wintel only gave me 200 Gb for storage. Any chance that could be impacting the server?

Daniel

aradford · ‎01-25-2017

That should be ok, at least to get started.

Did you allocate enough RAM / cpu to the virtual machine?

You need 32G RAM and 8vcpu

R0n1n · ‎01-25-2017

There is 64 Gb or RAM allocated to this server, but only 6 vCPU. The CPU has only peaked at 27% the past week.

aradford · ‎01-25-2017

that should be fine.

Disk IO?

Can you run

$ dd if=/dev/zero of=/tmp/foo bs=1M count=512 conv=fdatasync
512+0 records in
512+0 records out

536870912 bytes (537 MB) copied, 23.7006 s, 22.7 MB/s

I am interested in the MB/s

R0n1n · ‎01-25-2017

Here are those results:

Thanks.

aradford · ‎01-25-2017

I think we have a winner...

Disk IO is meant to be 200MB/s.

You can get away with less if you up cpu in a lab.

Once you drop below 30MB/s bad things happen.

Are you sure all of the services started? It is likely taking forever to start them up after a reset_grapevine.

Maybe try the command a couple more times to see if it is consistently bad.

R0n1n · ‎01-25-2017

Yes, the reset took quite a while. How would I go about increasing the disk IO?

aradford · ‎01-25-2017

Depends... is the storage local to the server?

Is the raid controller working correctly?

Are there other high I/O workloads running on the same server?

It would be interesting to know why the IO is so slow... is it your server or is there a server admin?

R0n1n · ‎01-25-2017

Adam - I asked one of the server guys to take another look at it. He found an issue that he wouldn't clarify completely (related to memory though). He corrected it and I am restarting the server to test it.

Thanks,

Daniel

aradford · ‎01-25-2017

Thanks Daniel,

please keep us updated.

R0n1n · ‎01-25-2017

Same issues unfortunately.