Unexpected CloudCenter Issue during installation

[michgri]

I'm trying to install CloudCenter 4.9.1 on VMware (ESXi 6.5) and after installation I'm getting the "Unexpected CloudCenter issue. Please contact your system administrator" error. I ran the pre-config check on all of the virtual machines and everything seemed to be working; firewall shouldn't be an issue since this is all going on the same subnet. VMs are all at the default size provided by the OVF.

On CCM there are a number of services that didn't start; whatever should be running at 127.0.0.1:8080 is not.

[root@ccm ~]# systemctl status mgmtserver.service -l
● mgmtserver.service - Management Server
   Loaded: loaded (/etc/systemd/system/mgmtserver.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Thu 2018-10-25 02:29:21 UTC; 12min ago
  Process: 665 ExecStart=/bin/sh -c . /etc/profile.d/jdk.sh;/usr/local/cliqr/apps/mgmtserver.war (code=exited, status=1/FAILURE)
 Main PID: 665 (code=exited, status=1/FAILURE)

Oct 25 02:28:24 ccm.toca.local sh[665]: | |    | | ___  _   _  __| | |     ___ _ __ | |_ ___ _ __  | \  / | __ _ _ __   __ _  __ _  ___ _ __
Oct 25 02:28:24 ccm.toca.local sh[665]: | |    | |/ _ \| | | |/ _` | |    / _ \ '_ \| __/ _ \ '__| | |\/| |/ _` | '_ \ / _` |/ _` |/ _ \ '__|
Oct 25 02:28:24 ccm.toca.local sh[665]: | |____| | (_) | |_| | (_| | |___|  __/ | | | ||  __/ |    | |  | | (_| | | | | (_| | (_| |  __/ |
Oct 25 02:28:24 ccm.toca.local sh[665]: \_____|_|\___/ \__,_|\__,_|\_____\___|_| |_|\__\___|_|    |_|  |_|\__,_|_| |_|\__,_|\__, |\___|_|
Oct 25 02:28:24 ccm.toca.local sh[665]: __/ |
Oct 25 02:28:24 ccm.toca.local sh[665]: |___/
Oct 25 02:28:24 ccm.toca.local sh[665]: packager : (v4.9.1), Powered by Spring Boot : 1.5.13.RELEASE
Oct 25 02:29:21 ccm.toca.local systemd[1]: mgmtserver.service: main process exited, code=exited, status=1/FAILURE
Oct 25 02:29:21 ccm.toca.local systemd[1]: Unit mgmtserver.service entered failed state.
Oct 25 02:29:21 ccm.toca.local systemd[1]: mgmtserver.service failed.
[root@ccm ~]# systemctl status cliqr-esb.service -l
● cliqr-esb.service - CliQr ESB Server
   Loaded: loaded (/etc/systemd/system/cliqr-esb.service; disabled; vendor preset: disabled)
   Active: inactive (dead)

Oct 25 02:29:11 ccm.toca.local systemd[1]: [/etc/systemd/system/cliqr-esb.service:18] Failed to parse service restart specifier, ignoring: Always
Oct 25 02:29:11 ccm.toca.local systemd[1]: [/etc/systemd/system/cliqr-esb.service:18] Failed to parse service restart specifier, ignoring: Always
Oct 25 02:29:11 ccm.toca.local systemd[1]: [/etc/systemd/system/cliqr-esb.service:18] Failed to parse service restart specifier, ignoring: Always
Oct 25 02:30:36 ccm.toca.local systemd[1]: [/etc/systemd/system/cliqr-esb.service:18] Failed to parse service restart specifier, ignoring: Always
Oct 25 02:30:36 ccm.toca.local systemd[1]: [/etc/systemd/system/cliqr-esb.service:18] Failed to parse service restart specifier, ignoring: Always
Oct 25 02:30:36 ccm.toca.local systemd[1]: [/etc/systemd/system/cliqr-esb.service:18] Failed to parse service restart specifier, ignoring: Always
Oct 25 02:40:37 ccm.toca.local systemd[1]: [/etc/systemd/system/cliqr-esb.service:18] Failed to parse service restart specifier, ignoring: Always
Oct 25 02:40:37 ccm.toca.local systemd[1]: [/etc/systemd/system/cliqr-esb.service:18] Failed to parse service restart specifier, ignoring: Always
Oct 25 02:40:37 ccm.toca.local systemd[1]: [/etc/systemd/system/cliqr-esb.service:18] Failed to parse service restart specifier, ignoring: Always
Oct 25 02:42:50 ccm.toca.local systemd[1]: [/etc/systemd/system/cliqr-esb.service:18] Failed to parse service restart specifier, ignoring: Always
[root@ccm ~]# systemctl status capacity-manager.service -l
● capacity-manager.service - CapacityManager
   Loaded: loaded (/etc/systemd/system/capacity-manager.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Thu 2018-10-25 02:28:38 UTC; 14min ago
  Process: 1250 ExecStart=/bin/sh -c . /etc/profile.d/jdk.sh;/usr/local/cliqr/apps/capacity-manager.jar (code=exited, status=1/FAILURE)
 Main PID: 1250 (code=exited, status=1/FAILURE)

Oct 25 02:28:23 ccm.toca.local sh[1250]: | /  \/  __ _  _ __    __ _   ___  _ | |_  _   _  | .  . |  __ _  _ __    __ _   __ _   ___  _ __
Oct 25 02:28:23 ccm.toca.local sh[1250]: | |     / _` || '_ \  / _` | / __|| || __|| | | | | |\/| | / _` || '_ \  / _` | / _` | / _ \| '__|
Oct 25 02:28:23 ccm.toca.local sh[1250]: | \__/\| (_| || |_) || (_| || (__ | || |_ | |_| | | |  | || (_| || | | || (_| || (_| ||  __/| |
Oct 25 02:28:23 ccm.toca.local sh[1250]: \____/ \__,_|| .__/  \__,_| \___||_| \__| \__, | \_|  |_/ \__,_||_| |_| \__,_| \__, | \___||_|
Oct 25 02:28:23 ccm.toca.local sh[1250]: | |                           __/ |                                __/ |
Oct 25 02:28:23 ccm.toca.local sh[1250]: |_|                          |___/                                |___/
Oct 25 02:28:23 ccm.toca.local sh[1250]: capacity-manager : (v4.9.1), Powered by Spring Boot : 1.5.13.RELEASE
Oct 25 02:28:38 ccm.toca.local systemd[1]: capacity-manager.service: main process exited, code=exited, status=1/FAILURE
Oct 25 02:28:38 ccm.toca.local systemd[1]: Unit capacity-manager.service entered failed state.
Oct 25 02:28:38 ccm.toca.local systemd[1]: capacity-manager.service failed.

On the rabbit VM, guacd did not start.

[root@rabbit ~]# systemctl status guacd.service
● guacd.service - LSB: Guacamole proxy daemon
   Loaded: loaded (/etc/rc.d/init.d/guacd; bad; vendor preset: disabled)
   Active: inactive (dead)
     Docs: man:systemd-sysv-generator(8)

Any suggestions here?

[Shaun Roberts]

I assume you went through the configuration steps outlined at https://docs.cloudcenter.cisco.com/display/CCD482/Phase+4+Overview

Was there any issues there in configuring?

If not and you still have issues, I suggest you raise a TAC case and work with the team on the issue.

[michgri]

I did follow the instructions, but apparently using an IP address for the database server doesn't work in a standalone environment. I changed CCM to use the name I put in /etc/hosts and now I'm able to log in. (I don't know why I didn't do that in the first place.)

I am going to note that the documentation is a little vague at points, which is not ideal for a product as complicated as CloudCenter. For instance, you need to specify the role in order to run the network prerequisite scripts, but the examples only show some of them and the error message doesn't tell you. I had to read through the python script in order to figure them out.

[Alexandre Semenov]

hello,

can you please share the configuration of DB because I have the same issue? is it portgre standalone? In CCM you use name of Postgre DB insteed of IP or name of CCM ?

thanks,

[michgri]

I believe what I meant was, when you run /usr/local/cliqr/bin/ccm_config_wizard.sh and use the Configure Database option, it only worked when I configured a host name. So postgres.toca.local worked but 10.10.10.10 did not.

I literally just reinstalled CloudCenter (lab environment) so I can go into a little more detail. What seems to work best is:

• Configure DNS names for everything (or sync /etc/hosts between all of your CloudCenter VMs).
• Configure reverse DNS entries for everything unless you are using /etc/hosts.
• Make sure you configure the host name for each CloudCenter VM to match the DNS entry. CCM seems to be the most critical here.
• Follow the instructions and verify that you've entered the correct information. Most of the problems I've had were because I made a mistake.

One other odd thing I ran into was that apparently /usr/local/cliqr/bin/ccm_config_wizard.sh does not like passwords beginning with an @ symbol. I tried that, and the database would not connect from CCM even though it worked just fine with that password using psql. I changed the password in the wizard to something without an initial @ symbol, and it worked fine.

The product does desperately need some kind of installation dashboard that checks whether or not you're connecting properly to the database, etc. I think that would save a lot of frustration (along with better examples for the checker python script).

I don't know if it is helpful, but here's my latest json file with deployment details (lab environment, no private information revealed, I may have messed up the formatting or the brackets when I copied it). All domain names are in DNS, the VMs are configured with static IP addresses, the name server is configured in /etc/resolv.conf, the host name is configured in /etc/hostname and reverse DNS entries for each VM's IP address point back to the appropriate domain names.

{
   "CloudCenterComponents":{
      "CCM":{
         "CCM_SA_IP":"ccm.toca.local",
         "MGMTPOSTGRES_IP":"postgres.toca.local",
         "mode":"NON-HA-STANDALONE"
      },
      "CloudRegions":[
         {
            "components":{
               "AMQP":{
                  "AMQP_IP":"rabbit.toca.local",
                  "mode":"NON-HA"
               },
               "CCO":{
                  "CCO_IP":"cco.toca.local",
                  "mode":"NON-HA"
               }
            },
            "name":"TOCAvmware"
         }
      ],
      "LogCollector":{
         "LogCollector_IP":"log.toca.local",
         "mode":"NON-HA"
      },
      "REPOS":{
         "BUNDLE_STORE":"http://cdn.cliqr.com",
         "DOCKER_REGISTRY":"http://repo.cliqrtech.com:5000",
         "PACKAGE_STORE":"http://repo.cliqrtech.com"
      }
   }
}

[Mu'awiyah]

1. Configure DNS names for everything (or sync /etc/hosts between all of your CloudCenter VMs).
2. Configure reverse DNS entries for Walgreens Listens everything unless you are using /etc/hosts.
3. Make sure you configure the host name for each CloudCenter VM to match the DNS entry.