Unable to upload to backend

[Ian Bradshaw]

Hey there,

Hoping someone can shed some light on this;

Recently setup my collector, CSPC 2.5.2.1 for SNTC

When logged on to the server as root on the CLI i can form a successful connection to cisco on 443 (i believe thats the IP it needs to talk to, however its open outbound so it can pretty much talk to anything for now. 

# nc -v 72.163.7.113 443
Connection to 72.163.7.113 443 port [tcp/https] succeeded!

and yet when trying to upload the licence to the GUI, or run an upload profile i an unable to talk to the backend. 

When trying to load the licence file, i get the following error;

Entitlement update done locally, but failed on tail-end gateway. No response from connectivity tail-end gateway. CSPC not run without connectivity tail-end gateway.

This appears to be the same as this user Link. However for that user the upload seamed to have worked. This is not the case for me.

Regards,

Ian

[CC Cisco]

Hello Ian,

Could you share a screenshot of the error message you see when trying to run an upload profile?

If you have root access, could you also attach the log file:

/opt/ConcsoTgw/tail-end-gateway-decoupled/bin/nohup.out

One other thing which might be an issue is that there could be a transparent proxy between the collector and the internet, if this is the case you would need to make sure that it has the Cisco IPs white listed for https traffic.

One other thing you could test grabbing the output of the below command to see if we can spot any errors.

wget https://72.163.7.113/HeadEndWebAppServlet/ --no-check-certificate

Regards,

Cesar

[Ian Bradshaw]

Hey Cesar,

Great to hear from you,

For the Nohup file i tried to pull it, but it looks like its empty. see attached

The wget you suggested delivered some interesting see attached

Screenshot of error attached

Ian

[CC Cisco]

Hi Ian,

Thanks for the update.

Based on the info you have provided, I see a couple of things:

The wget response actually looks normal, it seems that the https session can be established, which is good news.

The fact that nohup.out is empty is what makes me think that the upload module (tail end gateway) isn't running.

Could you please run the below command and see if anything gets put into nohup.out

service concsotgw restart

Restarting the tail end gateway should take a couple of minutes. nohup.out should have a bunch of entries, some of the errors listed will be expected. After a few minutes, ensure the TEG is up by running the command:

service concsotgw status

At this point, please try pushing a new upload and immediately after (specially if it fails), grab a copy of nohup.out and post that back here. Hopefully that will allow us to determine the root cause.

Regards,

Cesar

[Ian Bradshaw]

Cesar,

I restarted the service, it appears to restart well and the status says its running.

Then i enter the GUI, run the upload profile, which fails

I then check on the service and the nohup.out log you can see the output in the attached.

Regards,

Ian

[CC Cisco · Accepted solution]

Hi Ian,

Thanks for that info, it seems that your hostname is missing from the hosts file.

You can correct that by editing the /etc/hosts file and adding the entry for your hostname next to the loopback IP address (127.0.0.1).

You can also add your own server's IP address, but make sure it also has the hostname next to it, the loopback IP must always have the server's hostname next to it.

Your hosts file should look similar to the below:

[root@per5-netops-sntc-01 collectorlogin]# cat /etc/hosts
72.163.9.100 nettools-upload.cisco.com
192.135.250.12 nettools-upload.cisco.com
64.103.36.134 nettools-upload.cisco.com
72.163.7.113 concsoweb-prd.cisco.com
173.37.186.17 concsoweb-prd.cisco.com
72.163.4.161 www.cisco.com
72.163.7.60 dl.cisco.com
72.163.7.60 dl1.cisco.com
173.37.146.12 dl2.cisco.com
173.37.144.208 sso.cisco.com
::1     localhost.localdom localhost localhost ip6-localhost ip6-loopback
127.0.0.1    localhost    per5-netops-sntc-01
123.111.222.123    localhost    per5-netops-sntc-01

In the above example 123.111.222.123, would be your server's IP address.

After this is changed, please restart the entire CSPC service, and try to upload one more time.

service cspc restart

If you are wondering what the recommended way to setup a custom hostname without causing this to break. That would be by setting up the custom hostname from the admin shell using the command "hostname", rather than doing it directly from the linux CLI. In this case though, we can just apply the workaround.

Please try the steps above and reply back with any updates.

Regards,

Cesar

[Ian Bradshaw]

Cesar,

I have made the changes you mentioned, and we now have plenty of output in the nohup log

My host file currently looks like this

[root@per5-netops-sntc-01 ~]# cat /etc/hosts
72.163.9.100 nettools-upload.cisco.com
192.135.250.12 nettools-upload.cisco.com
64.103.36.134 nettools-upload.cisco.com
72.163.7.113 concsoweb-prd.cisco.com
173.37.186.17 concsoweb-prd.cisco.com
72.163.4.161 www.cisco.com
72.163.7.60 dl.cisco.com
72.163.7.60 dl1.cisco.com
173.37.146.12 dl2.cisco.com
173.37.144.208 sso.cisco.com
127.0.0.1 localhost per5-netops-sntc-01
X.X.X.X localhost per5-netops-sntc-01
::1 localhost.localdom localhost localhost ip6-localhost ip6-loopback
[root@per5-netops-sntc-01 ~]#

My IP is replaced by X.X.X.X above.

The upload still fails however, same error as previously.

[CC Cisco]

Hi Ian,

Thank you for the response.

Looking at the logs, I can see one exception repeating over and over:

javax.jms.JMSException: Could not connect to broker URL: tcp://localhost:61616. Reason: java.net.ConnectException: Connection refused
    at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:35)
Caused by: java.net.ConnectException: Connection refused

This normally means that an internal connection cannot be established because another process is using that port. Most of the time this other connection will end itself allowing the new required connection to be successful.

I would say, please try one more upload since it has been a while. If you still have issues, I'd suggest to stop the cspc service, wait a few minutes and then start the cspc service one more time.

service cspc stop

service cspc start

Regards,

Cesar

[Ian Bradshaw]

Cesar,

I tried the steps you listed above,  

service cspc stop

wait a few minutes

service cspc start

however that didnt seam to resolve the issue. I even tried a complete reboot of the VM to no avail.

However after reviewing the log on more time, i had a hunch to retry the licence registration. This was now successfull, the prompt told me the server would need to be rebooted to apply changes, so i did that. 

Now mu upload profile reports successful. 

Thankyou very much for your help cesar. 

I think the key to this problem is to check the /etc/hosts, hostfile.

You need to ensure the servers IP is listed in the 127.0.0.1 entry, and also in its own actual IP. 

[twylyghtcisco]

Hello Caesar,

Getting a similar issue to what has been described here.  I have attached a PDF of the HOSTS file, the headend test and the nohup.out

I appreciate any and all help you could offer in this matter.

Regards,

Jim Gambill

[CC Cisco]

Good morning Jim,

I notice 2 things:

1.- It looks like your hostname is "cspc", however on the hosts file, it is missing from the line with 127.0.0.1

The line should look like this:

127.0.0.1  localhost  cspc

In addition, I'm not sure if you have deployed the proper image, can you confirm that the name of the file has "SNTC" in the filename. If the file has "PSS", then the image is incorrect and you will need to redeploy.

Regards,

Cesar

[twylyghtcisco]

Cesar,

My thanks for the quick reply.  New HOSTS is as follows:

72.163.9.100 nettools-upload.cisco.com
192.135.250.12 nettools-upload.cisco.com
64.103.36.134 nettools-upload.cisco.com
72.163.7.113 concsoweb-prd.cisco.com
173.37.186.17 concsoweb-prd.cisco.com
72.163.4.161 www.cisco.com
72.163.7.60 dl.cisco.com
72.163.7.60 dl1.cisco.com
173.37.146.12 dl2.cisco.com
173.37.144.208 sso.cisco.com
127.0.0.1 localhost cspc
[redacted] localhost per5-netops-sntc-01

Also, in reviewing the original OVA that was applied, it shows as PSS rather than SNTC (PSS-CSPCServer-2.6.0.1-OVF10.ova).  I will revisit this with our VM team to get this re-deployed with the correct one (SNTC-CSPC2.6.1-RC1-OVF10.ova).

[CC Cisco]

Great, glad I could help.

I would definitely suggest in the future to go ahead and start a new discussion. You can always reference back to discussions with similar issues (such as this one), by pasting the link on your post.

This will help keep disussions easier to navigate.

In addition, if you're going to have a custom hostname, it must always be next to the 127.0.0.1 entry. I assumed it was just "cspc", but if you will have "per5-netops-sntc-01" it should also be on the 127.0.0.1 line.

Regards,

Cesar