Upload to SNTC Portal Complete?

[Matthew Lounsbury]

I originally installed the PSS Collector via the OVA PSS-CSPCServer-2.6.0.1-OVF10.ova.  I generated a new CSP-C entitlement in the SNTC Portal, and applied it to that collector.  After doing some reading of other threads around here, and doing a search, I found there was a SNTC Collector with an OVA of SNTC-CSPC2.6.1-RC1-OVF10.ova.  I decided to shut down my existing collector and redeploy the new SNTC Collector.  I did not generate a new CSP-C entitlement, and uploaded the existing one I had.  I received an error during the process that said:

Entitlement update done locally, but failed on tail-end gateway. No response from connectivity tail-end gateway. CSPC not run without connectivity tail-end gateway.

I found this post: https://supportforums.cisco.com/discussion/12703286/error-tail-end-gateway-during-entitlement-file-load

It let me log into the application and I was able to add devices to the inventory.  I then did a collection and had it upload.  I did not see any errors in the log, but I'm not seeing anything on the portal yet.  I did this after 3 or 4pm EST yesterday, and I know it could take some time for it to show up in the portal.  Is it possible to see if you can see the upload completed and that there is processing going on with the file?  If not, I can remove the license, delete the existing entitlement from the portal and then generate a new one and upload it to the collector (if that is the process I should take).

Would anyone be able to assist?

[Lynden Price]

Hi Matthew,

When you go to the SNTC portal, do you see an entry under Library > Administration > Upload Processing? It should show your uploads as either "Success" "Failure" or "In Processing"

Thanks,

Lynden

[Matthew Lounsbury]

Hi Lynden,

I don't see anything on the sidebar like that.  Attached is a screenshot of my sidebar.  I'm guessing maybe it doesn't because of what I did with the entitlement file?  Should I remove the existing entitlement from the portal and the collector and start fresh?

Thanks!

-Matt L.

[Matthew Lounsbury]

I decided to try and go ahead and delete the collector entitlement from the SNTC portal, and then re-add it.  I used the same collector name but changed the inventory name.  It let me create the entitlement but when I attempted to download it, I received the following error:

Download entitlement failed. Please try again. If the issue persists, find help in the community.

I then decided to start all over again.  I deleted the entitlement, I deleted the VM in my environment, and redeployed it as a new VM with a different name.  I created a new entitlement and attempted to download it, but I'm still receiving the same error.  Is there something that can be done so I can download the entitlement file?

[Lynden Price]

If you cannot see the "Upload Processing" tab, then you will need Customer Administrator level access in order to view the uploads. You can request this from your delegated administrator who grants that access in CSAM:

https://cdceb.cloudapps.cisco.com/csam/login.do?action=home

Thanks,

Lynden

[Matthew Lounsbury]

Lynden,

Thanks for the response.  So I'm the delegated administrator of the account, and I went in and gave myself both Customer User and Customer Administrator.  I logged out and back in, no change.  Didn't see the tab, and I cannot download the entitlement.  Thought maybe it was because I added both roles, so I removed the Customer User.  Logged out and back in, same thing.  No tab, cannot download the entitlement.

Is there some delay to the amount of time before the update takes place in the SNTC portal?  Or is something else going on?

Thanks.

-Matt L.

[Lynden Price]

Hi Matt,

Try a different browser to confirm it isn't some kind of java/cache issue. Under Dashboards > Admin in the SNTC Portal, does it show a users list? If so, check that it has you as a Customer Admin.

Thanks,

Lynden

[Matthew Lounsbury]

Hey Lynden,

Yesterday I attempted to use another browser, and I still do not see Upload Processing under Library => Administration.  I was able to use that browser to download the entitlement file.  I downloaded that file and then applied it to the collector.  I still received the same message:

Entitlement update done locally, but failed on tail-end gateway. No response from connectivity tail-end gateway. CSPC not run without connectivity tail-end gateway.

We do have a proxy here, and I did enter the proxy information into the CLI.  I've even run the nc -v command and run a wget to ensure that I'm getting out of the proxy without an issue.

Today around 11:53am ET, I ran an inventory job and it completed, but I am still not seeing anything in the portal saying that an upload is processing.  I've tried three different browsers on my PC, and three other browsers on a different PC.  All show the same exact thing.  I also don't even see anything on the collector as far as log files go to ensure that the upload even completed successfully.  Is there a place to see this?

Thanks again.

-Matt L.

Edited to add - You asked about "Under Dashboards > Admin in the SNTC Portal, does it show a users list?"  I don't have anything under dashboards.  I created a new dashboard, but going into the dashboard settings, and add dashlets, there's nothing to add that shows users.

[Lynden Price]

Hi Matt,

One way you can test is to set up an Upload Profile and run it to see if it completes or gives a failure message.

I've attached a document explaining how to do this.

You can also find some possible troubleshooting steps here:

https://supportforums.cisco.com/blog/12987426/self-service-onboarding-where-my-upload-cspc-collector-troubleshooting

Thanks,

Lynden

[Matthew Lounsbury]

Hey Lynden,

Thanks for the fast response.  I did check everything in the link, and it all matches up.  I then created the upload profile, and I tested an upload and the screenshot below are the results.  From what I can see, everything looks good?

I left it like that for a few minutes, and the last line never changed.  Then I went into the Upload Run Now Jobs and I see that it failed:




So now I can see that the upload is failing.  Is it possible to tell where it is failing in the upload process?  I entered the proxy information using the conf proxy command in the CLI.  Doing some Google searching finds an old version of the collector documentation that lists some other commands required to get the proxy working:

http://www.cisco.com/c/dam/en/us/td/docs/net_mgmt/smart_portal/Common_Services_Platform_Collector_Quick_Start_Guide_2_2.pdf

Is this required any longer or is just the one command required?

Thanks.

-Matt L.

[Lynden Price]

Based on the output, it looks like it is failing during the actual transfer, so I agree that it is probably the proxy. Do you have access to the proxy itself? It might have more detailed information about why it is failing, or at least you'll be able to tell if the traffic is making it to the proxy.

The older document no longer applies. You should be able to set the proxy with:

conf proxy <ipaddress> <port> <username> <password>

ex. conf proxy 192.168.1.1 8080 cisco cisco

and then verify that it is set with

show ip

Thanks,

Lynden

[Matthew Lounsbury]

Hey Lynden,

So I attempted a couple of other things this morning in an attempt to get the upload to work.  I changed the username and password I used on the proxy as the service account we use has a special character in it, and I know *nix has some problems with that.  Still got the same error.  I checked with our proxy/firewall team, and this is the e-mail response I got from them:

Good Morning Matt,

Looking over the 12/22 9:44 am – 10:04 timeframe, I see your application attempting to connect directly through the firewall as well as a through the proxy.

Regarding the Proxy traffic,

I see successful communication from your application to concsoweb-prd.cisco.com and sso.cisco.com.

Regarding the direct firewall connectivity(not allowed via current policy), I see connections to the following destinations blocked:

157.189.192.67

198.133.219.53

72.163.7.138

72.163.4.161 (www.cisco.com)

72.163.7.113 ( concsoweb-prd.cisco.com )

Can you confirm if the top 3 destination addresses are required for this data upload?

So it looks like we got two connections through the proxy, but then the collector attempted to perform a direct connection through the proxy.  Is this normal?  Also, the first IP address they are seeing in the list does not appear to be a Cisco IP, but it looks like the collector is attempting to contact it.  Any idea what that IP address is and if it does require a connection?  It looks like we might require some firewall changes to allow these connections, even though the collector has a proxy configured.

Thanks again.

-Matt L.

[Lynden Price]

Thank you for the investigation Matt, that's actually very interesting. The top 3 addresses are the old upload servers for the previous version of the service. They aren't required. I'm not sure why it's still trying to contact them. (You can do an nslookup to see that they are Cisco)

• 72.163.7.113 (TCP 443) concsoweb-prd.cisco.com 
• 72.163.4.161 (TCP 443/80) www.cisco.com
• 72.163.7.60 (TCP 443/80) dl.cisco.com
• 72.163.7.60 (TCP 443/80) dl1.cisco.com
• 173.37.146.12 (TCP 443/80) dl2.cisco.com
• 173.37.144.208 (TCP 443/80) sso.cisco.com

Those should be the required IP addresses. The first is needed for the upload itself, the rest are needed for downloading patches.

Thanks,

Lynden

[Matthew Lounsbury]

Lynden,

Happy New Year!  Sorry for the delay in responding, I have been out of the office since 12/23.

Just curious, what exactly is the collector using the proxy for?  Based on what my security team is telling me, the collector is connecting through the proxy to 72.163.7.113 (TCP 443) concsoweb-prd.cisco.com, but then after that it is attempting a direct connection without going through the proxy.  Shouldn't the system be using the proxy for every outbound connection it makes to Cisco?  Wouldn't configuring the application in this way negate a lot of people having to make firewall changes on their systems to allow this traffic?

Thanks.

-Matt

[Lynden Price]

The collector should be using the proxy for the uploads, so it should always be using the proxy to connect to concsoweb-prd.cisco.com. It should be using the proxy for every outbound connection.

When they say "after" what exactly do they mean? Is this during an upload? I'd like to try to replicate this. 

Thanks,

Lynden