CSPC 2.9 Upload Failed - Internal Server Error

[Jason Bandy]

Trying to setup connectivity to Smartnet Total Care Portal.

Getting the error:

Upload UPLOADING: FAILED; Certificate: CSP0009050381; Directory: _SmartnetUpload/10_2; File: transport-inventory-1603716300091.zip; Other Details: Internal Server Error

Any ideas of what to try?

[Justin Sprake]

The log located at the below directory path will often provide additional context for upload failures:

/opt/ConcsoTgw/tail-end-gateway-decoupled/bin/nohup.out

Additionally, if you execute a wget command towards concsoweb-prd.cisco.com as root you will also be able to quickly access the outbound access for your CSPC.

[Jason Bandy]

Justin, just sent you a PM

[Jason Bandy]

Seeings this in the logs:

1768: 0.0.0.0: Oct 27 2020 09:55:03.554 -0500: %CONN_TEG-6-CONN_STR_INFO: %[ch=connectivityteg][mid=10003][requestTranId=99999999][applianceId=CSP0009050381][customerId=7569328][source=Connectivity][component=CONN_TEG][threadName=qtp559830197-25][className=com.cisco.ca.csp.conn.gateway.tailend.core.command.SendFileCommandHTTPS][methodName=uploadFileILB][lineNumber=468][isRemote=false]: message: uploadFileILB...
1769: 0.0.0.0: Oct 27 2020 09:55:04.592 -0500: %CONN_TEG-6-CONN_STR_INFO: %[ch=connectivityteg][mid=10003][requestTranId=99999999][applianceId=CSP0009050381][customerId=7569328][source=Connectivity][component=CONN_TEG][threadName=qtp559830197-25][className=com.cisco.ca.csp.conn.gateway.tailend.core.command.SendFileCommandHTTPS][methodName=uploadFileILB][lineNumber=599][isRemote=false]: message: http status code-1:400
1770: 0.0.0.0: Oct 27 2020 09:55:04.592 -0500: %CONN_TEG-6-CONN_STR_INFO: %[ch=connectivityteg][mid=10003][requestTranId=99999999][applianceId=CSP0009050381][customerId=7569328][source=Connectivity][component=CONN_TEG][threadName=qtp559830197-25][className=com.cisco.ca.csp.conn.gateway.tailend.core.command.SendFileCommandHTTPS][methodName=uploadFileILB][lineNumber=635][isRemote=false]: message: Send File failed,http status code:400
1771: 0.0.0.0: Oct 27 2020 09:55:04.592 -0500: %CONN_TEG-6-CONN_STR_INFO: %[ch=connectivityteg][mid=10003][requestTranId=99999999][applianceId=CSP0009050381][customerId=7569328][source=Connectivity][component=CONN_TEG][threadName=qtp559830197-25][className=com.cisco.ca.csp.conn.gateway.tailend.core.command.SendFileCommandHTTPS][methodName=uploadFileILB][lineNumber=638][isRemote=false]: message: Send File failed,http status:Bad Request
1772: 0.0.0.0: Oct 27 2020 09:55:04.593 -0500: %CONN_TEG-6-CONN_STR_INFO: %[ch=connectivityteg][mid=10003][requestTranId=99999999][applianceId=CSP0009050381][customerId=7569328][source=Connectivity][component=CONN_TEG][threadName=qtp559830197-25][className=com.cisco.ca.csp.conn.gateway.tailend.core.rest.TailEndRestApi][methodName=uploadFile][lineNumber=157][isRemote=false]: message: Retry response2,Execute response status:FAILURE
1773: 0.0.0.0: Oct 27 2020 09:55:04.593 -0500: %CONN_TEG-6-CONN_STR_INFO: %[ch=connectivityteg][mid=10003][requestTranId=99999999][applianceId=CSP0009050381][customerId=7569328][source=Connectivity][component=CONN_TEG][threadName=qtp559830197-25][className=com.cisco.ca.csp.conn.gateway.tailend.core.rest.TailEndRestApi][methodName=uploadFile][lineNumber=160][isRemote=false]: message: Retry response2,Execute response message:A connectivity exception occured while processing the request. The Exception is :: HTTPs TransportMode has Failed 400:Bad Request
1774: 0.0.0.0: Oct 27 2020 09:55:04.593 -0500: %CONN_TEG-6-CONN_STR_INFO: %[ch=connectivityteg][mid=10003][requestTranId=99999999][applianceId=CSP0009050381][customerId=7569328][source=Connectivity][component=CONN_TEG][threadName=qtp559830197-25][className=com.cisco.ca.csp.conn.gateway.tailend.core.rest.TailEndRestApi][methodName=uploadFile][lineNumber=148][isRemote=false]: message: Request ID :135201360801939 Retry after 300000 milliseconds

[Jason Bandy]

TimeStamp set in the token - 1603814110580. Nonce set in the token - 6482248328342984948
TOKEN IS - ExKoQrM9DJCqVe2D2zL1PEtpk8w=
2020-10-27 10:55:11,410 ==== thread: qtp1743689686-23 ==== INFO >
connectivityteg.uploadFileILB() => (line: 599)
message: http status code-1:400

2020-10-27 10:55:11,410 ==== thread: qtp1743689686-23 ==== INFO >
connectivityteg.uploadFileILB() => (line: 635)
message: Send File failed,http status code:400

2020-10-27 10:55:11,411 ==== thread: qtp1743689686-23 ==== INFO >
connectivityteg.uploadFileILB() => (line: 638)
message: Send File failed,http status:Bad Request

2020-10-27 10:55:11,411 ==== thread: qtp1743689686-23 ==== INFO >
connectivityteg.uploadFile() => (line: 157)
message: Retry response2,Execute response status:FAILURE

2020-10-27 10:55:11,412 ==== thread: qtp1743689686-23 ==== INFO >
connectivityteg.uploadFile() => (line: 160)
message: Retry response2,Execute response message:A connectivity exception occured while processing the request. The Exception is :: HTTPs TransportMode has Failed 400:Bad Request

2020-10-27 10:55:11,412 ==== thread: qtp1743689686-23 ==== INFO >
connectivityteg.uploadFile() => (line: 148)
message: Request ID :138808460167310 Retry after 300000 milliseconds

[root@localhost collectorlogin]# wget https://concsoweb-prd.cisco.com
--2020-10-27 10:56:20-- https://concsoweb-prd.cisco.com/
Resolving concsoweb-prd.cisco.com (concsoweb-prd.cisco.com)... 72.163.7.113
Connecting to concsoweb-prd.cisco.com (concsoweb-prd.cisco.com)|72.163.7.113|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://cloudsso.cisco.com/as/authorization.oauth2?response_type=code&client_id=wam_prod_ac&redirect_uri=https%3A%2F%2Fconcsoweb-prd.cisco.com%2Fpa%2Foidc%2Fcb&state=eyJ6aXAiOiJERUYiLCJhbGciOiJkaXIiLCJlbmMiOiJBMTI4Q0JDLUhTMjU2Iiwia2lkIjoiZ20iLCJzdWZm... [following]
--2020-10-27 10:56:21-- https://cloudsso.cisco.com/as/authorization.oauth2?response_type=code&client_id=wam_prod_ac&redirect_uri=https%3A%2F%2Fconcsoweb-prd.cisco.com%2Fpa%2Foidc%2Fcb&state=eyJ6aXAiOiJERUYiLCJhbGciOiJkaXIiLCJlbmMiOiJBMTI4Q0JDLUhTMjU2Iiwia2lkIjoiZ20iLCJzdWZm...
Resolving cloudsso.cisco.com (cloudsso.cisco.com)... 72.163.4.74, 2001:420:1101:4::b
Connecting to cloudsso.cisco.com (cloudsso.cisco.com)|72.163.4.74|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 10762 (11K) [text/html]
Saving to: ‘index.html’

index.html 100%[==========================================================================================================>] 10.51K --.-KB/s in 0s

2020-10-27 10:56:21 (37.2 MB/s) - ‘index.html’ saved [10762/10762]

[Jason Bandy · Accepted solution]

So ended up kicking off a manual upload and that seemed to jump start everything as a fyi for anyone else having issues. Also dont schedule uploads, the only profile you should have to work out of is the "Manage Data Collection" profile to do the scheduled scans.