Hello Experts,
I need your assistance in resolving one of the issues which my colleague is facing. His CUCM application responds intermittently to an API request.
Below is the code for querying Cisco CallManger API for the Status of Services.
######################################################################################################################################################
#Use hard coded credentials
$CUCMAPIUser = "myaccount"
$CUCMAPIPassword = Get-Content ".\password.txt" | ConvertTo-SecureString
#Create a PSCredential Object for use with the New-WebServiceProxy commandlet
$CUCMAPICredential = New-Object System.Management.Automation.PSCredential($CUCMAPIUser,$CUCMAPIPassword)
#Server IP Address
$CUCMAddress = "hcs-zur-impw180.zurich.com"
$CUCMPort = ":8443"
$CUCMServerURI = $CUCMAddress+$CUCMPort
[System.Net.ServicePointManager]::ServerCertificateValidationCallback = {$true}
#[Net.ServicePointManager]::SecurityProtocol = "tls12, tls11, tls"
#[Net.ServicePointManager]::SecurityProtocol = "ssl3"
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls, [Net.SecurityProtocolType]::Tls11, [Net.SecurityProtocolType]::Tls12, [Net.SecurityProtocolType]::Ssl3
[Net.ServicePointManager]::SecurityProtocol = "Tls, Tls11, Tls12, Ssl3"
#Request Body for Services Running on Server
$services = @"
<!--Control Center - soapGetServiceStatus API Request-->
<!--If you leave service status empty you will get the status of all services, if not you can list specific services and get information for requested services only-->
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:soap="http://schemas.cisco.com/ast/soap">
<soapenv:Header/>
<soapenv:Body>
<soap:soapGetServiceStatus>
<soap:ServiceStatus></soap:ServiceStatus>
</soap:soapGetServiceStatus>
</soapenv:Body>
</soapenv:Envelope>
"@
$url = "https://$CUCMServerURI/controlcenterservice2/services/ControlCenterServices?wsdl"
#Using Stored Credentials
$results = Invoke-WebRequest -Method post -Uri $url -UseBasicParsing -ContentType 'application/json; charset=utf-8' -Body $services -Credential $CUCMAPICredential
$apistatuscode = $results.StatusCode
return $apistatuscode
##########################################################################################################################################################
When the API request works we get this response
However, intermittently the request fails with the following
The error message “ The request was aborted: Could not create SSL / TLS secure channel” is a result of not specifying the version of TLS / SSL which is specified in the code.
#[Net.ServicePointManager]::SecurityProtocol = "tls12, tls11, tls"
#[Net.ServicePointManager]::SecurityProtocol = "ssl3"
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls, [Net.SecurityProtocolType]::Tls11, [Net.SecurityProtocolType]::Tls12, [Net.SecurityProtocolType]::Ssl3
[Net.ServicePointManager]::SecurityProtocol = "Tls, Tls11, Tls12, Ssl3"
So as I stated the code is used to query the status of services. This is running every 5 minutes. However, intermittently and only on certain devices we are unable to poll for a specific amount of time and then it will respond again with no intervention.
Below is from PRTG you can see the intermittent response. The red line is the failure to respond.
He has also taken a Wireshark Trace on the PRTG probe and I have attached the packet capture for the same as well.
The two captures were filtered from an original capture taken around 14.15. hcs-zur-ucmb176 failed for an hour between 13.45 & 14.45 today.
The CUCM hcs-zur-ucma176 is a GOOD API Request (10.44.88.226)
The CUCM hcs-zur-ucmb176 is a BAD API Request (10.44.88.227)
The Probe has the address of 147.152.34.143
You can see in hcs-zur-ucma176 capture the TLS handshake and then the transfer of data.
However, in the hcs-zur-ucmb176 capture the TLS handshake fails with the CUCM sending a Close Notify
Why is CUCM doing this intermittently?
I can't see anything wrong with the TLS negotiation in the failure scenario, at least up until the server sends the unexpected Close Notify. I do notice that in the Client Hello the client lists a different set of available Cipher Suites, i.e. 97 in the working scenario and 28 in the non-working scenario; however I don't see that the UCM has any problem selecting a suite/algorithm (even though it selects different ones) in either scenario. Curious as to why the client is doing this - are the requests coming from different apps on the client host..?
It seems that connections are happening way more than once every 5 minutes:
Perhaps all of these are not Control Center Service API requests? However, if they are, it seems that querying for running services on UCM is a fairly heavy tasks for some reason (it takes a long while from the CLI), so maybe CUCM is getting bogged down..?
Could it be you have more instances of this script running than you think, maybe old versions..?
It seems that the root cause is higher up the UCM stack than TLS, so suggest opening a case with DevNet Developer Support, who can help analyze UCM-side logs: https://developer.cisco.com/site/support/
See this link for detail on gathering logs specific to the Service XML APIs: https://developer.cisco.com/site/sxml/help/enable-collect-logs/
If the requests aren't making it to the SOAP layers, then investigation might need to go down into the Tomcat/Apache logs.
