This document was generated from CDN thread
Created by: JAMES DEPHILLIP II on 13-06-2013 05:46:10 PM
Are there limits to the number of AXL get request and EMAPI requests?
We have been having issues where it once a certain load is placed onto
the Tomcat service (via EMAPI and AXL) the CallManager will start
failing to respond to a TCP SYN handshake on 8443. We confirmed with a
Wireshark capture on CallManager that some incoming TCP SYNs are not
responded to at all. Even TCP SYN retransmits from the same source
port are not responded to.
When we piloted the application in our production environment, the
load was fairly low (roughly 200-450 application requests per day) and
we didn't encounter issue. Right as we implemented to all users the
number of application requests jumped to about 12,000 per day, the
problem started the same day.
The percentage of failures increases daily. The first day in
production, about 1% of our application requests had EMAPI/AXL
failures. The second day it was about 3%. The percentage of failures
increased every day (even on weekends when there was less load), until
a week later when roughly 25% of requests were failing.
We restarted the tomcat service but that didn't resolve the issue.
When we rebooted the CallManager server, the error percentage dropped
back to 0 but then it started slowly increasing just like the first
day we put it in production.
The application that I am working on does axl lookups for device
logout profile and user profile. It will then do an EMAPI lookup for
the logged in user and EMAPI login or logout and sometimes both.
We've ruled out network issues, because we've confirmed in a Wireshark
capture that the CallManager server isn't responding to some TCP SYN
requests. Because TCP SYNs aren't being responded to, we think it must
be something on CallManager. Also, because of the fact that the error
percentage increases daily, it seems almost like a memory leak.
Does anyone have any ideas why this would happen or what we could do
to narrow down the problem?
Thanks,
Jim
Subject: RE: AXL/EMAPI Memory Leak?
Replied by: David Staudt on 14-06-2013 02:43:14 PM
With AXL requests of any scale, it is important to re-use the session-IDs provided in the two response cookies
POST https://10.88.131.141:8443/axl/ HTTP/1.1
Accept-Encoding: gzip,deflate
Content-Type: text/xml;charset=UTF-8
Authorization: Basic QWRtaW5pc3RyYXRvcjpjaXNjbyExMjM=
SOAPAction: "CUCM
B ver=8.5"
Content-Length: 249
Host: 10.88.131.141:8443
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.1.1 (java 1.5)
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ns="http://www.cisco.com/AXL/API/8.5">
<soapenv:Header/>
<soapenv:Body>
<ns:getCCMVersion>
</ns:getCCMVersion>
</soapenv:Body>
</soapenv:Envelope>
---------------------
HTTP/1.1 200 OK
Set-Cookie: JSESSIONIDSSO=C5374FFF9F8F25DA2A5C9DE1787D94B7; Path=/; Secure
Set-Cookie: JSESSIONID=25CD2D2A71298D55118201EEF3DE0550; Path=/axl; Secure
Content-Type: text/xml;charset=UTF-8
Content-Length: 347
Date: Fri, 14 Jun 2013 19:28:38 GMT
Server:
<?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><ns:getCCMVersionResponse xmlns:ns="http://www.cisco.com/AXL/API/8.5"><return><componentVersion><version>8.5.1.10000(26)</version></componentVersion></return></ns:getCCMVersionResponse></soapenv:Body></soapenv:Envelope>
---------------------
POST https://10.88.131.141:8443/axl/ HTTP/1.1
Accept-Encoding: gzip,deflate
Content-Type: text/xml;charset=UTF-8
Cookie: JSESSIONIDSSO=C5374FFF9F8F25DA2A5C9DE1787D94B7; Path=/; Secure
Cookie: JSESSIONID=25CD2D2A71298D55118201EEF3DE0550; Path=/axl; Secure
SOAPAction: "CUCMB ver=8.5"
Content-Length: 249
Host: 10.88.131.141:8443
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.1.1 (java 1.5)
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ns="http://www.cisco.com/AXL/API/8.5">
<soapenv:Header/>
<soapenv:Body>
<ns:getCCMVersion>
</ns:getCCMVersion>
</soapenv:Body>
</soapenv:Envelope>
---------------------
HTTP/1.1 200 OK
Content-Type: text/xml;charset=UTF-8
Content-Length: 347
Date: Fri, 14 Jun 2013 19:33:23 GMT
Server:
<?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><ns:getCCMVersionResponse xmlns:ns="http://www.cisco.com/AXL/API/8.5"><return><componentVersion><version>8.5.1.10000(26)</version></componentVersion></return></ns:getCCMVersionResponse></soapenv:Body></soapenv:Envelope>
Subject: RE: AXL/EMAPI Memory Leak?
Replied by: JAMES DEPHILLIP II on 16-06-2013 09:53:51 AM
Thanks David does this apply to EMAPI as well? We noticed more issues with EMAPI than AXL...
Subject: RE: AXL/EMAPI Memory Leak?
Replied by: David Staudt on 17-06-2013 11:11:13 AM
All (most?) of the UCM web services APIs run through the on-board Tomcat session manager, so believe E/M would benefit as well.
Created by: JAMES DEPHILLIP II on 13-06-2013 05:46:10 PM
Are there limits to the number of AXL get request and EMAPI requests?
We have been having issues where it once a certain load is placed onto
the Tomcat service (via EMAPI and AXL) the CallManager will start
failing to respond to a TCP SYN handshake on 8443. We confirmed with a
Wireshark capture on CallManager that some incoming TCP SYNs are not
responded to at all. Even TCP SYN retransmits from the same source
port are not responded to.
When we piloted the application in our production environment, the
load was fairly low (roughly 200-450 application requests per day) and
we didn't encounter issue. Right as we implemented to all users the
number of application requests jumped to about 12,000 per day, the
problem started the same day.
The percentage of failures increases daily. The first day in
production, about 1% of our application requests had EMAPI/AXL
failures. The second day it was about 3%. The percentage of failures
increased every day (even on weekends when there was less load), until
a week later when roughly 25% of requests were failing.
We restarted the tomcat service but that didn't resolve the issue.
When we rebooted the CallManager server, the error percentage dropped
back to 0 but then it started slowly increasing just like the first
day we put it in production.
The application that I am working on does axl lookups for device
logout profile and user profile. It will then do an EMAPI lookup for
the logged in user and EMAPI login or logout and sometimes both.
We've ruled out network issues, because we've confirmed in a Wireshark
capture that the CallManager server isn't responding to some TCP SYN
requests. Because TCP SYNs aren't being responded to, we think it must
be something on CallManager. Also, because of the fact that the error
percentage increases daily, it seems almost like a memory leak.
Does anyone have any ideas why this would happen or what we could do
to narrow down the problem?
Thanks,
Jim
Subject: RE: AXL/EMAPI Memory Leak?
Replied by: David Staudt on 14-06-2013 02:43:14 PM
With AXL requests of any scale, it is important to re-use the session-IDs provided in the two response cookies
POST https://10.88.131.141:8443/axl/ HTTP/1.1
Accept-Encoding: gzip,deflate
Content-Type: text/xml;charset=UTF-8
Authorization: Basic QWRtaW5pc3RyYXRvcjpjaXNjbyExMjM=
SOAPAction: "CUCM
B ver=8.5"Content-Length: 249
Host: 10.88.131.141:8443
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.1.1 (java 1.5)
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ns="http://www.cisco.com/AXL/API/8.5">
<soapenv:Header/>
<soapenv:Body>
<ns:getCCMVersion>
</ns:getCCMVersion>
</soapenv:Body>
</soapenv:Envelope>
---------------------
HTTP/1.1 200 OK
Set-Cookie: JSESSIONIDSSO=C5374FFF9F8F25DA2A5C9DE1787D94B7; Path=/; Secure
Set-Cookie: JSESSIONID=25CD2D2A71298D55118201EEF3DE0550; Path=/axl; Secure
Content-Type: text/xml;charset=UTF-8
Content-Length: 347
Date: Fri, 14 Jun 2013 19:28:38 GMT
Server:
<?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><ns:getCCMVersionResponse xmlns:ns="http://www.cisco.com/AXL/API/8.5"><return><componentVersion><version>8.5.1.10000(26)</version></componentVersion></return></ns:getCCMVersionResponse></soapenv:Body></soapenv:Envelope>
---------------------
POST https://10.88.131.141:8443/axl/ HTTP/1.1
Accept-Encoding: gzip,deflate
Content-Type: text/xml;charset=UTF-8
Cookie: JSESSIONIDSSO=C5374FFF9F8F25DA2A5C9DE1787D94B7; Path=/; Secure
Cookie: JSESSIONID=25CD2D2A71298D55118201EEF3DE0550; Path=/axl; Secure
SOAPAction: "CUCM
B ver=8.5"Content-Length: 249
Host: 10.88.131.141:8443
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.1.1 (java 1.5)
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ns="http://www.cisco.com/AXL/API/8.5">
<soapenv:Header/>
<soapenv:Body>
<ns:getCCMVersion>
</ns:getCCMVersion>
</soapenv:Body>
</soapenv:Envelope>
---------------------
HTTP/1.1 200 OK
Content-Type: text/xml;charset=UTF-8
Content-Length: 347
Date: Fri, 14 Jun 2013 19:33:23 GMT
Server:
<?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><ns:getCCMVersionResponse xmlns:ns="http://www.cisco.com/AXL/API/8.5"><return><componentVersion><version>8.5.1.10000(26)</version></componentVersion></return></ns:getCCMVersionResponse></soapenv:Body></soapenv:Envelope>
Subject: RE: AXL/EMAPI Memory Leak?
Replied by: JAMES DEPHILLIP II on 16-06-2013 09:53:51 AM
Thanks David does this apply to EMAPI as well? We noticed more issues with EMAPI than AXL...
Subject: RE: AXL/EMAPI Memory Leak?
Replied by: David Staudt on 17-06-2013 11:11:13 AM
All (most?) of the UCM web services APIs run through the on-board Tomcat session manager, so believe E/M would benefit as well.