08-12-2020 10:16 AM
We have a Enghouse QMS call recording server (Version: 7.4.2.20145 ) that uses the Cisco TSP for call control messaging. It quit working after windows updates were applied on 8/10/2020. I reached out to Enghouse support to investigate the issue and they indicated that they are not seeing any TAPI messaging from the Cisco TSP and that this issue is a Cisco problem and should be investigated with Cisco TAC (probably a software issue with the update). we can confirm that the TSP is showing the controlled phones but no status messages are being received. We have rebooted the recording server as well as reinstalled/reconfigured the TSP without success. I also restarted the CTI service on our 2 CUCM Servers with no change. CUCM version 11.5.1.14900-11. TSP version downloaded from CUCM, version11.5(2). Windows server is a currently patched Windows Server 2016.
We're seeing the following in the TSP logs
20:22:04.068 | CSelsiusTSPLink::getMessage() Link read time out. hSocket=3188
20:22:04.068 |<--CSelsiusTSPLink::getMessage()
20:22:04.068 |-->CSelsiusTSPLink::getMessage()
20:22:04.568 | CSelsiusTSPLink::getMessage() Link read time out. hSocket=3188
20:22:04.568 |<--CSelsiusTSPLink::getMessage()
20:22:04.568 |-->CSelsiusTSPLink::getMessage()
20:22:05.069 | CSelsiusTSPLink::getMessage() Link read time out. hSocket=3188
I reached out to Cisco TAC and they basically said pound sand we don't support TSP (even through its used in their own CUAC and CCE products). Apparently a community forum is how we have to try and get resolution for this... absolute joke.
Anyway - if anyone has a suggestion or advice, I'm all for it. I've attached the TSP log and the 2 windows updates that were installed. My next call is to our account manager.
08-12-2020 01:53 PM
Have not had any similar reports so far as I know, that certainly is troubling...
From your description it sounds like perhaps incoming messages from CUCM to the application are not being received or are blocked.
* Is it possible that the Windows firewall has somehow been enabled/re-enabled or the allowed connection ports cleared? For default/insecure connection port is 2748, if the TSP is configured for secure connection port is 2749.
* To recover function temporarily, would it be possible try removing one or both of the Windows update KBs?
* For further troubleshooting purposes it may be helpful to have:
1) All TSP logs from a system restart through reproduction of the issue (set the TSP config trace level to all-types/detailed)
2) A network packet capture (e.g. using Wireshark) from the Windows 2016 server for application startup/repro
3) CTI Manager logs, see: https://developer.cisco.com/site/tapi/help/case-opening-steps/
* The CUCM version is over 2.5 years old. There have been a number of changes to default/supported TLS version/cipher details for CUCM (and Windows - for example I think I recall a situation where a recent update for Win7 was forcing some TLS connection minimums which caused problems.) during that time. If your TSP is configured for secure connection it's possible there may be some conflict/issue. Not that you would necessarily need to upgrade to resolve this particular issue (at least until we know final root cause and any options for workarounds), but it may be a good time to start the wheels for considering an upgrade at some point.
Regarding the support experience, that's very disappointing :/ Indeed, a forum is not the intended support channel for production verified partner apps:
* Cisco IVT verified partner applications should receive 'coordinated/cooperative' support, where Cisco product TAC and the vendor support organization work together with Cisco DevNet to analyze and resolve issues involving integration between products. TAC/vendor/DevNet keep all 3 cases open and linked until resolution.
* I know Arc (now Enghouse per my understanding) has long been a verified partner (indeed OEM in the case of CUAC) for their operator console products, however I don't see that the QMS product has been through the official testing - at least it doesn't appear on DevNet Ecosystem Exchange: https://developer.cisco.com/ecosystem/solutions/#key=call%20recording . It would be good if it did, perhaps we can prompt them to consider testing if not already planned...
* Verification status is pretty secondary here, as this is an 'integration' issue - in such cases, part of the partner agreement is the understanding that the vendor is the primary owner of the issue - I don't believe they should have passed you to TAC (who are not trained for, nor will accept developer support issues.) Ideally, Enghouse would analyze the problem (which they did - i.e. not seeing inbound messages) and open a case with Cisco DevNet Developer Support, who can work with them directly to understand/troubleshoot the issue.
* In this particular case, I suspect Enghouse may not necessarily need to update their code or 'fix' anything on their side - so most of the actual investigation may involve DevNet trading trace requests and thing to try with your team - however Enghouse would benefit from understanding the issue/resolution in case future customers encounter it and/or they need to add info to their UI/help/docs/FAQs, etc. (e.g. maybe a specific Windows 2016 config/caveat)
- Generally an end-user customer would not open a DevNet case for an app they aren't developing themselves - the likelihood DevNet can resolve the issue without analysis and cooperation with the developer of the app is low.
In general, I would suggest putting together the traces/info mentioned above and sending with full description via email to developer-support@cisco.com; as well as requesting Enghouse open a DevNet dev support ticket for the investigation. In the mean-time I will give a heads-up to the DevNet support engineers about your issue in anticipation of your email, in the hopes we can get some investigation started with you directly with the Enghouse ticket coming in parallel.
08-12-2020 02:11 PM
Is it possible that the Windows firewall has somehow been enabled/re-enabled or the allowed connection ports cleared? For default/insecure connection port is 2748, if the TSP is configured for secure connection port is 2749. Windows Firewall was not turned on (no change, this is how this has been configured
To recover function temporarily, would it be possible try removing one or both of the Windows update KBs? We didn't try this, since they updates would have just re-installed themselves.
* For further troubleshooting purposes it may be helpful to have: 1) All TSP logs from a system restart through reproduction of the issue (set the TSP config trace level to all-types/detailed) 2) A network packet capture (e.g. using Wireshark) from the Windows 2016 server for application startup/repro 3) CTI Manager logs, see: https://developer.cisco.com/site/tapi/help/case-opening-steps/ I attached the TSP log (set to detailed). I didn't grab the CTI manager logs, although that didn't appear to be the issue since nothing changed on CUCM (and our other TSP driven apps were working fine). Wireshark captures were taken and we confirmed the SIP trunk was sending control and RTP traffic to the recording server.
* The CUCM version is over 2.5 years old. There have been a number of changes to default/supported TLS version/cipher details for CUCM (and Windows - for example I think I recall a situation where a recent update for Win7 was forcing some TLS connection minimums which caused problems.) during that time. If your TSP is configured for secure connection it's possible there may be some conflict/issue. Not that you would necessarily need to upgrade to resolve this particular issue (at least until we know final root cause and any options for workarounds), but it may be a good time to start the wheels for considering an upgrade at some point. We aren't running secure TSP (I've worked for a Cisco VAR for 10years and never seen secure TSP used in any production environments, its rare enough to even see secure RTP). CUCM 11.5 is the last non Smart License required version, I suspect many will stay on 11 as long as possible as will we. We aren't using Jabber and Cisco has made literally zero improvements or enhancements since about v10 for call control (unless you count changing the licensing model), or removing phone support.
Regarding the support experience, that's very disappointing :/ Indeed, a forum is not the intended support channel for production verified partner apps: * Cisco IVT verified partner applications should receive 'coordinated/cooperative' support, where Cisco product TAC and the vendor support organization work together with Cisco DevNet to analyze and resolve issues involving integration between products. TAC/vendor/DevNet keep all 3 cases open and linked until resolution. * I know Arc (now Enghouse per my understanding) has long been a verified partner (indeed OEM in the case of CUAC) for their operator console products, however I don't see that the QMS product has been through the official testing - at least it doesn't appear on DevNet Ecosystem Exchange: https://developer.cisco.com/ecosystem/solutions/#key=call%20recording . It would be good if it did, perhaps we can prompt them to consider testing if not already planned... * Verification status is pretty secondary here, as this is an 'integration' issue - in such cases, part of the partner agreement is the understanding that the vendor is the primary owner of the issue - I don't believe they should have passed you to TAC (who are not trained for, nor will accept developer support issues.) Ideally, Enghouse would analyze the problem (which they did - i.e. not seeing inbound messages) and open a case with Cisco DevNet Developer Support, who can work with them directly to understand/troubleshoot the issue. * In this particular case, I suspect Enghouse may not necessarily need to update their code or 'fix' anything on their side - so most of the actual investigation may involve DevNet trading trace requests and thing to try with your team - however Enghouse would benefit from understanding the issue/resolution in case future customers encounter it and/or they need to add info to their UI/help/docs/FAQs, etc. (e.g. maybe a specific Windows 2016 config/caveat) - Generally an end-user customer would not open a DevNet case for an app they aren't developing themselves - the likelihood DevNet can resolve the issue without analysis and cooperation with the developer of the app is low. In general, I would suggest putting together the traces/info mentioned above and sending with full description via email to developer-support@cisco.com; as well as requesting Enghouse open a DevNet dev support ticket for the investigation. In the mean-time I will give a heads-up to the DevNet support engineers about your issue in anticipation of your email, in the hopes we can get some investigation started with you directly with the Enghouse ticket coming in parallel.
No argument there, its a sad state of affairs with Cisco support these days, Your explanation of the DevNet support makes perfect sense, and TAC should not have recommended we pursue that method directly. I ultimately escalated to the engineers manager and they reluctantly agreed to review the TSP portion and could find nothing wrong (albeit it didn't work). They suggest re-moving and re-installing the TSP (again - I had already done that once, with reboots after removal and reinstall). I agreed, we reconfigured it the same as it was (and how it had been configured when TAC started looking at it) and the TSP and call recording software started working. SO.. just re-install it twice next time it breaks. Its back working but no idea why it broke, or if it'll happen again. I have so little faith in Cisco these days.
02-13-2023 09:31 PM
Did you manage to resolve it?
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide