WFO QM Backup Server

[TrickTrick]

Hello everyone,

How can I set up a recording backup server properly ? the documentation provided by Calabrio is not that clear about that particular point, Tried to install a second server and ended up by destroying the config in the primary server and did everything from scratch

my goal is to have a recording backup server that can handle recorded calls and archive them when the primary is down

Any input is appreciated

Thanks

[Graham Old]

There is some discussion about a cold standby server but to make that work you need recordings on a SAN and a separate SQL server. I have never tried to set one up.

I normally create a backup recording server. If the primary base server goes down then the backup recorder will continue to record calls but they will just sit on its disk drive until the primary comes back up. Calls will then be moved to long term storage and you can play them back via the web interface.

Install the Primary as a normal single server solution. On the backup server just install the CTI service and the network recording service, assuming you are using BiB for recording.

Add the backup CTI server to the "Signalling Groups" in "Telephony Groups"

Add the backup recording service under recording clusters.

Create a second SIP trunk to the backup CTI service, you should enable ping on the SIP.

The two CTI servers will run in an Active/Standby mode and only the active one will respond to the SIP pings.

You don't have control over which CTI service is the active one and if they do changeover they will not automatically change back.

Graham

[TrickTrick]

" Add the backup CTI server to the "Signalling Groups" in "Telephony Groups" "

Like this https://imgur.com/a/rHJ1M ? This is btw the primary recording server

"Add the backup recording service under recording clusters."
The ip address of the secondary is already shown and added in the cluster

btw in the recording route pattern I have 1 sip trunk which is the primary, I should create a route group I guess

"You don't have control over which CTI service is the active one and if they do changeover they will not automatically change back."
You mean, the backup could be Active and the primary standby ? and I have to change that manually ?

[TrickTrick]

Added a second SIP trunk, created a Route group and assigned both site trunks to it,

added the back recording server for backup CTI services ( under that both CUCM IP's, the Screenshot in my previous comment shows that)

tried to do an HA test, but didn't work, checked RTMT errors, it gives the following warning:

 Number of RouteListExhausted events exceeds configured threshold during configured interval 0 within 60 minutes  on cluster StandAloneCluster.  There are 1 RouteListExhausted events (up to 30) received during the monitoring interval From Wed Apr 11 11:29:18 WEST 2018 to Wed Apr 11 12:29:18 WEST 2018:   RouteListName : Recording_RL; Reason=41; RouteGroups(Recording_RG) CallingPartyNumber : 9YYY(this is the Agent DN) CallingDeviceName : SEP501CB076171C CalledPartyNumber : 9XXX(this is the recording DN) CalledPartition : 536d60d8-1226-860b-e0f7-d1b00e1d488c CalledPattern : 9XXX AppID : Cisco CallManager ClusterID : StandAloneCluster NodeID : CUCM-PUB  TimeStamp : Wed Apr 11 11:49:47 WEST 2018  

Also, in both recording servers, the backup signaling server state is : Connection refused

Any idea what could be the issue ?

[Graham Old]

Your two CTI server should look like this or may be the other way around

If it is saying connection refused then there is a configuration problem with the backup server.

Did it install JTAPI OK.

Have a look in the log files \Program Files\Cisco\WFO_QM\log at the ctiservice.dbg file

At the top should be something like this

TIME ZONE: Europe/London (current_offset = GMT+1:00)
SERVER: 172.17.101.61:52102
FAILOVER_MANAGER: com.calabrio.signaling.FailoverManagerPrimary (state=STANDBY)
SIP PORT: 5060
SIP SECURE PORT: 5061
CUCM Connection: CUCM-SUB01.xxxxx.local,CUCM-SUB02.xxxxxx.local;login=QMJTAPI
CUCM Version: 11.5.1.13902-2.i386
JTAPI Version: 11.5(1.13045)-1

Your earlier question about route groups.

I usually create a SIP profile for Calabrio, enable the SIP Ping Option and set the Ping Interval for both in service and out of service to 30 seconds to have a quicker fail over.

Create two SIP trunks one to each Calabrio server using your SIP profile with Ping and add them into a Route Group. If both SIP trunks are down then yes you will get Route Group exhausted messages.

I think if you can get your backup Calabrio CTI service running then the rest should work

Graham

[TrickTrick]

You're right about JTAPI services, I've forgot to install that in the secondary server, so now it's working no more "connection refused" state

Also, as you've mentioned above, once the primary comes up after a failure, it doesn't take the "ACTIVE" role automatically, how can you change that ? I have to do it all the time manually ?

I've created now a SIP profile with the ping feature enabled, only the secondary sip trunk is "in service", the primary has "no service" state somehow, I had to shutdown the backup server so the SIP trunk comes up as "full service" state again

checked the logs btw, this line triggered me : [ObserverThread(com.calabrio.qm.ctiservice.jtapi.CiscoDeviceProvider@f2be9ab)|AbstractFailoverManager#doInService:90] Telephony not yet obtained: recordingManagerInit=false; providerInService=true

the last thing, which is the most important, when even when the secondary server is the "ACTIVE" one, it doesn't send any records to the primary, tried to check the recording files,I find nothing in the recording folder. When the primary is UP, records are saved in the Primary server even when the back server is taking the "ACTIVE" role

Here's a TL;DR version :D : the state doesn't matter when both are up, the primary will save the files anyway,no problem obviously if the primary is up, it's not working when the secondary is up, primary UP, no files recorded at all

[Graham Old]

When you installed the backup Calabrio it should have rebooted and then launched post install. Did you set the IP addresses correctly on that initial screen. It's a little strange it did not install JTAPI for you.

In Telephone Groups under Recording Clusters are your recording servers.

Under "Primary Record Servers" should be your primary Calabrio server and under "Secondary Recording Servers" should be your backup Calabrio server.

The primary recording server will always be used if it's up and available.

If the primary recording server is down and the backup recording server is being used. When the primary comes backup existing recordings in progress will continue to use the backup but new recordings will then use the primary. This is different to the CTI service.

If the CTI servers change roles then any recordings in progress would be lost. This is why Calabrio does not automatically change roles on the CTI service. If you want to change roles then you need to restart the service on the Active CTI server to force it to change roles out of hours.

Normally the two Calabrio servers are on different sites and I set the CTI service to use its local UCM subscriber first. I don't worry which CTI service is active as the network traffic between the CTI service and the UCM subscriber is very low compared to the amount of audio traffic.

If you think you are having problems with the backup recording service have a look in the VoiceRecordingServerXXXX.dbg file, look in the one with the latest date/time and see if you can see any errors

Graham

[TrickTrick]

I solved the issue by modifying the Site parameters, in the primary I should specify the team and the recording cluster, while in the secondary I should only specify its IP address, once I add the team or the cluster, all is messed up afterwards, so now it works that way

[CubeNetworksNOC]

Hi Trick,

I have installed QM 11.5 and while configuring it via Post install, I am also getting CTI error 'connection refused' message. 

Actually same issue which I think you have posted in other forum as well; https://imgur.com/a/rdzCC

https://www.reddit.com/r/sysadmin/comments/89yct3/has_anybody_here_succeeded_to_setup_cisco_wfo_qm/

I would appreciate, how did you get by with that error

QM ver 11.5

CUCM ver 10.5

 Regards

Taha

Regards

M Taha

[TrickTrick]

Hi,
the solution for me was putting the ip address of the recording server there, and not cucm,i thought that I should use same CTI ports used by CCX in CUCM, while for the recording server it's not the case

share if you have any updates

[ahmed-hassan1110]

Hi Trick,
could you please let me know if you have installed only CTI and recording services on the backup QM recording server or you have installed all services just like the primary recording server ?

[ahmed-hassan1110]

Hi Graham,

I have a setup of CUCM 11.5 cluster , CCX 11.6 cluster, AQM 11.5 single server for call recording, all is working fine.

Now we are implementing a backup QM recording server and the Cisco documents lack to enough details.

I’m using Automatic Silent IP Phones Based Recording (BiB), recording route pattern, recording profile, and SIP trunk pointing to current QM recording server on the CUCM.

So I do appreciate if you can give more details about the implementation of this redundant QM recording server.

as you mentioned we should only install the CTI and recording services on the backup recording QM, then after that all configurations are done on the primary QM? i mean adding CTI server to signaling group and all similar configurations? so shouldn't we access the GU on this backup QM? just install the mentioned services and all remaining configs are made on CUCM and primary QM?

Thanks

[ahmed-hassan1110]

Hi Graham,
I have a setup of CUCM 11.5 cluster , CCX 11.6 cluster, AQM 11.5 single server for call recording, all is working fine.
Now we are implementing a backup QM recording server and the Cisco documents lack to enough details.
I’m using Automatic Silent IP Phones Based Recording (BiB), recording route pattern, recording profile, and SIP trunk pointing to current QM recording server on the CUCM.

So I do appreciate if you can give more details about the implementation of this redundant QM recording server.
as you mentioned we should only install the CTI and recording services on the backup recording QM, then after that all configurations are done on the primary QM? i mean adding CTI server to signaling group and all similar configurations? so shouldn't we access the GU on this backup QM? just install the mentioned services and all remaining configs are made on CUCM and primary QM?
Thanks

[TrickTrick]

Hello Hassan,
if I remember so well ( because it's been a while I didn't install a similar configuration), you have to install QM in the secondary server, and add that server in the primary server ( as a secondary recording server), after that you have to create another sip trunk for the secondary server in CUCM and create a route list that includs a route group which will have both recording Sip trunks, that way, if your primary server goes down, the secondary will continue recording until the it goes back online and send all the recording to it,
Hope my answer makes things clearer to you. If you still have any more questions hit me with them

[ahmed-hassan1110]

Hi Trick,
Thanks for replying and all details, my question was mainly about the QM components you have installed on the secondary server, did you install all QM components like the primary server or only some of them?
here is the QM components out there:
Web Base Services
Operational Base Services
Signaling Services
Recording Services
Site Upload Services
Database Services
Encoding Services
Reconciliation Services
Monitoring Services