Multiple Agent Phones ringing

[texasjet79]

ICM 7.5.8

CAD 7.5.8

CVP 7.0.(2) with latest ES ---- With just static routes

CallManager 6-1-2.1000-13

We are seeing multiple phones ring at the same time for the same inbound caller upon Queue selection.

Has anyone else seen this and if so what was the fix?

[geoff]

Never seen this.

It's hard to imagine that ICM/CVP is at fault. Let's walk through what happens when the Router chooses the next agent from the queue to take the call.

The Router chooses an agent by SkillTargetID and asks the Peripheral for the label on the Routing Client (CVP) of the device target of the aforementioned agent. Then it returns this label to CVP. CVP looks in the static routes and sees that the destination is a subscriber, so it sends a SIP INVITE to that subscriber. It responds, and CVP tears down the VRU leg and extends the leg to the Call Manager who uses SCCP to set up the phone path.

In all of this I can't see how more than 1 label could return. I would suggest the problem is in CUCM, but I don't know exactly what is wrong. Are there pickup groups configured?

Regards,

Geoff

[Anthony Holloway]

I have no first hand experience with this, but I have heard of this happening in UCCX 7x.  Two agents will receive the same inbound call, and end up on a conference with the caller.

There is currently an open TAC case, and if a resolution is provided, or defect identified, i will update this post.

[david.macias]

Is it always the same two phones?  Any line apperances, hunt groups, etc?

david

[texasjet79]

It appears to be random and not reproducable.

[shaheeramunir]

We also faced this issue few times but it was always resolved from E1/PRI service provider end so do take them in loop if you are sure that your PGs are stable.

[texasjet79]

What on the E1/pri was causing the issue. How did you resolve this?

[texasjet79]

So it happened first thing yesterday morning on a few calls and then dissapeared the rest of the day. Agents went home for the evening, came back in this morning and a handful of calls had the same and similar symptoms and now they are gone. Maybe I just need to warm up the servers each morning first !!!! LOL

Here is another description:

1.       When the issue happens, customer has noted that on-hold music is playing at the same time they can hear the agent.

2.       Agent gets a fast busy.

3.       Call requeues to another agent and works fine.

The other is multiple agents answer but only one call gets pegged up and works.

[geoff]

On your contact center lines, do you have the busy trigger 1 and no call waiting, no Unity?

Regards,

Geoff

[Anthony Holloway]

texasjet79 wrote:
Here is another description:
1.       When the issue happens, customer has noted that on-hold music is playing at the same time they can hear the agent.
2.       Agent gets a fast busy.
3.       Call requeues to another agent and works fine.
The other is multiple agents answer but only one call gets pegged up and works.

This is word for word, what I have been told, but in UCCX.  What's goin' on?

[khambetea]

If you are using extension mobility, I would check to see if the agent was logged in to multiple phones. I would route plan report in

the call manager.

[matt.harte]

Hi All, does anyone know the TAC case number?

Thanks

[Anthony Holloway]

Yeah, where is the resolution for this issue?  someone post back if they have had a successful fix.

[texasjet79]

So looking at packet captures we noticed some bizarre behavior related to ARP on

the CVP Servers. ARP was periodically refreshing for no reason and it caused partial packet loss. The fix was adding to Registry entries and that resolved it. Apparently this was fixed in 8.x, unfortunately it took TAC two weeks to escalate to development and once they did there was an immediate fix provided.

Hope this helps.

Here is a write up from cisco that resolved OUR issue:

Two registry values are affected:

ArpCacheLife

ArpCacheMinReferencedLife

1. On the CVP server, in regedit, change the value of ArpCacheMinReferencedLife/ArpCacheLife

Location: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters  to the maximum (0xFFFFFFFF)

2. Bring the CVP services out-of-service.

3. Reboot the CVP machine.

4. Bring the CVP services back into service.

If the parameter is not there by default,  add it manually:

* Right click in the window and add the value (ArpCacheMinReferencedLife, ArpChacheLife) as a DWORD value.  Then set it to 0xFFFFFFFF by double clicking on it.

Microsoft acknowledges ARP implementation "problems" in the Win2000 Server and WS2003 TCP/IP stack.

ARP functionality in WS2008 has been completely rewritten and this issue has been resolved in that rewrite.

Once this registry item is set to 0xFFFFFFFF, the problem goes away completely. 

This problem has been in the Windows TCP/IP stack since Windows 2000 Server. 

There is no patch for this from Microsoft  as they will not issue any new patches for WS2003. 

Setting this registry value is the only workaround and doing so should not have an adverse impact on normal operation.

This can also occur on ICM Servers apparently.

The crux of the problem is that when the ARP cache is being refreshed, ALL outbound messages are held in the transmit queue until it is complete. Once the cache is refreshed, normal message flow resumes.  At the TCP level, you'll see duplicate ACKs and retransmissions (in a sniffer capture) likely because of (TCP) timeouts.  At the app level, we see MDS round-trip times go through the roof, queues back up and significant latency with high-priority messaging.  It is particularly problematic on PG private nets because of the volume of messaging between PGA/B.