Getting huge packet loss for multi site call.

[veeranagowda Berigi]

Hi Can any one please help me over here, getting huge packet loss for the multi site call, Find the below event logs from the one of the endpoint and let me know how to fix this.

DEVICES ARE INVOLVED HERE ALL ARE CISCO SX20, aLL ARE REGISTERED TO cucm WITH SIP URI.

Endpoint 1 ME1 Details
IP: 172.25.XX.XX
Sip URi: 83650@apcucm.com
software version: TC7.3.7.01c84fd has multisite option


Endpoint 2 HK0 APHQ VICTORIA HARBOUR
IP: 
Sip URi: 87717@apcucm.com
Softeware version: TC7.3.7.01c84fd has multisite option


Endpoint 3 SY1 HARBOUR ROOM
IP: 172.16.XX.XX
Sip URi: 82049@apcucm.com
Softeware version: TC7.3.7.01c84fd do not have Multisite option


2017-03-09T11:08:03.161+08:00 a8 appl[1663]: 629820.65 MainEvents I: ActiveSpeakerIndication(c=21,p=28)
2017-03-09T14:08:06.000+08:00 a8 vcodec: DEC_FSM-0: Still experiencing decode errors (32)
2017-03-09T11:08:16.730+08:00 a8 appl[1663]: 629834.22 CuilApp   User admin(1001) about to execute command '/Audio/Sound/Stop' from .
2017-03-09T14:08:25.000+08:00 a8 vcodec: WARNING: Encoder overproduction (ttvenc_h264_ivahd):
2017-03-09T14:08:25.000+08:00 a8 vcodec:     configured rate 320000, actual rate 486561
2017-03-09T14:08:25.000+08:00 a8 vcodec:     4 frame periods of extra delay would be seen if data paced at configured rate
2017-03-09T14:09:01.000+08:00 a8 vcodec: WARNING: Encoder overproduction (ttvenc_h264_ivahd):
2017-03-09T14:09:01.000+08:00 a8 vcodec:     configured rate 320000, actual rate 539536
2017-03-09T14:09:01.000+08:00 a8 vcodec:     5 frame periods of extra delay would be seen if data paced at configured rate
2017-03-09T11:09:55.908+08:00 a8 appl[1663]: 629933.40 MainEvents I: InputAudioChannelChanged(p=29,gid=514) channels=2 encryption=Off muted=False protocol=Off
2017-03-09T11:09:56.396+08:00 a8 appl[1663]: 629933.88 MainEvents I: InputAudioChannelChanged(p=29,gid=514) channels=1 encryption=Off muted=False protocol=AACLD
2017-03-09T11:10:17.979+08:00 a8 appl[1663]: 629955.47 MainEvents I: ActiveSpeakerReported(c=21,p=29)
2017-03-09T11:10:17.981+08:00 a8 appl[1663]: 629955.47 MainEvents I: ActiveSpeakerIndication(c=21,p=29)
2017-03-09T11:11:13.841+08:00 a8 appl[1663]: 630011.33 MainEvents I: ActiveSpeakerReported(c=21,p=28)
2017-03-09T11:11:13.843+08:00 a8 appl[1663]: 630011.33 MainEvents I: ActiveSpeakerIndication(c=21,p=28)
2017-03-09T14:11:35.000+08:00 a8 vcodec: DEC_FSM-0: No decode errors in 209 s, turning output on again
2017-03-09T14:11:35.000+08:00 a8 vcodec: DEC_FSM-0: Decode Err: 0x80000000|Prev. frame lost|
2017-03-09T14:11:35.000+08:00 a8 vcodec: DEC_FSM-0: Decode Err: 0x80000000|Prev. frame lost|
2017-03-09T14:11:35.000+08:00 a8 vcodec: DEC_FSM-1: No decode errors in 174 s, turning output on again
2017-03-09T14:11:35.000+08:00 a8 vcodec: DEC_FSM-1: Decode Err: 0x02000a00|Concealment Applied|Missing slice|
2017-03-09T14:11:42.000+08:00 a8 vcodec: DEC_FSM-0: Decode Err: 0x02000a00|Concealment Applied|Missing slice|
2017-03-09T14:11:42.000+08:00 a8 vcodec: DEC_FSM-0: Decode Err: 0x02000a00|Concealment Applied|Missing slice|
2017-03-09T14:11:42.000+08:00 a8 vcodec: DEC_FSM-0: Decode Err: 0x02000a00|Concealment Applied|Missing slice|
2017-03-09T14:11:42.000+08:00 a8 vcodec: DEC_FSM-0: Decode Err: 0x82000a00|Concealment Applied|Missing slice|Prev. frame lost|
2017-03-09T14:11:42.000+08:00 a8 vcodec: DEC_FSM-0: Decode Err: 0x02000a00|Concealment Applied|Missing slice|
2017-03-09T14:11:42.000+08:00 a8 vcodec: DEC_FSM-0: Decode Err: 0x02000a00|Concealment Applied|Missing slice|
2017-03-09T14:11:42.000+08:00 a8 vcodec: DEC_FSM-0: Decode Err: 0x02000a00|Concealment Applied|Missing slice|
2017-03-09T14:11:42.000+08:00 a8 vcodec: DEC_FSM-0: Decode Err: 0x02000a00|Concealment Applied|Missing slice|
2017-03-09T14:11:42.000+08:00 a8 vcodec: DEC_FSM-0: Continually experiencing decode errors, suppressing further output

Appreciate for your help.

[Patrick McCarthy]

Packet loss is generally caused by one of two problems: either you don't have enough bandwidth to handle the call(s) at the rate chosen and/or you have a duplex mismatch. For example, if endpoint 1 is at a site with a T1 and you try to host a call to the other two at 1.2 MB you will have significant packet loss because you'd be trying to make two calls when you only have enough bandwidth to support 1 (add 20% to the call rate for overhead). Try calling at lower rates - do 384K or 512K and see what happens. The resolution of course will go down, but check the specs - does the packet loss drop significantly or go away completely?

The other common problem is a duplex mismatch. If the SX is plugged into a switch that's set to 100/full and the codec is set to auto, the codec will negotiate to 100/half and you will get packet loss. The reverse is also true. Make sure both are set the same - if one side is hard set, set the other to match, or leave both on auto. Note this issue could be anywhere in the network, most often I've seen it on the codec itself, but if there is a router/switch connection with a duplex mismatch and the video traffic is using that link, it will cause problems there too.

The one thing with this problem is you'll see it regardless of call rate - even at 384K you might see 1-2% loss, perhaps more at higher rates. If the problem is purely bandwidth the loss should pretty much go away at lower rates. 

One other thing - probably goes without saying - if you have QOS on the network make sure you have these set appropriately. 

[Justin Ferello]

Patrick,

I have also seen those overproduction errors and decode errors in a high packet loss environment.  My question would be are those encode errors the cause of the packet loss or the result of the packet loss?

Thanks,
Justin

[Patrick McCarthy]

Those errors are the result of the packet loss, not the cause. From my experience packet loss is always a network related problem - I've never once seen an exception to that. 

[dneggers1]

Hi There,

I have experienced the same thing with our SX20 > Telepresence Server.

We Have over 60+ Posts around the globe in countries where large bandwidth is not possible.

What we have found is that even though we specficy the max call rate in CUCM regions for calls, the actual video stream keeps bursting way over that.

E.g

1. We set our max call rate to 512K (CUCM Region)

2. The call establishes, then within a few minutes, the actual call rate (we see on Wireshark) is pushing up over 600K

3. This violates our WAN policers which are set to a specific BW for AF41 , and traffic is getting dropped.

4. Cisco TAC were unable to solve the issue, after many weeks, they have mentioned that the extra overhead is due to "Flux" messages, which cannot be prevented.

5. Their solution was to go and buy an extra 100-200K of AF41 Bandwidth from the WAN Carrier.

6. The same call with SX20 to the older HW MCU does not experience this issue, so its to do with the way TPS & SX20 are adapting the call rate

Still no fix for us, we have had to drop the call rate to 448K so it does not go over the allocated BW.

Cheers

[Ray Boland]

Hi

512k will be the call rate (448k video+64k audio) - you also need to factor in the call control signalling, thus it's generally recommended call rate +25%

So for 512k call, on the WAN you should be reserving 640k (512k+25%).

[dneggers1]

Hi Ray,

I understand that, but call control signalling is marked in a different class (AF31) vs (AF41)

We are seeing the AF41 (RTP) pushing up over 650K.

Cheers

Dion

[Ray Boland]

That's interesting alright - was it just one SX20 or all of them and did the same this happen when it was SX20 to SX20?

[dneggers1]

Hi Ray,

the whole fleet.. (110 units)

P2P calls are fine, its only when dialing into Multiparty conference on TPS.

When using the older HW MCU, the problem does not occur..

Cheers!

Dion

[Patrick Sparkman]

What software version is the TelePresence Server running, and does the packet loss appear when content is or isn't being presented?

[dneggers1]

Hi Patrick,

Running 4.4 (1.9)

This is with or without content channel.. I have tried both.

I have tried TC and CE code on the SX20 side.

I have also tried C20.. 

Cheers

Dion

[Ray Boland]

There are settings on the MCU for lost packet recovery that can affect packet loss

There have been issues with the old codian MCU, if Flow Control was enabled it caused a huge spike in traffic. i tihnk the endpoint will re-transmit packets that are lost, which might be whats causing it?

Thats assuming the old MCU and the TP MCU are both the same in terms of the call flow (e.g. SIP registrered to the CUCM)

Interesting issue though!

[Patrick Sparkman]

There is a bug with the TelePresence Server, CSCuz11834, that will cause the bandwidth to spike above the call rate that the endpoint is set to use, but the software version you're running contains the fix for that particular bug. Have you tried to upgrade to the most recent TelePresence Server release, which is 4.4(1.16). 

[dneggers1]

Hi Pat,
That is interesting, it sounds like the same symptoms that we were having. At the time there was no BUG identified.

I think I will try the updated version in our TEST lab and see the results..

Thanks for the info

Dion

[veeranagowda Berigi]

Hi Patrick,

Am not observing any packet loss for the point to point call, But multi site call packet loss is crossing 6%.