CMS - What causes the bad presentation quality? (Example attached)

[DaveLawang]

Hi community

Edit, August 2023

Since CMS 3.7.1 the presentation quality is acceptable and they've added more options to adjust it in CMS 3.7.2. 

However the suspected, yet unconfirmed, cause (see here) for the once very bad presentation quality persists in CMS 3.7.2.

Observed behavior (MS Edge, Chrome):

• Video streams of participants usually are crisp and clean (except their connection is bad)
• Presentation often suddenly starts looking as if it got shredded in the lower third of the image

Example of the problem:

Any technical suggestions what we could be looking at here? It always draws that kind of pattern where the lower part of the image seems to be turned into those huge square pixels.

• Packet loss?
• Codec doing something funky?
• Delayed I-Frames, P-Frames, B-Frames?

I kind of suspect QoS issues/misconfiguration but have a somewhat hard time to proof it to the network team. When joining the meeting through a smartphone (cellular network) the presentation looks crisp again.

Another thing that would be helpful for educational purposes @people that have CMS running and can do presentations smoothly:

Would you be willing to open edge://webrtc-internals/ during an ongoing meeting and take a screenshot of the various graphs (incoming RTP stream of the presentation)? I'd be curious how some "good" reference stats of a smooth presentation should look like. 

Below is what it looks like in our setup when I share a simple Excel sheet in a presentation and scroll around in it a bit. On the receiving end the grid of the cells is gone most of the time and it just kind of looks like a pixelated, laggy, blurry soup. When I stop scrolling it takes 2-3 seconds until quality is adjusted. Interestingly with MS Teams presentations work flawlessly (I mention it because I think both are based on WebRTC technology when using the browser).

Anyone else ever experienced similar issues? And got it fixed? Any feedback / suggestions are welcome. 

Best regards
Dave

[DaveLawang]

Just wanted to bump this with an update.

• We've upgraded to CMS Version 3.7 (Currently latest version)
• Problem of bad presentation quality persists (Check screenshot below, all presentations look more or less like that)
• In CMS 3.7 there are some neat new stats -> We noticed that the presentation stream always has a much higher jitter rate than the video streams for no obvious reason (in the screenshot it's "only" 218ms, it occasionally spikes up to 1500ms while the video streams always stay below 10ms)
• CPU of client machines spikes when we share screen (Intel® Core™ i7-10700 -> screen sharing causes + 25-30% CPU load, on a bit older machines it's quite a bit more load)
• Issue seems to be browser related (chromium to be precise)
• Meeting Server 1000 platform looks perfectly healthy when the problems occur (CPU <5% load, RAM, network load, etc. all look fine)
• Problem persisted even when clients were hooked up to the same switch as CMS (video now with <1ms jitter, presentation jitter still all over the place)
• jitter on iperf streams between client / server is perfectly fine. In wireshark we see a straight line with max. throughput (same port ranges, same dscp tagging) (sidenote: make sure your iperf versions on client/server match otherwise you'll run into issues)
• We checked if we're affected by faulty RAM (we're not. Check FN 72368)
• We checked the quality settings but most of them don't apply to the webbridge anyway
• Problem doesn't seem to occur on mobile devices (iOS Safari, Webkit rendering engine)
• Changing congestion algorithms on the ESXi host ("New Reno" vs. "CUBIC") has no impact on presentation quality in CMS 

CMS Presentation quality still bad

By now we suspect that the clients themselves / the browser might be the bottleneck. Since Zoom, MS Teams, Jitsi are all working fine we're kind of lost and out of ideas. Idea donations about what we could investigate or how we could tackle the bad presentation quality would still be welcome.

Cheers,
Dave

[mohamed.elraie771]

We also upgraded to the latest version of 3.7 and the issue existing , what will be the next step?

[LeeC]

We have the same problem. There are moments that it is fine and after a while the ping increases and everything blurs. We tried everything.

[matthewporterctr]

We are having the same problem.   I agree that it appears that the client (browser) is the bottleneck, but not entirely sure why.   

Anyone have any ideas?

[LeeC]

in version 3.7.1 it's much better. Unfortunately, after locking the presentation to 720p, the presentation mode is blurry. Try this update!

[matthewporterctr]

Are you referring to the callLegProfile settings for qualityPresentation in the API?    What about if you set the presentation to "unrestricted" or "max1080p30"?

[LeeC]

In version 3.7.1 cisco locked presentation resoluton to 720p.

and when u look to api guide. This is note to qualityPresentation  

Note: This parameter does not apply to
incoming video from Cisco Meeting App or
Cisco Meeting App WebRTC app.

You can set what u want and presentation quality still stay 720p.

[matthewporterctr]

So there is no way to change that presentation setting that Cisco has locked?    I wonder if there is a command in the API that you can change that setting.   Has anyone else tried this?

[LeeC]

I think it works but only in SIP connection

[matthewporterctr]

We haven't seen too many issues from SIP endpoints.  Where we see the problem is from participants coming from the browser, just as the OP mentioned above.   

Are there ways to test/monitor the WebRTC connections to see if that is indeed where the bottleneck is coming from?

[DaveLawang]

Edit
tl;dr: Could the lack of AV1 codec support in CMS be the reason for the bad presentation quality during screen sharing?

Edit 2:
Did a quick test with Microsoft Teams. Shared screen and checked what codec is used. In my setup it was h.264, no macroblocking observed, despite Teams servers not even running here on-prem. Repeat same setup with CMS and there's macroblocking and incredible rubberbanding for the audience everywhere (presentation is lagging behind more and more over time and at some point they get confused about what I'm talking about as their slides are lagging 5sec or so behind). I'm lost. No clue where things might go wrong with screen sharing in CMS and out of ideas how to contribute to a solution.

----------------------
Unfortunately can't provide a solution but at least a better problem definition.

Came across this FAQ where screen resoultion of sender and receiver is mentioned as factor in presentation quality. There are some other interesting things mentioned as well that lead down a deep rabbit hole.

6. Minimize the difference between the presentation sender desktop resolution and the receivers’ desktop resolution

The difference between the presentation sender’s resolution and receiver’s resolution is very important. If the sender is sending its desktop at 4K resolution (8M pixels) but the receiver is only receiving at 720p (0.9M pixels), the presentation video will be scaled down to 1/8 of its original size and this is likely to cause significant softening of the picture, any text will likely to be completely unreadable. On the other hand, if the sender is presentation at 1080p and the receiver is receiving in 1080p full-screen, then there will be no degradation due to downscaling.

By default, the receiving WebRTC app will show both main video and presentation in a stacked layout. To improve sharpness of the content in the presentation video, it is recommend switching to the “presentation only” layout on your app, as it will make the best use of your browser tab’s viewing size.

7. Avoid transition effects, animations and fast movements in presentation content

Transition effects and animations on desktop (such as Windows Aero effects) or in applications (such as PowerPoint) should be minimized or avoided to improve the overall experience on the receiver. This is due the fact that fast movements are not optimized for the sharpness-focused presentation video. In extreme cases sharing a video stream within the presentation video will have significant framerate drops and macroblocking (1). This is due to how the H.264 video codec is used in real-time communication scenarios.

Frequently Asked Questions - Best practices for Cisco Meeting App for WebRTC on Google Chrome (on Windows)

Within the cisco community you can only find 3 postings that mention macroblocks. Even on google or youtube you don't find a whole lot about the topic which has its roots somewhere deep in the realm of video codecs. Looking at the screenshot of our problem it sure does look an awful lot like we're experiencing some sort of macroblocking issues with CMS webbridge while sharing the screen.

Structures defined by the H.264/AVC standard.

Source: Structures defined by the H.264/AVC standard. A frame consists of... | Download Scientific Diagram (researchgate.net)

(1) Macroblocking

Video content that contains large IFrames can cause macroblocking during playback on SV-4K and DMP-2K media players. Macroblocking is a video artifact where areas of a video image appear as small blocks or squares.

Release 4.1 and Later Releases: Cisco StadiumVision Content Creation Design and Specifications Guide

Macroblocking demo video by SSIMWAVE (R) can be found on youtube.

Macroblocking example, (c) ssimwave.com

In this community posting here @Mohammed al Baqari is taking a deep dive into Video Signaling with SDP Debugs. Not sure if really relevant for debugging this particular issue (in a very quick trace I did in wireshark I couldn't immediately find an sdp when sharing screen but I might have overlooked it) but I mention it because macroblocks are relevant for some of the attributes in the SDP. Might be helpful information to drill down deeper.

max-mbps=245000	Max Decoding speed = Max Macroblocks/sec = 245000 (Baseline profile level 2.2 value = 20250)
max-fs=9000	Max Frame Size = 9000 Macroblocks (Baseline profile level 2.2 value = 1620)
max-smbps=245000	Max Static Macroblock processing rate – macroblocks/second
profile-level-id=428016	The Profile-Level-ID describes the minimum set of features/capabilities that are supported by this endpoint Profile-Level-ID consists of 6 hex-digits. The first 4 hex-digits define the profile-id while the other 2 hex-digits define the level.  In our case the profile-id is 4280 while the level is 16. Profile-ID describes the subset of coding tools that the codec supports by the endpoint. The profile ID 4280 represents baseline profile (BP,66) which supports encoding features such as  Flexible Macroblock Ordering, Arbitrary Slice Ordering, Redundant Slices.

What causes macroblocking according to random sources on the internet

• Macroblocking may occur due to heavy video compression, especially in video frames of fast motion or abrupt scene changes, but the most annoying types of macroblocking in practical visual communication systems are often caused by transmission errors such as packet loss.
• Really, we only see macroblocking in very, very overcompressed video of the sort we all consumed a lot of in the 2000s. You’ll see it in your cell phone videos and camera videos from that time but it’s fairly rare to see it in professionally produced pieces now. 
• Macro blocking is a reconstruction error. Usually caused by the decoder running out of data. This can be caused by a number of conditions including the data rate exceeding the available bandwidth.
• The causes of macroblocking are related to one or more of the following factors: video compression, data transfer speed, signal interruption, and video processing performance.

My sneaking suspicion is that we could be dealing with some sort of buffer-underrun. Party A can't encode fast enough, add some jitter on top of that and every now and then party B's buffer to reconstruct the shared screen just runs empty and blocks are displayed. Just a hypothesis though.

It's possible to enable/disable various codecs by launching Edge or Chrome with command line switches. You can also configure some parameters in CMS API. I did not notice significant differences in screen sharing quality though (regardless of codec the quality was bad) BUT I also didn't do an exhaustive testing of all possible combinations. Below an example how the browser can be forced to (not) use specific codecs, google should lead you a some deep rabbit hole if you're interested in it. You can use edge://webrtc-internals/ during a videocall to check what codec your browser is using for screen sharing.

"C:\Program Files\Google\Chrome\Application\chrome.exe" --disable-features=UseChromeOSDirectVideoDecoder

The encoding/decoding performance of codecs varyies significantly.  Below are the codecs that CMS as of today supports.

Video standards

●  H.261 ●  H.263 (+, ++) ●  H.264 AVC (baseline and high profile) ●  H.264 SVC ●  WebM, VP8 ●  Microsoft RTV ●  HTML5/WebRTC ●  SIP, H.323, TIP ●  BFCP ●  RDP ●  Far End Camera Control (FECC) passthrough ●  RTMP and RTMPS

Source: Cisco Meeting Server, Web App, and Meeting Management Revolutionize Team Collaboration Through High Scale and Advanced Interoperability in Audio, Web, and Video Conferencing Data Sheet - Cisco

I notice that the AV1 codec is missing, which apparently has a much better performance than H.264.  Webex uses AV1, CMS as it seems does not. Could that be the source of our problems?

AV1 Encoder - Chrome Platform Status (chromestatus.com)

AV1 encode is requested by a number of RTC applications, including Duo, Meet, and Webex. The primary benefits of AV1 are:

• Better compression efficiency to reduce bandwidth consumption and improve visual quality
• Enabling video for users on very low bandwidth networks (offering video at 30kbps and lower)
• Significant screen sharing efficiency improvements over VP9 and other codecs

What I didn't check so far:

• Check what codec competitors use that have no issues with screen sharing
• Check any Cisco roadmaps for AV1 support in CMS
• Open a business case for CMS to support the AV1 codec (in hopes that screen sharing gets better with it)
• Exhausive testing with all the various command line switches for Chrome / Edge to enable/disable hardware acceleration and all the various video codecs 
• Check if screen resolution of sender/receiver has any impact on presentation quality

That's about it. I still have no clue what the root cause for the terrible presentation quality is but I thought I'll just drop some of my findings here. Maybe somebody more intelligent / educated than me can provide clues about what might be going on.

To me it feels like a bug or some sort of technical limitation within the product.

Cheers,
Dave

Edit - Checked on 24. July 2023
According to reddit Microsoft Teams screen sharing uses the AV1 codec.
Zoom seems to be using AV1 as well.
Google Duo uses AV1.
Cisco Webex has implemented AV1.
Discord uses AV1.
CMS does not use AV1.

At around the 14:45min mark Thomas Davies demonstrates what sharing an excel sheet with AV1 codec looks and feels like in Webex. According to Wikipedia which references the linked video "a particular area of improvement was in high resolution screen sharing, compared to their previously deployed H.264 encoder".

[DaveLawang]

Attention: Please take everything written below with a big grain of salt. I'm not the video expert, just an annoyed UCC admin trying to find solutions for my users. I might have gotten things wrong on my journey to find a solution.

tl;dr:
Seems like CMS is knowingly or unknowingly imposing a ceiling by picking a value in the profile-level-id attribute that's far too low for screen sharing with nowadays screen resolutions.


Might have found something while comparing screen sharing in Microsoft Teams and CMS. Both use Openh264, a Cisco implementation. Same codec. In CMS shreen sharing sucks, in Microsoft Teams it doesn't. I wondered how that could be?

Those documents / postings might help us grasp what could be going on.

Understand Video Signaling with SDP Debugs - Cisco Community
Advanced Video Coding - Wikipedia (H.264 Levels)
BRKUCC-2006.pdf (ciscolive.com) (Page 255)

Here is what edge://webrtc-internals showed me while sharing my screen in Microsoft Teams and in CMS.

Attribute	MS Teams	CMS	What does it mean
???	107	104	No idea
level-asymmetry-allowed	1	1	No idea
packetization-mode	1	0	No idea (Check RFC6184 or this paper: Video Streaming Framework)
profile-level-id	42C02A	42800D	The 5th and 6th hex digits of the profile-level-id describe the Level. The Level describes the resolution, frame rate and bit rate that the endpoint can support. 16 hex = 22 dec = Level 2.2 = 352 x 480 pixels @ 30 frames per second (From the BRKUCC-2006.pdf linked above, P.255) Microsoft Teams: Level = 2A 2A(h)-> 42(d) -> H.264 Level 4.2 = 522 240 macroblocks / second Cisco CMS: Level 0D 0D(h) -> 13(d) -> H.264 Level 1.3 = 11 880 macroblocks / second
scalabilityMode	L1T1	L1T1	No idea
x-google-max-bitrate	NULL	2000	Probably not relevant
x-google-start-bitrate	NULL	2000	Probably not relevant
max-br	NULL	1666	Probably not relevant Edit: Maybe relevant. The values after the profile-level-id describe the receive capabilities of the endpoint above and beyond those described by the profile and level in the profile-level-id Max video bit rate = 1666kbps, Baseline profile level 1.3 value = 768kbps // Seems rather low for screen sharing to me, no?
max-cpb	NULL	4000	Probably not relevant Edit: Maybe relevant Max Coded Picture Buffer size = 4000 kbits (Baseline profile level 1.3 value = ?? kbits)
max-dpb	NULL	4752	Probably not relevant
max-fps	NULL	3000	Probably not relevant Edit: Maybe relevant Max Frames Per Second in 1/100s of a frame/second = 30 fps (Baseline profile level 1.3 value = 40 fps)
max-fs	NULL	32400	Probably not relevant Edit: Maybe relevant Max Frame Size = 32400 Macroblocks (Baseline profile level 1.3 value = 396) //This frame size seems very odd (!!) check page 294 in the standard. It's not included: H.264 : Advanced video coding for generic audiovisual services (itu.int) (Table A-1 – Level limits) // Here might be an explanation where this could be coming from, it's for a 16:9 4k resolution. // 3840 x 2160 / (16^2)
max-mbps	NULL	270000	Probably not relevant Edit: Maybe relevant Max Decoding speed = Max Macroblocks/sec = 270000 (Baseline profile level 1.3 value = 11880) //But even if that max-mbps is used it would be less than half of what MS Teams uses (522240)

Edit: Cisco is indirectly confirming the values I've measured here Configure and Troubleshoot CMS Live Streaming with VBrick DME - Cisco 

a=fmtp:97 profile-level-id=42800d;max-mbps=270000;max-fs=32400;max-cpb=4000;max-dpb=4752;max-br=1666;max-fps=3000

If I understood how those levels in the h264 specification work correctly then to me it seems like CMS is putting an artificial ceiling at 11880 macroblocks/second and I honestly have no clue why they might be doing that. If the webbridge respects that ceiling it could be that it's sending out far too few macroblocks per second to get a decent experience. If that's what is happening it is choking the videostream.

Based on those findings above I did some calculations.

My shared screen size is 2560x1440. With the macroblock ceiling Microsoft Teams is giving me above (2A -> 522240mbps) I could achieve a theoretical max. rate of ~36fps under ideal conditions. That's in case I've got my math right.

522240 (max macroblocks) * 16*16 (macroblock size in pixels) / (2560 * 1440) (my screen resolution) = ~36fps.

Cisco CMS with its 0D level seems to give me a ceiling of 11880 macroblocks per second. My current screen resolution of 2560x1440 consist of 14400 macroblocks though, I think you see the problem too. With my current screen resolution I'd get less than 1fps. Probably that's what my average would be, according to edge://webrtc-internals screen sharing fluctuates somewhere between 0-4fps. 

Wrapping things up from here:

Cisco lowering the max. screen resolution to 720p in CMS 3.7.1 won't fix the problem, you just get a few fps more before you hit the max. macroblock ceiling again. Cisco devs probably need to adjust the profile-level-id to some appropriate value.

How could someone verify this suspicion?
- Fire up Burp Proxy intercept traffic, replace CMS' profile-level-id 42800D with Microsofts value 42C02A and see what happens.

If this turns out to be the problem someone at Cisco owes me a beer.

-------------------------------

Final edit, 27. July 2023:

• Issue seems to be browser related (chromium to be precise)

As noted in one of the initial postings this indeed seems to be true. I am by now convinced that it is a client side bug that affects Edge and Chrome (Edit: 01. August: Not so sure about that anymore after diving into some wireshark traces. If I interpret them correctly it seems like the webbridge itself is the source of the profile-level-id 42800D, not the clients). Maybe everything else that is running on top of Chromium.

In Firefox we experience no major issues. Upon checking what profile-level-id Firefox uses, I noticed that it is 42e01f which translates into decimal value 31. This means Firefox in my setup uses h264 level 3.1 which supports those maximum resolutions/frame rates (or 108000macroblocks/second which is 10x more than Edge in my setup supports):

• 720×480@80.0
• 720×576@66.7
• 1,280×720@30.0
  
  
  

[matthewporterctr]

---

If this turns out to be the problem someone at Cisco owes me a beer.

---

I second that!   Cheers!!   

[DaveLawang]

Edit: Opened a TAC case.

Allright guys, care to help me out?
Tried to simplify troubleshooting this issue as much as I can. I'm fairly certain that this problem isn't related to our custom CMS configuration. Curious if somebody else is measuring the same or similar values like I do.

Based on the values I measure I suspect that this is what is happening when presenting in CMS.

Tool: Quick WebRTC Video Quality Checker (Attached as .zip-> rename .txt file inside to .html)

1. Either download & rename the file in the attached .zip here and skip to step 4. or
2. Copy code from here Quick WebRTC Video Quality Checker - Pastebin.com
3. Paste it here W3Schools Tryit Editor
4. Start up CMS, share your full screen
5. Open a tab with chrome://webrtc-internals or edge://webrtc-internals
6. Check what profile-level-id value you get in your RTP upstream for the presentation (Ctrl+F -> "Screenshare")
7. Check what profile-level-id value you get in your RTP downstream
8. Optional: Post the result here

Preview of the tool in the attached .zip

Current screen resolution:	2560x1440
Browser (Name, Version):	Check table below
What screen resolution was your outgoing screen presentation RTP upstream (to webbridge) set to (copy from tool above):	Check table below
What screen resolution was your the incoming presentation RTP downstream (from webbridge) set to (copy from tool above):	Check table below

I did a bunch of tests with the tool above. The values I measure let me assume that the current CMS implementation (I'm on 3.7 atm) is choking the presentation video upstream from the browser to webbridge to a 768kbit/s stream (careful: not confirmed.). Whatever happens between the webbridge and stream recipients (downstream) thus is not really relevant anymore. They either receive the same low quality videostream or if they try to receive a higher quality image they regulary run into buffer-underruns simply because not enough data to process is sent up to the webbridge in the specific timeframe. Just a hypothesis though, that's why I'd be curious about what you guys out there measure in your environments.

Product / Browser	Outbound settings (Upstream)	Inbound settings (Downstream)	Interpretation
Microsoft Teams (Edge v114.0.1823.67 )	42c02a 2,048×1,080@60.0 (4)	42e01f 1,280×720@30.0 (5)	Client that is presenting has a profile that is capped at 50,000kbits/s which is good for the screen size (2560x1440). The receiver receives a slightly scaled down version.
CMS (Edge v114.0.1823.67)	42800d 352×288@30.0 (6)	420032 3,672×1,536@26.7 (5)	Client that is presenting has a profile that is theoretically capped at 768kbit/s Receiving end receives a stream that is capable of 135,000kbits/s at 3,672x1,1536@26.7fps. A bit weird that receivers get offered a better quality stream than the sender but not sure if this is a problem. The rfc states that a certain level (here: 5) must support all lower levels. I understand that as the stream could go up to 135,000kbits/s but doesn't have to. Jitter of presentation is significantly higher than that of webcam (on average 5-10x higher, spikes up to >100x higher)
Microsoft Teams (Chrome v114.0.5735.199)	42c02a 2,048×1,080@60.0 (4)	42e01f 1,280×720@30.0 (5)	Client that is presenting has a profile that is capped at 50,000kbits/s which is good for the screen size (2560x1440). The receiver receives a slightly scaled down version.
CMS (Chrome v114.0.5735.199)	42800d 352×288@30.0 (6)	420032 3,672×1,536@26.7 (5)	Client that is presenting has a profile that is theoretically capped at 768kbit/s Receiving end receives a stream that is capable of 135,000kbits/s at 3,672x1,1536@26.7fps. Jitter of presentation is significantly higher than that of webcam (on average 5-10x higher, spikes up to >100x higher)
Microsoft Teams (Firefox v102.13.0esr (64-Bit))	42c02a 2,048×1,080@60.0 (4)	42e01f 1,280×720@30.0 (5)	Client that is presenting has a profile that is capped at 50,000kbits/s which is good for the screen size (2560x1440). The receiver receives a slightly scaled down version.
CMS (Firefox v102.13.0esr (64-Bit))	42e00d 352×288@30.0 (6)	42e01f 1,280×720@30.0 (5)	Client that is presenting has a profile that is theoretically capped at 768kbit/s Receiving end receives a stream that is capable of 14,000kbits/s at 1,280x720@30.0fps. Jitter of presentation isn't significantly higher than that of webcam (only +5-10ms max.)

Have a great weekend!

Cheers,
Dave