Webex Audio and Video Quality degraded

[Kathy N.]

We have over 2,000 staff using Webex to teach classes and in the last couple of days we've noticed a significant increase in complaints about their audio and video quality.  Most of them just upgraded to 40.12.3.11 and are now having issues with increased CPU usage during meetings as well as memory requirements.  Many of them are on devices that are a couple of years old but since this hasn't been an issue for them until this week, I'm hesitant to blame the computer.  Many of them have used the health checker and spark speed test and those come back with no issues.  We have asked them to turn off virtual backgrounds and disable video when possible but as teacher's the video is very important during class time.

Has anyone else noticed an increase from users with these types of complaints?  Any ideas on other ways to troubleshoot these issues?

[Kathy N.]

The improvements to the Webex App and migration to the Webex Suite has reduced the issues in my original post from December, 2020 significantly.   If you're having issues with Webex, please either find a more recent post with version either 44.x or 45.x to see if others are having an issue or start a new post with your specific issues.  Please avoid responding to this post as versions older than 44.2 are at End-of-Life and no longer meet Cisco's policy requirements.

[nimramalik]

I am facing the same issue here on all accounts. 

[aopp7486452]

It sounds like the update might be straining older hardware. You could try lowering Webex's video resolution settings or recommending a browser-based option to reduce resource usage. Updating device driver's or testing on another network might also help pinpoint the issue.

[Kathy N.]

@smithmethew88, the post you are replying to is from 2020.  We no longer have these issues.  Thank you for your suggestions as it appears that while you are shown as a beginner, you provided a well thought out response.

[dexterhale398]

Hi there,
The increased CPU usage could be tied to new processing demands in version 40.12.3.11, particularly with video encoding or virtual background handling. Verify if hardware acceleration is enabled for video processing in Webex settings. Additionally, check system diagnostics for processes like wbxrender or similar, which may indicate higher GPU/CPU utilization.

[shigehirom]

Hi Everyone,

Looks like my domain has been rolled back and fixed.  Here's the entire breakdown.

The upgrade to release 40.12 last week Wednesday basically broke the audio and video.  The upstream video to the cloud was working, as we saw recordings look good (at least for us).  I am still getting conflicting answers from varying engineers at Cisco.  One says its only our cluster while others are saying its more widespread.  We agreed that something unique about schools is that, our meetings generally are larger than business meetings (this could be why its more noticeable for us).  However, I've seen the issue w/ as few as 5-6 people on a call.  Working this issue as a P1 ticket finally got the attention needed to have my domain moved.

Here's what else I learned and what you may need to do:

* AW cluster is comprised mostly (maybe exclusively) with schools.  This is why all of us here are likely working for a school

* They have the ability to rollback domains to 40.11 individually.  Meaning, a provisioning personnel can do them one by one as opposed to moving the entire cluster.  In fact, I don't think they can roll back the whole cluster easily or they probably would have done that already.

* There is a long list being created at Cisco from customers requesting to be rolled back.  If your CSM isn't already on top of this, I highly suggest you contact him/her as quickly as possible

* If you have a CSM (I don't have one at the moment), you don't need to open a ticket.  The CSM can open an internal RITM ticket to have your domain moved by their provisioning folks.  In my case, my local Cisco account team escalated the issue for me.  Again, if you haven't created a ticket and pushed them to roll you back, the CSM can do that for you.

* The Hawaii DOE has not been rolled back as of 6am this morning 12/10 but they are working on it right now (thats what my sources tell me)

* After the rollback last night for my domain, I've tested w/ 25+ user meetings and everything tested normal

* Once you have been rolled back, they will lock your domain from any further upgrades until the issue is resolved

* The developers need to work on the root cause, so it'll be awhile before this is fixed

Hope this all helps you.

Shige

[btaitt]

When your site was rolled back to 40.11, did you also have to roll back the desktop clients on your computers to 40.11 as well, or if they have 40.12 installed is it still working better?

[pscanlan29]

Meetings will launch using the 40.11 meeting app, and the Desktop client can be on version 40.12 and everything still works fine

[shigehirom]

Good Question.... I missed posting that.

No, you do not need to downgrade the client software.  I was suprised by that too.

I confirmed w/ TAC that we do not need to downgrade.

The client sofware when you click about will show you a version like 40.12.4.7.  However, when you actually start a meeting or join one, you'll notice that when you click Help->About Cisco Webex Meetings, the version will actually show 40.11.5.11

All managed in the cloud.

[npitrone]

We are experiencing the same audio and video degradation and it did appear with the rollout of 40.12.3.11.   This issue seems to be even more prominent as users share content.  We were working well before this update and now we may be looking at a different platform.  We have hosted over 150,000 meetings as a district with over 2.5million participants so this new issue is a frustrating one.  I do have an open TAC issue and gone back and forth with the engineers.  We did get the new updated clients 40.12.4.7 but still are seeing the same issue.  Cisco WebEx is no longer a suitable solution for online learning with this issue present.  We are on cluster Aa

[pscanlan29]

Agreed on the issue being more prominent when sharing content, or just having the camera-enabled on the device was impacting the overall meeting performance. Our staff devices are not the newest, but they also aren't considered old by any means.

Cisco rolled back our version to 40.11 last night and so far things appear to be back to normal, but we are also only a few hours into teacher class sessions. We are keeping our ears to the ground as this progresses and hopefully, the issues with 40.12 are found and resolved so that we can upgrade again at some point.

[shigehirom]

I've been continuing to work w/ TAC on this issue.  If you haven't already, I strongly urge that you do so.  I'm currently trying to get the severity changed to a P1.  In Hawaii, I know this is also affecting the Dept of Education customers using WebEx (a district of almost 200,000 students).  I heard they are trying to rollback to 40.11 but I also know that my counterparts in other business sectors in Hawaii are NOT having this issue and they are on 40.12.  However, they are on a different cluster.

[Shawna P.]

This morning they just posted an update 40.12.4.7
Not sure if that’ll help but I just saw it this morning

[pscanlan29]

Our Webex instance took this update last night as well, nothing in the bug patch notes about resolving the issues that have been outlined in this post:

https://help.webex.com/en-us/c3r7uf/Open-and-Resolved-Bugs-for-the-Latest-Webex-Meetings-Updates#Cisco_Reference.dita_6134f5e0-f41a-4a22-903e-5b68e210bd3a

[jack.christensen]

We have the same issue with degraded service (video & audio).  How can we be alerted when TAC has identified and fixed the issue?  I have several users asking if we can use a different tool other than WebEx Meetings.  Our VP of Sales is asking for permission to purchase Zoom - he says he cannot continue to have sales calls with valued customers and not be able to communicate effectively.  The robotic audio & sketchy video is extremely unprofessional looking.

[pscanlan29]

Not sure about being alerted from TAC unless you have your own case open. I recommend you establish a case with TAC and let them know the number of users in your environment that are experiencing the issue. Is your Webex instance also in the AW cluster?

Are your users on Macs, Windows?

[pscanlan29]

We are on cluster AW as well and our users have been experiencing very similar issues with audio/video quality since the 40.12.3 update on 12/3. Does anyone have any more details or info on this?

We have a TAC case open and have been collecting logs from devices, all of our users are on Macs (most devices don't support virtual backgrounds). When we were on 40.11, no widespread issues or complaints from users about audio/video quality.

[shigehirom]

Same here on all accounts.  We have about 2000 ipads and 2000 macbooks.  All experiencing the issue to a degree.

Where are you located geographically?  or what State are you in?

[pscanlan29]

Located in the Bay Area (CA). We have a few thousand Macbook Airs being used by staff, students on Chromebooks and iPad in distance learning. Staff that are off-network (teaching from home) are also reporting the meeting performance issues.

[shigehirom]

I too have the same problem.  I am the site admin for over 5000 users at school in Hawaii.   Same issues started on 12/3 and we are also on version 40.12.3.11 on the client software.

I've created and escalated a ticket w/ Webex TAC who have not been helpful so far.  They keep saying something about hardware acceleration being off but there are no options on Macs to enable/disable this feature.  It's also happening on PCs as well as IOS devices.

At this point we are just hoping someone at Webex will isolate the issue.  I am planning to site wide disable the virtual backgrounds hoping this will help to reduce some of the issues being experienced.  To recap, we are also seeing users w/ no video available (a yellow triangle appears) or a gray box w/ a spinning wheel appears (like its trying to render the image).   In some situations the Webex session is crashing all together on our teachers laptops.

Lastly, we did discover that our cluster (AW) was upgraded to support 40.12 last Wednesday.  The Hawaii DOE is also on the same cluster as us (the DC also went through an extended upgrade this past weekend).  If you want to validate what cluster you are on just ping your domain: ie. ping hidoe.webex.com

The letters that follow nebula (in this case "aw") is the cluster you are on.  Than check out status.webex.com for information regarding when your node was upgraded.