Webex Audio and Video Quality degraded

[Kathy N.]

We have over 2,000 staff using Webex to teach classes and in the last couple of days we've noticed a significant increase in complaints about their audio and video quality.  Most of them just upgraded to 40.12.3.11 and are now having issues with increased CPU usage during meetings as well as memory requirements.  Many of them are on devices that are a couple of years old but since this hasn't been an issue for them until this week, I'm hesitant to blame the computer.  Many of them have used the health checker and spark speed test and those come back with no issues.  We have asked them to turn off virtual backgrounds and disable video when possible but as teacher's the video is very important during class time.

Has anyone else noticed an increase from users with these types of complaints?  Any ideas on other ways to troubleshoot these issues?

[panthersdevelopers]

The recent increase in complaints about audio/video quality and high CPU/memory usage after the Webex upgrade to 40.12.3.11 is concerning. While older devices may contribute, the timing suggests the new version could be the culprit.

A few troubleshooting steps to consider:

• Check for any known issues or updates from Cisco regarding version 40.12.3.11
• Ensure all latest updates and drivers are installed on the affected devices
• Try rolling back to the previous Webex version to see if issues persist
• Optimize Webex settings for better performance (e.g., disabling HD video, adjusting video codecs)
• Check network bandwidth and connectivity for potential bottlenecks

Please let me know if the issues persist after trying these steps.

[smithmethew88]

I'm sorry to hear about the challenges your staff is facing with Webex after the recent upgrade. It's unfortunate that the audio and video quality issues, coupled with increased CPU usage and memory requirements, are impacting the teaching experience. While it's commendable that users have performed health checks and speed tests, it might be worth exploring the possibility of rolling back to the previous version (40.12.3.10) to see if the issues persist. Additionally, encourage users to check for any conflicting software or background processes that might be contributing to the increased resource usage. Lastly, considering the importance of video during classes, investigating alternative video editing solutions, like using an vn editor old version, might provide a temporary workaround until the Webex issues are resolved.

[Kathy N.]

@smithmethew88, the post you are replying to is from 2020.  We no longer have these issues.  Thank you for your suggestions as it appears that while you are shown as a beginner, you provided a well thought out response.

[paulypage3]

you can use the VN mod apk to enhance the video and audio quality

[shigehirom]

Hi Everyone,

Looks like my domain has been rolled back and fixed.  Here's the entire breakdown.

The upgrade to release 40.12 last week Wednesday basically broke the audio and video.  The upstream video to the cloud was working, as we saw recordings look good (at least for us).  I am still getting conflicting answers from varying engineers at Cisco.  One says its only our cluster while others are saying its more widespread.  We agreed that something unique about schools is that, our meetings generally are larger than business meetings (this could be why its more noticeable for us).  However, I've seen the issue w/ as few as 5-6 people on a call.  Working this issue as a P1 ticket finally got the attention needed to have my domain moved.

Here's what else I learned and what you may need to do:

* AW cluster is comprised mostly (maybe exclusively) with schools.  This is why all of us here are likely working for a school

* They have the ability to rollback domains to 40.11 individually.  Meaning, a provisioning personnel can do them one by one as opposed to moving the entire cluster.  In fact, I don't think they can roll back the whole cluster easily or they probably would have done that already.

* There is a long list being created at Cisco from customers requesting to be rolled back.  If your CSM isn't already on top of this, I highly suggest you contact him/her as quickly as possible

* If you have a CSM (I don't have one at the moment), you don't need to open a ticket.  The CSM can open an internal RITM ticket to have your domain moved by their provisioning folks.  In my case, my local Cisco account team escalated the issue for me.  Again, if you haven't created a ticket and pushed them to roll you back, the CSM can do that for you.

* The Hawaii DOE has not been rolled back as of 6am this morning 12/10 but they are working on it right now (thats what my sources tell me)

* After the rollback last night for my domain, I've tested w/ 25+ user meetings and everything tested normal

* Once you have been rolled back, they will lock your domain from any further upgrades until the issue is resolved

* The developers need to work on the root cause, so it'll be awhile before this is fixed

Hope this all helps you.

Shige

[btaitt]

When your site was rolled back to 40.11, did you also have to roll back the desktop clients on your computers to 40.11 as well, or if they have 40.12 installed is it still working better?

[pscanlan29]

Meetings will launch using the 40.11 meeting app, and the Desktop client can be on version 40.12 and everything still works fine

[shigehirom]

Good Question.... I missed posting that.

No, you do not need to downgrade the client software.  I was suprised by that too.

I confirmed w/ TAC that we do not need to downgrade.

The client sofware when you click about will show you a version like 40.12.4.7.  However, when you actually start a meeting or join one, you'll notice that when you click Help->About Cisco Webex Meetings, the version will actually show 40.11.5.11

All managed in the cloud.

[npitrone]

We are experiencing the same audio and video degradation and it did appear with the rollout of 40.12.3.11.   This issue seems to be even more prominent as users share content.  We were working well before this update and now we may be looking at a different platform.  We have hosted over 150,000 meetings as a district with over 2.5million participants so this new issue is a frustrating one.  I do have an open TAC issue and gone back and forth with the engineers.  We did get the new updated clients 40.12.4.7 but still are seeing the same issue.  Cisco WebEx is no longer a suitable solution for online learning with this issue present.  We are on cluster Aa

[pscanlan29]

Agreed on the issue being more prominent when sharing content, or just having the camera-enabled on the device was impacting the overall meeting performance. Our staff devices are not the newest, but they also aren't considered old by any means.

Cisco rolled back our version to 40.11 last night and so far things appear to be back to normal, but we are also only a few hours into teacher class sessions. We are keeping our ears to the ground as this progresses and hopefully, the issues with 40.12 are found and resolved so that we can upgrade again at some point.

[shigehirom]

I've been continuing to work w/ TAC on this issue.  If you haven't already, I strongly urge that you do so.  I'm currently trying to get the severity changed to a P1.  In Hawaii, I know this is also affecting the Dept of Education customers using WebEx (a district of almost 200,000 students).  I heard they are trying to rollback to 40.11 but I also know that my counterparts in other business sectors in Hawaii are NOT having this issue and they are on 40.12.  However, they are on a different cluster.

[Shawna P.]

This morning they just posted an update 40.12.4.7
Not sure if that’ll help but I just saw it this morning

[pscanlan29]

Our Webex instance took this update last night as well, nothing in the bug patch notes about resolving the issues that have been outlined in this post:

https://help.webex.com/en-us/c3r7uf/Open-and-Resolved-Bugs-for-the-Latest-Webex-Meetings-Updates#Cisco_Reference.dita_6134f5e0-f41a-4a22-903e-5b68e210bd3a

[jack.christensen]

We have the same issue with degraded service (video & audio).  How can we be alerted when TAC has identified and fixed the issue?  I have several users asking if we can use a different tool other than WebEx Meetings.  Our VP of Sales is asking for permission to purchase Zoom - he says he cannot continue to have sales calls with valued customers and not be able to communicate effectively.  The robotic audio & sketchy video is extremely unprofessional looking.

[pscanlan29]

Not sure about being alerted from TAC unless you have your own case open. I recommend you establish a case with TAC and let them know the number of users in your environment that are experiencing the issue. Is your Webex instance also in the AW cluster?

Are your users on Macs, Windows?

[pscanlan29]

We are on cluster AW as well and our users have been experiencing very similar issues with audio/video quality since the 40.12.3 update on 12/3. Does anyone have any more details or info on this?

We have a TAC case open and have been collecting logs from devices, all of our users are on Macs (most devices don't support virtual backgrounds). When we were on 40.11, no widespread issues or complaints from users about audio/video quality.

[shigehirom]

Same here on all accounts.  We have about 2000 ipads and 2000 macbooks.  All experiencing the issue to a degree.

Where are you located geographically?  or what State are you in?

[pscanlan29]

Located in the Bay Area (CA). We have a few thousand Macbook Airs being used by staff, students on Chromebooks and iPad in distance learning. Staff that are off-network (teaching from home) are also reporting the meeting performance issues.

[shigehirom]

I too have the same problem.  I am the site admin for over 5000 users at school in Hawaii.   Same issues started on 12/3 and we are also on version 40.12.3.11 on the client software.

I've created and escalated a ticket w/ Webex TAC who have not been helpful so far.  They keep saying something about hardware acceleration being off but there are no options on Macs to enable/disable this feature.  It's also happening on PCs as well as IOS devices.

At this point we are just hoping someone at Webex will isolate the issue.  I am planning to site wide disable the virtual backgrounds hoping this will help to reduce some of the issues being experienced.  To recap, we are also seeing users w/ no video available (a yellow triangle appears) or a gray box w/ a spinning wheel appears (like its trying to render the image).   In some situations the Webex session is crashing all together on our teachers laptops.

Lastly, we did discover that our cluster (AW) was upgraded to support 40.12 last Wednesday.  The Hawaii DOE is also on the same cluster as us (the DC also went through an extended upgrade this past weekend).  If you want to validate what cluster you are on just ping your domain: ie. ping hidoe.webex.com

The letters that follow nebula (in this case "aw") is the cluster you are on.  Than check out status.webex.com for information regarding when your node was upgraded.

[dcoope]

I just did the ping.  We're on AW as well.

Located in Minnesota