🚀 NEW My notes, your AI note taker Â
Automatically captures, summarizes, and extracts action items from any virtual or in-person meeting. Learn more
Audio and video quality are at the core of the virtual meeting experience. When one or the other is subpar or worse, doesn’t function at all, the user experience is degraded, causing frustration. At Zoom, we always want to know how Zoom Meetings compare to other meeting solutions. So we tasked TestDevLab to compare video and audio quality under a variety of conditions and call types for the following vendors:
The findings provided in this report reflect TestDevLab testing results from April 2024.
TestDevLab (TDL) offers software testing & quality assurance services for mobile, web, and desktop applications and custom testing tool development. They specialize in software testing and quality assurance, developing advanced custom solutions for security, battery, and data usage testing of mobile apps. They also offer video and audio quality assessment for VoIP or communications apps and products for automated backend API, as well as web load and performance testing.
TestDevLab team comprises 500+ highly experienced engineers who focus on software testing and development. The majority of their test engineers hold International Software Testing Qualifications Board (ISTQB) certifications, offering a high level of expertise in software testing according to industry standards and service quality.
To maintain a fair test across all vendors, all tests were conducted using TestDevLab equipment and accounts. In addition, the individual vendor accounts were kept at a default configuration with the exception of enabling a maximum resolution of HD (1080p) for vendors that default to limiting video resolution.
The testing scenarios included a “Receiver” device, a “Sender” device, and additional devices that acted as additional participants. All devices were connected to the same video call. Depending on the testing scenario, the call layout and network were configured to reproduce the required test parameters.
To provide an overall picture of video and audio quality in Meetings, we asked TDL to test the most common meeting scenarios, using commonly used hardware.
Variable network scenarios
For these scenarios, we wanted to test how the different platforms react in various network conditions using a few common compute platforms. To achieve this goal, TDL used four common hardware platforms: MacBook Pro computers, Windows computers, iPhones, and Android devices. We configured the most common meeting type with three participants, no shared content, and a presenter/speaker view setting to maximize video resolution.
The network scenarios that were tested included:
Each call lasted for three minutes and was repeated five times. Across all vendors, hardware variations, and network conditions, TDL conducted 420 meetings.
When limitations were placed on a call, they were limited to the tested device acting as a “receiver.”
Variable call type scenarios
For these scenarios, we wanted to test the various call types and technologies used by most users. The hardware used was limited to MacBook Pro M1 and Windows computers. The scenarios included five- and eight-person meetings arranged in gallery mode. TDL tested meetings with participants only, meetings with content sharing, and meetings where participants activated virtual backgrounds.
In all cases, the network remained clean and unrestricted. As before, each meeting lasted three minutes and was repeated five times. Overall, TDL conducted an additional 240 meetings in this category.
To generate a complete picture of meeting quality, we asked TDL to capture metrics in four major categories related to video and audio quality, network utilization, and resource usage. The metrics were captured every second of the meeting and were saved to a centralized database.
Audio quality
Video quality
Resource utilization
Network utilization
Not all metrics can be captured on all platforms. For example, not all resource utilization metrics can be captured on mobile devices. TDL maximized metrics capture in each call scenario.
To simplify the presentation of the vast amount of information collected by TDL, we scored each of the above metrics on a 1-4 scale, where 1 is the best score and 4 is the worst. We then totaled the scores from the metrics into the four categories. The lowest totals represent the best quality scores in each category.
This section explores the results at a high level with various filters applied to focus on different call categories. Later, we will focus on individual call scenarios and provide more detailed information on the behavior of the different platforms in those scenarios. See “Detailed Results” for more in-depth information.
The above results include scores from all 660 meetings, including all devices and network scenarios:
The overall score above represents the entirety of TDL tests, but in reality, most regular users won’t experience many of the network limitations simulated by TDL. The results below filter out all tests that had network limitations, keeping only the clean network tests.
When looking at clean network meetings, Zoom (25) scores even better, Google Meet (37) scores second, and Microsoft Teams (39) and Cisco Webex (39) score last.
Scenarios using desktop devices only
While many people use mobile devices to connect to meetings, an average of ~66% of all Zoom participants connect using desktop devices (according to internal Zoom statistics). When filtering the data to Desktop only and meetings without network issues, Zoom’s lead increases even further, and also leads in all four metrics.
Zoom outperforms the tested alternative meeting services in every category TDL measured under ideal conditions and the most commonly used devices. As conditions are not always ideal, it’s important to measure circumstances where users may experience network problems.
Vendors can automatically adjust to changing network conditions to maintain high-quality video and audio. Each vendor’s protocol adapts differently, potentially changing video resolution, compression quality, FPS, and other factors. The overall user experience during a video call depends on how well the vendor can keep these parameters optimized as network conditions fluctuate.
TestDevLab collected multiple metrics to measure the overall user experience in a call. The two most impactful metrics collected are VMAF, which measures the perceived image quality, and Frames Per Second (FPS), which shows the fluidity of the video.
After analyzing the combined test results of 660 tests, TDL found that Google Meet provides the highest average frame rate at 20 frames per second (fps). In comparison, Microsoft Teams and Zoom both average 17 fps, while Webex averages 16 fps.
However, Google Meet's higher frame rate comes with a trade-off in video quality. According to the Video Multimethod Assessment Fusion (VMAF) chart, Zoom leads in quality with an average score of 73.2. Microsoft Teams follows with a score of 63.5, Google Meet scores 59.5, and Webex has an average score of 58.9.
Video quality with packet loss
The packet loss test conducted by TestDevLab included three stages.
As shown in the charts below:
Network recovery test
In the network recovery test, TDL started the test with a clean network.
This test was designed to test the behavior of the tested platforms when a temporary issue occurs on the network and the ability of each platform to recover when the issue no longer exists.
The test results show that while Zoom maintains high-quality video, it takes ~10 seconds to readjust its codec parameters to the new network conditions, resulting in a very low frame rate for a number of seconds. Regarding recovery time, Zoom is in the middle of the pack, between Webex with very low recovery time to Teams and Google with fast recovery.
Congested network test
The goal of the Congested Network Test was to evaluate the behavior of the tested platforms under varying durations of network congestion and their ability to maintain performance on networks with intermittent traffic bloats.
The test results show that Zoom maintains image quality throughout the test while sacrificing frame rate and video delay to accommodate the changing network conditions. Notably, Webex completely dropped the video by the end of the test.
Virtual background quality test
In our testing scenarios, we tasked TDL with evaluating the user experience of the virtual background feature. This feature replaces the user's real background with an image or video, requiring significant processing to generate a high-quality composite image.
According to TDL's results, Zoom delivered the best image quality. Some meeting clients prioritize video quality, and some prioritize frame rate when scoring by VMAF:
These findings are consistent with Zoom internal virtual background quality tests conducted last year, which are shown in this video.
Additionally, TDL's tests revealed that Zoom uses the least CPU and memory resources when utilizing the virtual background feature.
TDL's extensive testing highlights significant differences in performance among Zoom, Microsoft Teams, Google Meet, and Cisco Webex under various network conditions and meeting scenarios.
