QoE Testing Paradigms for Media Streaming Platforms: Is Automation Your Answer

In a previous blog, I spoke about how evolving Quality of Experience (QoE) paradigms, especially in testing and validation, are transforming the Over the Top (OTT) media landscape. With features like on-demand and customized content availability, multi-device viewing capabilities, and superior quality helping drive, define and reshape the ecosystem, it is no surprise that the OTT revolution, powered by QoE, will only continue to get stronger in the days to come. 

Let us delve a little deeper in the details of QoE and its testing to understand its role and impact on the success of global media streaming platforms.

The Factors Affecting QoE

QoE is primarily impacted by four workflows :

• Player Workflows: Where new application features are added or updated, new SDKs are integrated, etc.
• Content Workflows: Where the mezzanine video get transcoded, encrypted , packaged, and pushed to the CDN
• Encoding Profile Workflows: Where  there are multiple sub-changes and optimizations in the Transcoding section of content Workflow
• Infra Workflows: Where other various other systems of the OTT application backend like DRM , protocols, watermarking, and CMS is updated

                                                                                                                      The QoE  Quality Workflow

These key components of the OTT ecosystem require extensive testing in the pre-production environment. And once the application is released (or pushed) to the production/customer stage, a player beacon captures and reports on the customer QoE, establishing a direct correlation between this vital indicator and the success of the streaming platform.

What we need to remember here is that since any QoE entity tested is a combination of various parts of the OTT ecosystem, it is important to test it systematically and in an organized manner - with a clear understanding of the viewer ecosystem and the expected usage patterns.

Manual vs. Automated QoE Testing: Navigating the Conundrum

End device testing is the best way to identify the QoE of a media streaming platform. A QoE test comes under the NFR category, which is undertaken in considerably lesser facilities as compared to functional tests.

Again, QoE testing today is mostly manual or semi-automated (at best), which results in the need for considerable time and investments to ensure and deliver on the expected parameters. With viewer demand patterns continuing to be significantly dynamic, there is an urgent need to optimize on these cycle times to ensure enhanced outcomes.

The scenario is further complicated by the fact that manual (or semi-automated) QoE testing does not always help uncover the right correlation required, including such vital aspects like start up time, buffering across bandwidth streams, ABR performance, and the impact of other infrastructural changes.

We therefore need to compare and contrast the changes and drive the development of new streaming optimization levers for redefining existing QoE testing paradigms.

Enter automation.

While the automation of QoE Testing is considerably more complex than normal functional automation due to kits need for an altogether different logic and thought process, it is not an unachievable target. The journey initially comprises of various functional automation scripts and logics when we automate an application and navigate through its EPG to the playback scenarios.

What is critical here is the need to design the APIs and scripts capable of the streaming QoE KPIs from video playback initiation. The requirement is to focus on the right usage of automation scripting techniques and capable scripting of APIs provided within the desired automation tool or framework.

We need to remember that the normal NFR and performance measurement APIs or techniques may not be directly usable in automated testing scenarios. And further, the KPIs have to be measured along with a set of other parameters that define the state of the system when the measurement had taken place. Measuring buffering ratio against varying Wi-Fi signal strength levels (received at the end device) is one example that comes to mind.

Our target is not only to rely on the return from the QoE APIs , but to aggregate and compare the right metadata and environmental conditions to derive meaningful insights where the presentation layer of these test and benchmark data becomes important. A good QoE testing system, therefore, not only provides the APIs or functions to measure the relevant KPIs, but is also capable of network manipulation, collecting network requests, application or player logs, and device logs etc.

Such an auto, mated solution should be capable of automatically searching for all data points necessary to triage and decide on actions to be taken for improvement. This capability is a clear line of separation between QoE tools built for purpose and NFR capable test automation tools.

Challenges in QoE Automation

• Measuring the KPIs in 20 milli second intervals or with frame level accuracy
• Power-intensive GPU-based based processing
• Bringing together all relevant metadata to correlate and generate meaningful insights
• Approach  to reliably manipulate network models
• Availability of an applicable set of network model or templates to apply for QoE tests

Navigating the Future

The challenges and roadblocks aside, the future clearly belongs to a renewed focus on ensuring and delivering a best-in-class QoE, one that would help differentiate the target streaming platform from its rivals and competitors. And automated testing, powered by deep digital engineering domain expertise, is the way forward.

Operation

1.A QoE test is started off the back of a CI trigger, schedule or manual execution

2.QoE Test Manager kicks off the test on a specific device

3.QoE Test solution automate the app into a play ready state

4.Cogmation is independent of the player, simulating what the customer would see

5.A common test pattern stresses the player across a range of likely scenarios

6.A network proxy controls the stream being viewed, the bandwidth its viewed at and records the data exchanged

7.Asset playback runs , containing a number of stop-start events and advertisements

8.The specially developed QoE metrics Capture API captures key facets of QoE

9.QoE KPIs captured are reported with relevant metadata

10.video analytics reported values are validated

11.Results are presented in a report and returned to the calling CI pipeline as a pass/fail where appropriate