A bearded man uses his mobile device

Innovations in Ad Tech Mobile SDK Software Testing @ PubMatic

Headshot of Ganesh Salpure
By Ganesh Salpure, Manager, Software Development Engineer in Test
January 11, 2024

This post was cowritten by Mayur Waghmare, Senior Lead Software Development Engineer in Test

With the rapid increase in smartphone use, there has been a corresponding growth in the number of mobile apps. Ad-supported apps are one of the most popular revenue models, because they allow app developers to generate income while making the app free to the user.

Ads for these apps are mostly integrated and served using third-party software development kits (SDKs). When integrating third-party vendors’ technology, app publishers want high-quality SDKs to prevent creating a negative user experience for their app.

For that reason, it’s crucial for SDK development teams to thoroughly test their solution. SDK testing is not limited to writing and executing test scripts. At PubMatic, our software development engineer in test (SDET) teams have developed several innovative quality assurance methods to make sure that we’re delivering a high quality, industry-leading SDK solution:

Mobile device and operating system compatibility testing

There are a wide array of mobile devices differing in screen sizes, resolutions, hardware setups, and custom operating systems. Comprehensive testing of the SDK requires evaluating a broad range of devices and assessing the SDK with various iOS and Android versions.

To cover these device and platform alternatives, the SDK engineering team has developed native Android and iOS test apps that closely resemble a publisher app. The SDK is then integrated into this test app, exposing SDK functionality for testing and automation. Test automation scripts then cover all the underlying SDK features, including open measurement SDK (OMSDK) compliance, mobile rich media ad interface (MRAID)/Interstitial/Banner ads, StoreKit Ad Network (SKAdnet), privacy compliance, callbacks, ad events, reporting, and ad rendering. The automation interacts with banner, native, and video ads to replicate end-user interaction.

This testing method generates more than 30,000 automated tests. Completing those tests sequentially would require a prolonged testing and release cycle. To address this challenge, the SDET team has integrated their automation framework with multiple mobile device farms. The SDET team also developed a Multi-Cloud Test Distribution Engine which connects to the device farms and runs tests simultaneously on multiple mobile devices, accommodating both Android and iOS, as shown in this high-level flow diagram.

A flowchart for Multi Cloud Test Distrobution

PubMatic’s Multi-Cloud Test Distribution Engine coupled with multiple device farms provides the following benefits:

  • 30,000-plus tests run in couple of hours rather than multiple days
  • Shortened SDK release cycle time allows multiple quick releases each year
  • Accelerated issue reporting to developers
  • Expanded coverage for Android and iOS versions within a shorter timeframe
  • Enhanced mobile device coverage, enabling the identification of device-specific issues

Mobile app crashes and memory leaks

As SDET teams perform regression tests across diverse devices and operating systems, it’s essential to detect and report crashes and memory leaks.

To guarantee an SDK’s stability, PubMatic’s test app incorporates both Crashlytics and LeakCanary. As a result, when the automation is executed on a variety of devices and operating system versions within the device farm, the SDET team receives real-time reports of crashes and memory. The SDET team has customized the framework to gather additional details from the test which are furnished with the crash report making debugging and pinpointing easier for engineering team. This automated approach elevates the quality of the SDK to a whole new level.

Validating SDKs in a production-like environment

Validating the SDK within a setup that resembles a production environment poses another set of challenges. The PubMatic SDK supports Google Ad Manager and extends mediation support to IronSource, AdMob, and AppLovin. SDET teams ensure accurate ad rendering workflow and callbacks from these platforms, while also meeting their specific reporting standards.

To ensure end-to-end ad workflow, the automation framework sends test traffic to all integration endpoints. The SDET team has also developed a mock SSP setup which bids for test inventory traffic. This end-to-end testing approach covers the entire ad serving process, from the ad server to SDK rendering and reporting. Furthermore, the automation also validates mediation-specific aspects such as callbacks, ad rendering, and custom parameters. By mimicking a production-like environment for publishers, this setup plays a crucial role in maintaining the high quality of the SDK throughout the entire ad ecosystem.

Test App Flowchart

  • Mobile Device and Operating System Compatibility Testing
    The mobile landscape boasts a wide array of mobile devices differing in screen sizes, resolutions, hardware setups, and custom operating systems. To ensure stability across manufacturers, organizations require access to a broad range of devices. Additionally, testing compatibility with various iOS and Android versions adds complexity, demanding more time from SDET teams and impacting release schedules. Moreover, the test execution time is significantly higher with these hardware and software combinations as the SDET team has more than 30,000-plus automated test scripts/cases.
  • Mobile App Crashes and Memory Leaks
    The most pressing issue for app publishers revolves around averting app crashes and memory leaks. As SDET teams perform regression tests across diverse devices and operating systems, a mechanism is essential to detect and report instances of crashes and memory leaks. Reporting a crash might be easy but pinpointing the crash or leak when 30,000-plus tests are running is also a challenge.
  • Testing SDKs in Isolation
    An SDK is a single piece of software on its own. This characteristic poses a unique challenge for testing – How will SDET teams comprehensively test critical SDK functionalities such as ad rendering, VAST events, OS events, privacy compliance, OM SDK compliance, and user interactions with banner and interstitial ads within the confines of this standalone binary architecture without an app context?
  • Validating SDKs in a Production-Like Environment
    Validating the SDK within a setup resembling a production environment poses comparable difficulties. The PubMatic SDK supports Google Ad Manager (GAM and extends mediation support to IronSource, AdMob, and AppLovin. SDET teams need to ensure accurate ad rendering workflow and callbacks from these platforms, while also meeting their specific reporting standards. Setting up these workflows in a test environment is not very straightforward and involves many moving components, especially if one wants to automate it end-to-end.

Mobile SDK Software Testing Solutions at PubMatic

  • In-house test app integration with Android and iOS OpenWrap SDK
    The SDK engineering team has developed native Android and iOS test apps that closely resemble a publisher app. The SDK is then integrated into this test app as a publisher would. This exposes SDK functionality for testing and automation to SDET teams. Test automation scripts interact with the test app giving test coverage to all underlying SDK features, such as OMSDK compliance, MRAID/Interstitial/Banner ads, SKAdnet, privacy compliance, callbacks, ad ents, reporting, and ad rendering. The automation scripts interact with banner, native, and video ads in a manner that replicates end-user interactions. Through this approach, the test automation and the test app ensure a seamless in-app advertising experience delivered by OpenWrap SDK.
  • Parallel Execution with Device Farm Integration
    As mentioned, the SDK has to ensure compatibility with a vast combination of hardware and operating systems; executing 30,000-plus automated tests in a sequential manner would take multiple days to complete the testing and release cycle. To address this challenge, the SDET team successfully integrated their in-house pytest-based automation framework with multiple mobile device farms. Also, an in-house Multi-Cloud Test Distribution Engine was developed from scratch by the SDET team. The automation framework, integrated with CI/CD, connects to the device farms and runs tests simultaneously on multiple mobile devices, accommodating both Android and iOS coverage. Here’s the parallel framework architecture at a high level:A flowchart for Multi Cloud Test Distrobution
    PubMatic’s Multi-Cloud Test Distribution Engine innovation coupled with multiple device farms has resulted in the following benefits:
    • 30,000-plus tests run in couple of hours rather than multiple days
    • Shortened SDK release cycle time facilitating multiple quick releases in year
    • Accelerated issue reporting to developers
    • Expanded coverage for Android and iOS versions within a shorter timeframe
    • Enhanced mobile device coverage, enabling the identification of device-specific issues
  • Crashlytics and LeakCanary Integration
    To guarantee an SDK’s stability, the test app mentioned earlier incorporates Crashlytics and LeakCanary alongside the SDK. As a result, when the automation is executed on a variety of devices and operating system versions within the device farm, the SDET team receives real-time reports of crashes and memory leaks through a Slack integration. Also, with a few framework customizations done by SDET team, additional details from the test and devices are furnished with this crash report making debugging and pinpointing easier for engineering team. This automated approach elevates the quality of the SDK to a whole new level.
  • Production-like setup with AdServer and Mediation Flow
    OpenWrap SDK seamlessly integrates with GAM and provides mediation support for AdMob, IronSource, and AppLovin. To ensure end-to-end ad workflow, the automation framework sends test traffic to all integration endpoints. The SDET team has also developed a mock SSP setup which bids for test inventory traffic. This end-to-end testing approach covers the entire ad serving process, from the ad server to SDK rendering and reporting. Furthermore, the automation also validates mediation-specific aspects such as callbacks, ad rendering, and custom params. By mimicking a production-like environment for publishers, this setup plays a crucial role in maintaining the high quality of the SDK throughout the entire ad ecosystem.

Test App Automation Flowchart

In summary, the PubMatic SDET team has developed an innovative automation solution to solve the unique challenges the adtech industry faces in SDK testing. Our customers’ business success using our products is testament to the value of our strategy.