This post was cowritten by Mayur Waghmare, Senior Lead Software Development Engineer in Test
With the rapid increase in smartphone use, there has been a corresponding growth in the number of mobile apps. Ad-supported apps are one of the most popular revenue models, because they allow app developers to generate income while making the app free to the user.
Ads for these apps are mostly integrated and served using third-party software development kits (SDKs). When integrating third-party vendors’ technology, app publishers want high-quality SDKs to prevent creating a negative user experience for their app.
For that reason, it’s crucial for SDK development teams to thoroughly test their solution. SDK testing is not limited to writing and executing test scripts. At PubMatic, our software development engineer in test (SDET) teams have developed several innovative quality assurance methods to make sure that we’re delivering high-quality Ad Tech mobile SDK software testing innovations and an industry-leading SDK solution:
Mobile device and operating system compatibility testing
There are a wide array of mobile devices differing in screen sizes, resolutions, hardware setups, and custom operating systems. Comprehensive testing of the SDK requires evaluating a broad range of devices and assessing the SDK with various iOS and Android versions.
To cover these device and platform alternatives, the SDK engineering team has developed native Android and iOS test apps that closely resemble a publisher app. The SDK is then integrated into this test app, exposing SDK functionality for testing and automation. Test automation scripts then cover all the underlying SDK features, including open measurement SDK (OMSDK) compliance, mobile rich media ad interface (MRAID)/Interstitial/Banner ads, StoreKit Ad Network (SKAdnet), privacy compliance, callbacks, ad events, reporting, and ad rendering. The automation interacts with banner, native, and video ads to replicate end-user interaction.
This testing method generates more than 30,000 automated tests. Completing those tests sequentially would require a prolonged testing and release cycle. To address this challenge, the SDET team has integrated their automation framework with multiple mobile device farms. The SDET team also developed a Multi-Cloud Test Distribution Engine which connects to the device farms and runs tests simultaneously on multiple mobile devices, accommodating both Android and iOS, as shown in this high-level flow diagram.
PubMatic’s Multi-Cloud Test Distribution Engine coupled with multiple device farms provides the following benefits:
- 30,000-plus tests run in couple of hours rather than multiple days
- Shortened SDK release cycle time allows multiple quick releases each year
- Accelerated issue reporting to developers
- Expanded coverage for Android and iOS versions within a shorter timeframe
- Enhanced mobile device coverage, enabling the identification of device-specific issues
Mobile app crashes and memory leaks
As SDET teams perform regression tests across diverse devices and operating systems, it’s essential to detect and report crashes and memory leaks.
To guarantee an SDK’s stability, PubMatic’s test app incorporates both Crashlytics and LeakCanary. As a result, when the automation is executed on a variety of devices and operating system versions within the device farm, the SDET team receives real-time reports of crashes and memory. The SDET team has customized the framework to gather additional details from the test which are furnished with the crash report making debugging and pinpointing easier for engineering team. This automated approach elevates the quality of the SDK to a whole new level.
Validating SDKs in a production-like environment
Validating the SDK within a setup that resembles a production environment poses another set of challenges. The PubMatic SDK supports Google Ad Manager and extends mediation support to IronSource, AdMob, and AppLovin. SDET teams ensure accurate ad rendering workflow and callbacks from these platforms, while also meeting their specific reporting standards.
To ensure end-to-end ad workflow, the automation framework sends test traffic to all integration endpoints. The SDET team has also developed a mock SSP setup which bids for test inventory traffic. This end-to-end testing approach covers the entire ad serving process, from the ad server to SDK rendering and reporting. Furthermore, the automation also validates mediation-specific aspects such as callbacks, ad rendering, and custom parameters. By mimicking a production-like environment for publishers, this setup plays a crucial role in maintaining the high quality of the SDK throughout the entire ad ecosystem.
- Mobile Device and Operating System Compatibility Testing
The mobile landscape boasts a wide array of mobile devices differing in screen sizes, resolutions, hardware setups, and custom operating systems. To ensure stability across manufacturers, organizations require access to a broad range of devices. Additionally, testing compatibility with various iOS and Android versions adds complexity, demanding more time from SDET teams and impacting release schedules. Moreover, the test execution time is significantly higher with these hardware and software combinations as the SDET team has more than 30,000-plus automated test scripts/cases. - Mobile App Crashes and Memory Leaks
The most pressing issue for app publishers revolves around averting app crashes and memory leaks. As SDET teams perform regression tests across diverse devices and operating systems, a mechanism is essential to detect and report instances of crashes and memory leaks. Reporting a crash might be easy but pinpointing the crash or leak when 30,000-plus tests are running is also a challenge. - Testing SDKs in Isolation
An SDK is a single piece of software on its own. This characteristic poses a unique challenge for testing – How will SDET teams comprehensively test critical SDK functionalities such as ad rendering, VAST events, OS events, privacy compliance, OM SDK compliance, and user interactions with banner and interstitial ads within the confines of this standalone binary architecture without an app context? - Validating SDKs in a Production-Like Environment
Validating the SDK within a setup resembling a production environment poses comparable difficulties. The PubMatic SDK supports Google Ad Manager (GAM and extends mediation support to IronSource, AdMob, and AppLovin. SDET teams need to ensure accurate ad rendering workflow and callbacks from these platforms, while also meeting their specific reporting standards. Setting up these workflows in a test environment is not very straightforward and involves many moving components, especially if one wants to automate it end-to-end.
Mobile SDK Software Testing Solutions at PubMatic
- In-house test app integration with Android and iOS OpenWrap SDK
The SDK engineering team has developed native Android and iOS test apps that closely resemble a publisher app. The SDK is then integrated into this test app as a publisher would. This exposes SDK functionality for testing and automation to SDET teams. Test automation scripts interact with the test app giving test coverage to all underlying SDK features, such as OMSDK compliance, MRAID/Interstitial/Banner ads, SKAdnet, privacy compliance, callbacks, ad ents, reporting, and ad rendering. The automation scripts interact with banner, native, and video ads in a manner that replicates end-user interactions. Through this approach, the test automation and the test app ensure a seamless in-app advertising experience delivered by OpenWrap SDK.
- Parallel Execution with Device Farm Integration
As mentioned, the SDK has to ensure compatibility with a vast combination of hardware and operating systems; executing 30,000-plus automated tests in a sequential manner would take multiple days to complete the testing and release cycle. To address this challenge, the SDET team successfully integrated their in-house pytest-based automation framework with multiple mobile device farms. Also, an in-house Multi-Cloud Test Distribution Engine was developed from scratch by the SDET team. The automation framework, integrated with CI/CD, connects to the device farms and runs tests simultaneously on multiple mobile devices, accommodating both Android and iOS coverage. Here’s the parallel framework architecture at a high level:
PubMatic’s Multi-Cloud Test Distribution Engine innovation coupled with multiple device farms has resulted in the following benefits:
-
- 30,000-plus tests run in couple of hours rather than multiple days
- Shortened SDK release cycle time facilitating multiple quick releases in year
- Accelerated issue reporting to developers
- Expanded coverage for Android and iOS versions within a shorter timeframe
- Enhanced mobile device coverage, enabling the identification of device-specific issues
- Crashlytics and LeakCanary Integration
To guarantee an SDK’s stability, the test app mentioned earlier incorporates Crashlytics and LeakCanary alongside the SDK. As a result, when the automation is executed on a variety of devices and operating system versions within the device farm, the SDET team receives real-time reports of crashes and memory leaks through a Slack integration. Also, with a few framework customizations done by SDET team, additional details from the test and devices are furnished with this crash report making debugging and pinpointing easier for engineering team. This automated approach elevates the quality of the SDK to a whole new level. - Production-like setup with AdServer and Mediation Flow
OpenWrap SDK seamlessly integrates with GAM and provides mediation support for AdMob, IronSource, and AppLovin. To ensure end-to-end ad workflow, the automation framework sends test traffic to all integration endpoints. The SDET team has also developed a mock SSP setup which bids for test inventory traffic. This end-to-end testing approach covers the entire ad serving process, from the ad server to SDK rendering and reporting. Furthermore, the automation also validates mediation-specific aspects such as callbacks, ad rendering, and custom params. By mimicking a production-like environment for publishers, this setup plays a crucial role in maintaining the high quality of the SDK throughout the entire ad ecosystem.
In summary, the PubMatic SDET team has developed an innovative automation solution to solve the unique challenges the adtech industry faces in SDK testing. Our customers’ business success using our products is testament to the value of our strategy.