Advancement in the artificial intelligence (AI) and machine learning (ML) fields has revolutionized software development and testing. At PubMatic, we embrace this evolution and are constantly working on ways to improve our technology for internal and partner benefits.
Recently, our team built a prototype QA automation bot to stay at the forefront of automated testing and to deliver faster, high-quality features to our clients.
Our innovative journey in using automation bots started with a review of traditional strategies to improve upon, then development of API and UI bots which resulted in key benefits for our internal teams and partners.
Problems with Traditional Automation Strategies
While useful, traditional automation strategies present unique challenges including slower automation adoption at scale. Here are a few of the pain points we were looking to solve for:
- Manual Code Writing
- Automation code writing is still manual, time-consuming and requires technical expertise.
- Complex API Test Data Creation
- With the growth of micro-services, API test data creation time has increased due to the complexity of JSON objects.
- Maintenance Overhead
- With frequently changing products in an agile environment, automation code maintenance is time-intensive and requires financial support.
- Slow Coverage of the Automation Backlog
- There is no easy way to cover automation backlogs. It’s slow, especially with thousands of test cases and it takes time to realize ROI on your automation investment.
- Domain Expertise Required
- Business test scenarios need domain expertise so you can QA realistic uses.
QA Automation Bot Design and Development
The QA group at PubMatic developed a beta project to create QA automation bots. We wanted to address the key problem areas with traditional automation strategies (listed above) and improve both internal and client experiences.
Our objective was to build an AI-powered automation bot which could maintain our existing automation suites while simultaneously looking at the existing code repository to determine mapping between different components. The ideal result would be auto-generated automation code in the desired programming language.
Automation Bots
We worked on two categories of automation bots: API and UI. The API bots, in particular were important for PubMatic. The number of APIs developed and consumed are increasing exponentially due to the wide adoption of the micro-service architecture.
At PubMatic, we have more than 3,500 API end points to keep our system up and running smoothly. Any change in the API has to undergo integration testing before going to production. To protect platform stability, we needed an API automation bot that could build regression packs based on production usage patterns of our platform and then build automation suites once manual testing began.
Creating the API Automation Bot
PubMatic Network Layer Transformer
The network layer transformer we use is in-house software which monitors all API requests and responses going through the cluster of micro-services. The primary job of this layer is to associate unique identifiers to each API request and its corresponding response. From there it will create feed files to be later consumed by the core engine of the API bot.
Core Engine of the API Bot
Our API bot core engine scans several GBs of transformed log files and uses machine learning algorithms to select test data from log files and generate new combinations to cover additional business cases.
Creating the UI Automation Bot
Using customer feedback, we continue to make changes to our UI to enhance the user experience and improve speeds. This requires frequent changes to code and continuous testing. Our main objective for the UI automation bot we built was that it would be able to visit and scan all PubMatic UI pages and generate page object class files in the desired programming language (i.e. Python, Java, etc.).
Core Engine of the UI Bot
PubMatic has developed a custom library of UI core components; these components act as the building blocks of our pages. The core engine of our UI bot runs on an AI algorithm called “Mapper” which scans through the page’s document object model (DOM) and maps DOM elements to UI core components. The bot then automatically generates the automation code in the desired programming language.
As a result of our QA automation blots, the PubMatic team experienced the following benefits:
- Reduction in the Time to Write Code
- Our tests show that the time to write automation code decreased by 60%.
- Simplified Automation Maintenance
- Teams can now support multiple UI revamps because they only need to re-run the UI Automation Bot to generate new code.
- Accelerated Automation Coverage
- Automation backlog coverage no longer needs manual intervention to write the code.
- Our tests showed a 75% reduction in the time taken for API Automation.
- Future-Enabled Automation
- Given changes can now be automated, we are better prepared for future updates.
Last but not least, our partners benefit from these improvements with shorter time-to-market, improved product quality, and better coverage of business cases in the QA phase.
What’s Next?
PubMatic is passionate about innovating to improve efficiency and user experience. To learn more about what we are developing, check out our latest posts or contact us.