Advances in the artificial intelligence (AI) and machine learning (ML) fields have revolutionized software development and testing. At PubMatic, we embrace this evolution and are constantly working on ways to enhance our technology for internal, customer, and partner benefit. Negative testing is an integral part of software quality validation, as unexpected events outside of the happy path could impact a system and stability.
Recently, we built a bot that can identify and automate most negative test cases in a few seconds. We do not believe there is any other such exact bot currently available, and we think this could help solve the industry-wide challenge of negative testing coverage.
Problems with Traditional Automation Strategies for Identifying Negative Test Cases
As part of negative testing, covering all possible JSON combinations for a complex REST API is sometimes time-consuming. Releasing a hotfix or business priority feature might have strict deadlines and could leave little time to cover all negative cases related to JSON combinations and other API parameters. It is possible to miss a few of the negative scenarios when there is less time for the QA cycle.
As a solution, we have built an in-house bot that identifies and automates most of the negative scenarios for a REST API.
Here are a few of the pain points we were looking to solve:
- Manual test cases identification
– Test cases identification is manual and time-consuming
– As it requires manual work, there is a chance of missing a test case - Manual test cases documentation in Excel
– Excel is widely used for documentation of test cases
– This process can also be manual and time-consuming - Addition of test cases in automation framework
– Automating test cases require a QA automation skill set and can take a huge amount of time if the API to be automated accepts complex JSON - API Test data creation for complex JSON objects
– With the growth of microservices and machine learning algorithms, API test data creation time has increased due to the complexity of JSON objects
NeTBot Overview
We wanted to develop a solution that will solve problems in less time, and our team built a solution, NeTBot, as part of the PubMatic Hackathon 2020.
At PubMatic, we have thousands of API endpoints to keep our system up and running smoothly. Any change in the API has to undergo integration testing before going to production. To protect platform stability, we wanted an API automation bot that could identify and build most of the negative scenarios into a Ready to Execute automation test suite.
We have seen a significant increase in automation coverage after adopting NeTBot.
NeTBot Architecture
JSON Combinations Generator
The JSON Combinations Generator is the core component of NeTBot, which takes API details as input with valid parameters. It scans through the valid API request details and generates all possible JSON combinations in a list if the request body is present in the input. It also generates a list of boolean values, which stores whether the generated JSON is Valid or Invalid. Once all the JSON combinations are generated, it returns both the lists to our OutputStream Writer to write test cases into the Excel sheet and write the ready to execute automation test suite.
OutputStream Writer
OutputStream Writer takes all the JSON combinations transferred to it and writes all those combinations to an Excel sheet to maintain our test cases. It also generates ready to execute automation test suite using those combinations. While generating a ready to execute automation test suite, it generates negative test cases related to QueryParams, headers and invalid requests. This makes it a generalized bot for all types of APIs.
Key Benefits of NeTBot
- Reduction in production issues due to negative test cases.
- Increased negative test automation coverage
- Reduction in time to identify and write test cases
- Reduction in time to write automation code
- No manual intervention is required for writing and automating negative test scenarios
NeTBot Benchmarking:
NeTBot generates thousands of test cases and lines of code within seconds.
Our customers and partners benefit from these improvements with shorter time-to-market, improved product quality, and better coverage of business cases in the QA phase.