Introduction
In early 2023, we recognized the transformative potential of Generative AI (GenAI). Our engineers started exploring and monitoring this trend, conducting small-scale experiments with different Large Language Model (LLM) to define high-impact use cases for our company. Within two months, the number of engineers in the local GenAI Slack group grew from 0 to 280, representing more than half of the engineering team. The CTO team released the first draft of GenAI usage policies for responsible usage to protect PubMatic’s IP (Intellectual Property) and data privacy.
Today, we have legal contracts in place with GitHub Copilot and Bito, and we have GenAI-specific task forces working on specific projects. One of the highest impact areas for PubMatic in GenAI is increasing engineering productivity, as it will have a multi-fold high business impact by increasing the speed of software development and testing.
Going Deeper
Earlier this year, we started evaluating, educating, and scaling GenAI use cases for developer productivity. Our teams started learning the tools and using them on production code. We set a goal of 30% adoption of GenAI tools for the first quarter of this year.
We created programming language-specific task forces for GenAI to address language-specific challenges around prompt engineering, addressing hallucinations, evaluating the accuracy of code suggestions, etc. These task forces cut across multiple teams and multiple use cases, covering languages such as Java, Go, C++, Python, and others. These teams are also the GenAI thought leaders at PubMatic.
Our task force engineers evaluated multiple use cases and identified the key high-productivity use cases for PubMatic, which fall into two categories: software development and software testing.
Software development has three specific use cases that we found most valuable:
- Code generation
- Unit testing
- Code review
Software testing is also greatly benefited by these tools, in three use cases:
- Automation of code generation
- Test case generation
- Automation of code review
One project in our Private Marketplaces product development roster was accelerated by 80% through faster test case writing. In many other projects, we have seen between 25-50% project acceleration. We have seen that in some modules, around 80% of the unit tests generated from GenAI have been accepted by the human team members.
We have found GenAI tools to be particularly useful for re-architecting modules, moving modules over to advanced languages, and migrating to new libraries, such as a new rendering JavaScript library. Multiple projects have benefited from this.
GenAI Tools Used
Our task forces compared several popular tools and made observations based on tens of thousands of lines of code generated and many hundreds of test cases written.
Our team has found that GitHub Copilot is preferred by most engineers for code generation and generating unit test cases. It understands code context well and gets better as the user uses it frequently. We have seen that 20-30% of the code generated by Copilot is accepted by our engineers. We have used very specialized custom prompts to ensure that the test cases cover all corner cases and address data inconsistencies.
Bito.ai is performing well as a code review agent. The code review comments have generally been accepted by our engineers. Also, for Angular codebase, Bito.ai is doing well for code suggestions.
Our team also used ChatGPT, and feedback from some engineers is that they still use ChatGPT for small script writing.
PubMatic ROI (Return on Investment) Metric
To measure the return of investment of developer productivity on our business outcomes, our in-house team has built an advanced team-level real-time dashboard for each product. These metrics will help us track the adoption of GenAI at each task level and will help in showcasing the actual timesaving in executing each of these projects.
The first half of the year (H1) has been very successful for the Engineering team with the evaluation, education, and actual execution of GenAI use cases. We are seeing engineering productivity increase by 35%, and more than 80,000 lines of code have been written using GenAI tools.
