By Emilio Salvador, vice president of strategy and developer relations at GitLab

Without a clear understanding of the opportunity for productivity improvements, adding AI into the software development process will at best lead to marginal incremental value to an organization. By strategically identifying priority areas to incorporate AI based on data uncovered by Task, Time, and Team metrics, organizations can begin to harness AI's promise of software development productivity gains.

Organizations are understandably excited about the potential for AI coding assistants to increase the productivity of software developers. In a recent survey by GitLab, 83% of respondents said it is essential to implement AI in their software development processes to avoid falling behind. 

The promise that AI can automate most software development or at least make developers significantly more productive raises the question of how developer productivity is measured. Hint: it's not about code volume.

While most measurements of developer productivity focus on task metrics such as lines of code produced or the number of code commits, developer productivity cannot be measured in a single dimension. Multiple dimensions of developer performance must be considered to accurately measure developer productivity.

Task - a deceptive measurement

The task dimension covers the most direct measure of individual developer output - efficiently completing an assigned task within a set timeframe. While seemingly straightforward, measuring task completion as an indication of productivity can be misguided, for two reasons.

First, software developers typically work in teams, with individuals working on different aspects of software development simultaneously. Therefore, measuring individual developers using the same task-oriented metrics provides an inaccurate representation of team productivity. 

Second, completing more tasks isn't necessarily the smartest way to solve a problem. For example, less skilled programmers or AI coding assistants can produce large amounts of boilerplate code. For example, a recent vendor study showed that developers could finish coding a web server in JavaScript 90 minutes faster using AI than without it. It was an impressive result, but they could have reused an existing web-server abstraction to create the same outcome in minutes.

Problem-solving is a critical skill for any software developer because they regularly face complex challenges that require innovative solutions. Developers must be able to accurately identify problems, collaborate with team members to generate potential solutions, and ultimately choose the best course of action to address the issue. 

Individual developer efficiency, which requires a combination of technical skills, critical thinking, and creativity, is a more optimal metric than number of tasks completed. In one famous example, Bill Atkinson, the principal designer and developer of the graphical user interface of Apple's Lisa computer, optimized Quickdraw's region calculation routine to be six times faster and 2,000 lines of code shorter. His contribution defied Apple's productivity metric of tracking individual engineers by the amount of code they wrote each week in favor of a simpler, faster, and more efficient solution.

Integrating AI into the software development process to augment developer efficiencies, such as via test automation, code refactoring, and documentation, can free up developers' time to take on more complex work that will deliver much greater value for organizations and end users. Therefore, now more than ever, measuring output, such as lines of code, is irrelevant and we should focus on measuring business outcomes.

Time - the velocity measurement

Software development teams know that iterative development practices outperform traditional waterfall models. High cadence leads to tight feedback loops where improvements made in one cycle, such as an abstraction to replace boilerplate code, lead to further improvements in later cycles.

Time metrics are an objective measurement. In Google's framework for measuring developer productivity, the DevOps Research and Assessment (DORA) rubric, three of the four metrics are directly time-based:

• Deployment frequency (hours/days/weeks/months)
• Lead time for changes
• Time to restore service

DORA metrics measure the team's performance over a specific timeframe. They serve as key indicators of a team's agility, operational efficiency, and velocity and reflect how well an organization balances speed and software stability.

More than six years of research by the DORA team has established that these metrics can be used to identify exactly where and how to improve, with short iteration cycles consistently leading to better outcomes for development organizations.

DORA metrics can also reveal where poor developer experiences may be hurting developer velocity. For example, a development team might be producing great code, but showing terrible DORA metrics because of an error-prone testing and deployment process. 

The solution isn't asking a talented but over-burdened development team to work harder; it's a call for development team leaders to take a holistic view of the development pipeline and identify where process improvements can be implemented. For example, introducing AI and automation into the code review and testing process can improve DORA metrics and software delivery performance.

Team - the human-centered measurement

The team dimension considers business outcomes in a wider organizational context. While software development teams must work efficiently together, they must also work with teams across other business units. 

Often, non-technical factors, such as peer support, working environment, psychological safety, and job enthusiasm, play a significant role in boosting productivity. 

Another framework is SPACE, which is an acronym for satisfaction, performance, activity, communication, and efficiency. SPACE was developed to capture some of the more nuanced and human-centered dimensions of productivity. SPACE metrics, in combination with DORA metrics, can fill in the productivity measurement gaps by correlating productivity metrics to business outcomes.

McKinsey found that combining DORA and SPACE metrics with "opportunity-focused" metrics can produce a well-rounded view of developer productivity. That, in turn, can lead to positive outcomes, as McKinsey reports: 20% to 30% reduction in customer-reported product defects, 20% improvement in employee experience scores, and 60% improvement in customer satisfaction ratings. 

What are you trying to optimize?

Just like the three dimensions of a cube all contribute to its volume, the Task, Time, and Team dimensions of developer productivity all contribute to total business impact.

While developers may focus on the most easily measured Task dimensions of their productivity, such as lines of code written per day or merge requests accepted per week, those metrics must be considered in the larger business context, such as when more lines of code may negatively impact the Team and Time dimensions. 

The importance of engineering talent to a company's success and the fierce competition for developer talent underscores the need to acknowledge that software development requires measurement to improve.

##

ABOUT THE AUTHOR