Engineering leaders rarely suffer from a shortage of data. Git platforms count commits, issue trackers count tickets, CI systems count builds, observability platforms count incidents and finance teams count engineering spend. The problem is that these numbers do not automatically explain whether the organisation is becoming better at turning ideas into reliable customer value.
DORA metrics remain an excellent view of software delivery performance. They help leaders examine throughput and stability through measures such as change lead time, deployment frequency, change failure percentage, recovery performance and deployment rework. But they are not a complete definition of engineering productivity. A team can deploy frequently while developers spend hours navigating fragmented tooling. Another can have long lead times because it is solving a genuinely difficult regulatory problem. A third can improve local delivery speed while creating downstream support costs.
The right question is therefore not “Which single productivity metric should we use?” It is “What measurement system helps us improve the engineering system without rewarding harmful behaviour?”
What DORA metrics can and cannot tell a CTO
DORA measures the performance of the delivery system. It is particularly useful for identifying whether changes move through production quickly, safely and recoverably. Trends can reveal slow review queues, fragile deployment processes, poor test confidence or an operating model that forces large batches.
DORA does not directly measure whether developers understand the architecture, whether internal documentation is usable, whether a build consumes twenty minutes of attention, or whether teams are building the right product. It also cannot explain causality on its own. A drop in deployment frequency may reflect deteriorating capability, or it may reflect a deliberate platform migration that reduces long-term risk.
Use DORA as an outcome layer, not an individual scorecard. Compare services with their own historical baselines and operating context. A mobile application, a payments platform and an internal analytics pipeline should not be forced into the same target.
Build a balanced engineering productivity model
A useful executive model has five connected dimensions: delivery performance, developer experience, flow efficiency, product and quality outcomes, and organisational sustainability.
Delivery performance asks whether software reaches users predictably and safely. Developer experience asks whether engineers can understand, build, test and operate systems without unnecessary friction. Flow examines where work waits. Product and quality outcomes connect engineering activity to customer and operational value. Sustainability tests whether the pace depends on burnout, heroics or accumulating risk.
No dimension should be interpreted alone. Faster lead time accompanied by higher escaped defects is not improvement. Higher developer satisfaction with falling customer reliability requires investigation. A balanced view prevents one metric from becoming a target that distorts behaviour.
Measure developer experience through real workflows
Annual engagement surveys are too broad to diagnose engineering friction. Measure the experience of completing specific high-frequency tasks:
- Setting up a development environment
- Finding the owner and documentation for a service
- Running a representative test suite
- Getting a pull request reviewed
- Deploying a standard change
- Diagnosing a failed build or production alert
- Requesting infrastructure, access or security approval
For each workflow, combine perception and telemetry. Ask developers whether the task is clear and cognitively manageable. Measure elapsed time, active work time, failure rate and the number of handoffs. The gap between elapsed and active time often exposes waiting that dashboards miss.
Keep surveys short and recurring. Three questions about a specific workflow every month usually produce more actionable evidence than a long survey twice a year. Segment results by platform, service age and team context without exposing individuals.
Track flow efficiency, not just ticket velocity
Cycle time becomes useful when decomposed. Measure how long work spends actively progressing versus waiting for review, environments, decisions, dependencies or release windows.
A practical flow dashboard should distinguish:
- Coding and local validation time
- Pull-request pickup and review time
- CI queue and execution time
- Security or compliance approval time
- Environment provisioning time
- Release waiting time
- Rework after review, testing or production feedback
This decomposition changes the management response. If coding represents fifteen percent of elapsed lead time, demanding more developer output will not solve the problem. The constraint may be reviewer concentration, slow test infrastructure or a governance queue.
Do not equate story points with value or compare velocity across teams. Story points are planning aids calibrated locally. Turning them into performance measures encourages inflation and discourages honest discovery work.
Connect productivity to quality and customer outcomes
Engineering productivity is the efficient creation of durable value, not the production of more code. Add outcome measures appropriate to the service:
- Reliability against service-level objectives
- Escaped defect rate and severity
- Customer-reported issues after releases
- Feature adoption and task completion
- Support volume attributable to product defects
- Cost to serve a transaction or tenant
- Security findings introduced and remediated
- Time to validate or abandon a product hypothesis
Not every team directly owns a revenue metric, but every team should understand its value chain. A platform team may influence onboarding time, deployment success and service reliability. A data platform may influence reporting latency, trust and analyst cycle time. Map proximal engineering outcomes to the business outcome rather than inventing a simplistic revenue attribution.
Treat cognitive load and system complexity as measurable constraints
Developer experience deteriorates when engineers must hold too much accidental complexity in their heads. Useful signals include the number of tools required for a standard change, undocumented service dependencies, build configurations with unclear ownership, and repeated requests for tribal knowledge.
Architecture can be measured indirectly through change patterns. Look for services that frequently change together, modules with concentrated incidents, ownership gaps, excessive cross-team coordination and code areas that teams avoid modifying. These are not grounds for blaming owners; they are candidates for architectural investment.
Track the time needed for a capable engineer to make a safe first change in an unfamiliar service. This “time to confidence” is often a better indicator of maintainability than raw code volume.
Measure AI-assisted development without counting prompts
AI coding tools make activity metrics even less reliable. Lines generated, prompts submitted and acceptance rates do not prove value. AI can accelerate a local coding task while increasing review load, duplicated logic or security risk downstream.
Evaluate AI adoption at the workflow level. Compare time to complete representative tasks, review effort, defect escape, test quality, developer confidence and maintainability. Separate mechanical tasks—such as test scaffolding or API client generation—from ambiguous architectural work.
Use controlled pilots and baseline comparable teams or periods where possible. Review the complete delivery system because AI frequently acts as an amplifier: teams with clear standards and fast feedback benefit more, while teams with weak architecture and quality controls may create problems faster.
Design metrics that cannot become surveillance
Productivity data becomes dangerous when attached to individual ranking, compensation or forced comparison. Commit counts, pull-request volume, online time and AI usage can be gamed and penalise mentoring, design work and incident leadership.
Establish explicit governance:
- State the improvement question for every metric.
- Define who can access team-level and raw data.
- Prohibit individual performance ranking from operational telemetry.
- Set retention limits for granular event data.
- Let teams challenge interpretations and add context.
- Review whether a metric has changed behaviour in harmful ways.
- Remove measures that no longer drive a decision.
Trust is not a soft concern. Developers who believe metrics will be weaponised will optimise the measurement surface or avoid exposing problems. Both outcomes make the data worse.
Create a CTO scorecard with leading and lagging indicators
A practical executive scorecard can contain ten to fifteen measures, not hundreds. Include a small set across the five dimensions.
Leading indicators might include developer task success, build feedback time, review pickup time, platform adoption and documentation coverage. Lagging indicators might include delivery lead time, change failure percentage, reliability, escaped defects, customer adoption and cost to serve.
Show distributions rather than averages where possible. A median build time of five minutes can conceal a critical repository where engineers wait forty minutes. Track trends and outliers, and annotate major migrations, organisational changes and incidents.
Each metric needs an owner, a decision it supports, an expected review frequency and a documented caveat. A metric without a decision is reporting overhead.
Run a ninety-day implementation sequence
During the first thirty days, define the business outcomes, select two or three representative value streams and establish metric governance. Capture existing DORA baselines and run workflow-focused developer interviews.
During days thirty-one to sixty, instrument the major waiting states across source control, CI, deployment and service management. Launch a short developer-experience pulse focused on two painful workflows. Validate data quality with the teams who perform the work.
During days sixty-one to ninety, identify one system constraint per value stream and fund an improvement experiment. Examples include test parallelisation, automated environment provisioning, review rotation or a paved-road service template. Measure whether the intervention improves the complete workflow without damaging quality.
The first objective is not a perfect dashboard. It is a repeatable connection between evidence, investment and learning.
Common measurement mistakes
The most damaging mistake is declaring a universal benchmark without considering architecture, risk or product stage. Other failures include measuring individuals, rewarding volume, hiding definitions, ignoring developer feedback, and collecting data without changing investment decisions.
Avoid metric theatre. If leaders celebrate deployment frequency but repeatedly defer test infrastructure, teams learn that the dashboard matters more than the system. Publish the actions taken because of the evidence.
Questions CTOs should ask every quarter
- Where does valuable work spend the most time waiting?
- Which developer workflow creates the greatest cognitive or operational burden?
- Are delivery improvements accompanied by equal or better quality?
- Which platform capabilities are actually adopted and retained?
- Is AI reducing end-to-end task time or merely shifting effort to review?
- What metric is most likely to be gamed or misunderstood?
- Which engineering investment produced measurable system improvement?
Final recommendation
Keep DORA, but place it inside a broader measurement system. Combine delivery outcomes with developer experience, flow, quality, customer value and sustainability. Measure teams and systems rather than individuals. Use data to choose and evaluate improvements, not to manufacture certainty.
The highest-performing engineering organisations do not ask developers to generate more visible activity. They continuously remove friction, shorten feedback loops, improve architectural clarity and make the safe path the easy path. That is the productivity system a CTO should measure—and the one worth investing in.
Sources and further reading
DORA research and measurement frameworks
DORA platform engineering capability guidance
SPACE framework paper
Executive diagnostic workshop
Run a ninety-minute workshop with engineering, product and platform leaders. Select one value stream and map its path from approved idea to measurable customer outcome. Mark every queue, handoff, automation boundary and feedback loop. Then identify which current metrics illuminate each stage and which merely count activity.
Ask teams to bring three examples: a routine change, an urgent defect and a complex cross-team change. Comparing these paths exposes whether the measurement system represents normal work only or can explain high-impact exceptions. The output should be a small hypothesis backlog—for example, “review pickup time is the primary constraint for the payments team”—with an owner and a way to test it.
Example scorecard design
A balanced quarterly scorecard could pair five delivery measures with five experience and outcome measures. Delivery measures might include change lead-time distribution, deployment frequency by service class, change failure percentage, recovery performance and deployment rework. Experience measures might include build feedback time, review pickup time, task-success rate for provisioning, time to first safe change and developer satisfaction with the paved road.
Add two business-linked measures selected by the team, such as incident-related support contacts and feature adoption. Annotate migrations and policy changes. The scorecard should show whether an intervention changed the system, not declare whether a team is “productive.”
Data architecture and definitions
Create a semantic layer rather than joining tool events directly in every dashboard. Define what constitutes a change, deployment, failure, recovery and service. Map repositories to services and services to owning teams with effective dates so reorganisations do not corrupt history.
Preserve raw events separately from derived measures. Publish definitions and known limitations beside each chart. Monitor missing events, clock differences and integration failures. A precise-looking metric built from incomplete deployment data is more dangerous than an openly provisional measure.
Investment decisions the system should support
The measurement system should help leaders compare interventions: additional reviewers, faster CI, platform templates, architectural modularisation, test-environment capacity or improved incident tooling. Establish the expected mechanism before funding the intervention, then examine leading and lagging signals.
Do not demand immediate financial attribution for foundational work. Instead, connect reduced workflow time and failure to capacity, risk and customer outcomes. Over several quarters, this creates an evidence base for engineering investment without pretending that a single dashboard can calculate the value of every line of work.
