The Nuances of Measuring Quality

• Sun, Aug 9, 2026
• 5-minute read

Companies love dashboards to measure quality. we often build dashboards full of metrics and assume the numbers tell the story. But metrics are usually windows into reality, not reality itself. The challenge is deciding whether a measure is showing the truth—or merely a slice of it.

The single indicator is only a proxy.

The software industry has been searching for perfect measures of quality since the first 1 or 0 was produced, the first spec was writen, the first code was compiled, and the first dollar was made. Phrases like zero defects, scalable performance, and churn reduction are all working to be part of the quality measurement, whether or not that is realistic. Then we try to put this into a pristine number, I think it is 42 to be honest.

Quality is not something we measure. Quality is something we discover through inquiry.

Using this definition, it should be evident that quality is nuanced at best. At worst, it is used to determine relative performance or unhealthy productivity conversations.

Examining pull request quality - A lesson in metrics

As any seasoned (in this case it is the poor metric of salt & pepper hair) engineer knows, pull request measurement is s slippery slope. Let’s assume for a minute the results of PR Quality measurement is put on the dashboard as that single number, which is now being read by management. Mikey had several PRs that had a lower score, and managers evaluated performance by that trend as a key indicator.

• The dashboard was accurate.
• The conclusion was wrong.
• Those changes of those PRs however were critical to platform stability and resiliency, even if they were 1 liners with no tests or documentation that was rolled up in the quality score.

Maybe there should more indicators?

Breaking Down PR Quality. What Does That Even Mean?

The first trap of measuring quality is treating a complex concept as if it were a single thing.

“PR Quality” sounds singular, but it is actually a collection of attributes.

Here are some categories that may have multiple measures underneath which refer to PR auality

• Correctness
• Clarity
• Test Coverage
• Maintainability
• Risk
• Business Value

Wait! Even those categories don’t make a lot of sense without context. What would go under those categories? Pretty soon, we could be in analysis paralysis for what needs to be measured.

Even the categories of metrics are used as proxies

Weighting the categories

Here comes the second trap. Let’s weight the categories above to develop the one singular metric. Should correctness be rated higher than risk? Should I make test coverage more important than maintainability? Clarity must be the top area, but is it 40% of the total or 70%? How about config only changes? How do those even fit in these categories?

Not so simple is it? 5 categories might be comprised of 50+ metrics, aggregated and then grouped. Then those metrics have weights inside the category. Mountains of subjective reasoning that doesn’t rely on science, but rather gut.

Visualizing metrics: The Dashboard Problem

One of the illusions of the “single measurement” is how it is visualized. Many times this is put on some kind of report/dashboard as a source of finality or truth. Few truly know the calculation behind this metric, but just use the scale to evaluate the worth of that metric. It no longer becomes an indicator, but rather fact, until proven otherwise. After all, smart people use these dashboards to make decisions.

If the dashboard could talk it would say:

I can measure it, that means you should understand it. Don’t ask me about nuance

Look at the comparison of two developers PR Quality, both being measured with the same quality metric. Let’s assume there is a composite quality score per developer:

Which one has the higher quality?

• The dashboard says A.
• Context might say B.

None of below items are quality measurements but rather indicators. If these were put on a dashboard what would the messaging be?

• Lines of code - Wow, 3 million loc changes = rearranging methods in the code base in alphabetical order.
• Number of PRs - Amazing productivity, more than 10 a day = Single line PRs for configurations
• Review turnaround time - 90% in 1 hour = Rubber stamping
• Story points - Ratio of 1 SP to PR = Are they really shippable features or just tweaks?
• Deployments - 20 PRs per this deployment = Monthly cadence cycle

The more we compress reality into a score, the more assumptions we hide.

Reframing a quality score

instead of asking “What is the quality score?” maybe we should be asking “What windows do I need to understand productivity?”

Think of a window as a snapshot or point in time reflections where questions can be better indicators of value.

Since we have been talking about PRs, here are some questions maybe that could be asked?

1. Commitments vs outcomes: Did we do what we said we were going to do?
2. Alignment to goals: Did the work in this PR get us closer to our goals?
3. Learning: What did we learn from doing the work?
4. Knowledge sharing/Communication: Is the PR details intuitive enough for developers after the fact?
5. Collaboration quality: Who was part of the review? Is there appropriate engagement over rubber stamps?
6. Execution / Work Completed: Was there enough time to produce the outcomes? Did the work meet the scoped commitments

Additional thoughts on building a metric

• There is usually not one definitive metric. Use multiple metrics to ask better questions.
• Treat metrics as signals not finality.
• Prefer trend lines over rankings for visual comparison
• Measure outcomes more than activity.
• Dashboards are useful. But the most useful dashboards are the ones that remind us they are merely windows—and encourage us to keep looking outside.

A metric is not a number waiting to be discovered. They are realities we are trying to understand.

Where this aligns with the Places framework

• Planning emphasizes selecting measures that support better decisions rather than merely collecting data.
• Learning reinforces that metrics should improve understanding and adaptation.
• Execution focuses on outcomes and effectiveness, not activity in isolation.