QA metrics should help the team decide what to do next. A useful report shows quality risk, testing progress, delivery confidence and change over time without using numbers to judge individuals.
On this practical guide
Principles for useful QA metrics
- Start with a question or decision, not available data.
- Show trends and context rather than one isolated number.
- Segment by severity, feature, environment or cause when useful.
- Pair speed with stability and quantity with impact.
- Define each metric, source, owner and update frequency.
- Use the same counting rules across reporting periods.
- Explain major changes and data limitations.
- Never rank individual testers or developers by bugs or test counts.
Good question: “Is release risk decreasing?” Better signals include critical risk coverage, open severe defects, regression stability and production incidents—not the number of cases written.
Core metrics for a team dashboard
| Metric | What it answers | Useful view |
|---|---|---|
| Scope testing status | What is passed, failed, blocked or not yet tested? | Count and percentage by story or risk. |
| Critical-risk coverage | Have the most important risks been tested? | Covered / total critical risks. |
| Open defects by severity | What known product risk remains? | Current count plus sprint trend. |
| Escaped defects | Which defects reached users or a later stage? | By impact, source and cause. |
| Regression result | Is existing behaviour stable? | Pass, fail, blocked and not run. |
| Automated test reliability | Can the team trust pipeline results? | First-run pass and flaky-test rate. |
| Feedback time | How quickly does a change receive useful test feedback? | Median and p90 pipeline duration. |
| Release readiness | Can the team release with understood risk? | Criteria met plus residual risks. |
Test progress and coverage
Test execution rate
Execution rate = Executed tests / Planned tests × 100
Pass rate = Passed tests / Executed tests × 100
Always report failed, blocked and not-run tests. A high pass rate can hide large untested scope.
Risk or requirement coverage
Risk coverage = Risks with sufficient evidence / Risks identified × 100
Requirement coverage = Requirements with tests / Requirements in scope × 100
Coverage means a test exists or ran. It does not prove the requirement works correctly. Add execution result and risk strength.
Automation coverage
Automation coverage =
Automated eligible scenarios / Scenarios selected as automation candidates × 100
Define the denominator. “80% automated” is meaningless if it mixes unsuitable manual tests, low-value cases and different test levels.
Defect metrics
| Metric | Use | Caution |
|---|---|---|
| Open defects by severity | Show known current risk. | Include age, owner and affected area. |
| New vs resolved defects | Show whether backlog is growing. | One sprint can contain discovery bursts. |
| Defect ageing | Reveal severe defects waiting too long. | Measure business days consistently. |
| Reopen rate | Show incomplete fixes or misunderstood criteria. | Review causes rather than blaming people. |
| Escaped defects | Find gaps in prevention and detection. | Classify by impact, not only count. |
| Defect detection effectiveness | Compare defects found before and after release. | Needs a stable observation period. |
Defect detection effectiveness =
Pre-release defects /
(Pre-release defects + Post-release defects) × 100
Use defect root-cause categories such as unclear requirement, code logic, integration, test gap, environment, data or deployment. Review patterns and prevention actions.
Automation and CI metrics
| Metric | Recommended calculation |
|---|---|
| First-run pass rate | Tests passing before retry / tests executed. |
| Flaky-test rate | Tests with inconsistent outcomes / tests executed. |
| Pipeline success rate | Successful pipelines / completed pipelines. |
| Pipeline duration | Median and p90 time from start to feedback. |
| Failure investigation time | Time from failed job to classification or owner. |
| Automated regression duration | Time for the agreed regression suite. |
| Maintenance load | Time spent repairing tests / automation effort. |
Separate failure causes: product defect, test defect, environment, data and infrastructure failures require different actions.
Delivery and production metrics
QA information becomes stronger when combined with delivery and production outcomes.
- Change lead time: time for a change to reach production.
- Deployment frequency: how often successful production deployments occur.
- Change failure rate: deployments that cause degradation and require remediation.
- Failed deployment recovery time: time to restore service after a failed change.
- Deployment rework rate: deployment work spent correcting earlier changes.
- Production incident count and impact: user-visible quality outcome.
- Rollback or hotfix rate: release stability signal.
DORA currently groups software delivery metrics into throughput and instability. Use them at application or team level and as prompts for improvement, not as individual performance targets.
Suggested reporting cadence
| Cadence | Audience | Content |
|---|---|---|
| Daily | Delivery team | Blockers, failed pipeline, severe defects and immediate risks. |
| Sprint | Scrum team and Product Owner | Scope status, defects, regression, automation reliability and trend. |
| Release | Release decision-makers | Criteria, evidence, open defects, deferred scope and residual risks. |
| Monthly / quarterly | Engineering and product leadership | Escapes, delivery stability, quality trends and improvement actions. |
| Incident review | Relevant teams | Impact, detection, recovery, root cause and prevention actions. |
A compact QA dashboard
Keep the first view small. Let users drill down into detail.
Release confidence
Ready, at risk or not ready, with the top reason.
Scope status
Passed, failed, blocked and not tested.
Defect risk
Open Critical/High defects and ageing.
Regression
Latest result and trend across recent runs.
Automation health
First-run pass, flaky tests and pipeline feedback time.
Production quality
Escapes, incidents, hotfixes and recovery.
Show evolution
- Use the same weekly or sprint boundary.
- Show at least several comparable periods.
- Add an annotation for releases, migrations and major scope changes.
- Show the current value, previous value and direction.
- Add a short explanation and action beside significant movement.
Copy-ready sprint QA update template
# QA Update — Sprint [Number / Name]
Period: [Start date] to [End date]
Build / environment: [Details]
Overall confidence: [Green / Amber / Red]
Summary: [One or two direct sentences]
## 1. Scope status
| Status | Current | Previous sprint | Change |
|--------|---------|-----------------|--------|
| Passed | [#] | [#] | [+/-#] |
| Failed | [#] | [#] | [+/-#] |
| Blocked | [#] | [#] | [+/-#] |
| Not tested | [#] | [#] | [+/-#] |
Critical-risk coverage: [Covered / Total]
Stories accepted: [# / total]
Stories still in QA: [#]
Deferred scope: [Items and reason]
## 2. Defects
| Severity | Open | New | Resolved | Previous open |
|----------|------|-----|----------|---------------|
| Critical | [#] | [#] | [#] | [#] |
| High | [#] | [#] | [#] | [#] |
| Medium | [#] | [#] | [#] | [#] |
| Low | [#] | [#] | [#] | [#] |
Oldest Critical/High defect: [Age and link]
Reopened defects: [#]
Escaped defects: [# and impact]
Main defect trend: [Area / root cause / explanation]
## 3. Regression
Latest smoke: [Pass/Fail — date and link]
Focused regression: [Result]
Full regression: [Passed / Failed / Blocked / Not run]
Key failures: [Links and impact]
## 4. Automation and CI
Automated candidates completed: [# planned / # completed]
First-run automated pass rate: [%]
Flaky-test rate: [%]
Pipeline success rate: [%]
Median / p90 feedback time: [Duration]
Main failure causes: [Product / test / environment / data / infrastructure]
## 5. Non-functional quality
Accessibility: [Result / findings]
Performance: [Key threshold and result]
Security: [Result / findings]
Compatibility / visual: [Result]
## 6. Risks and blockers
| Risk / blocker | Impact | Owner | Action | Target date |
|----------------|--------|-------|--------|-------------|
| [Item] | [Impact] | [Owner] | [Action] | [Date] |
## 7. Trend and interpretation
Improved: [Metric and reason]
Declined: [Metric and reason]
Unchanged but concerning: [Metric and reason]
## 8. Actions for next sprint
- [Action, owner and expected result]
- [Action, owner and expected result]
## 9. Decisions needed
- [Decision, owner and deadline]
Evidence:
- Test report: [Link]
- Defect dashboard: [Link]
- Regression run: [Link]
- Quality dashboard: [Link]
Release quality summary template
Release: [Version]
Overall recommendation: [Ready / Ready with risk / Not ready]
Evidence:
- Critical-risk coverage: [Value]
- Regression: [Result]
- Open defects: [Critical #, High #]
- Automated reliability: [First-run pass and flake rate]
- Non-functional checks: [Results]
- Production readiness: [Monitoring and rollback]
Residual risks:
1. [Risk, impact, mitigation and owner]
Decision required:
[Who must decide what and by when]
Metrics to avoid or qualify
| Metric | Why it misleads alone |
|---|---|
| Number of test cases | Rewards volume rather than risk coverage or effectiveness. |
| Number of bugs per tester | Encourages competition, duplicates and poor behaviour. |
| Automation percentage | Can hide low-value, flaky or incorrectly chosen tests. |
| Pass rate | Can exclude blocked and unexecuted scope. |
| Code coverage | Shows executed code, not assertion quality or missing risks. |
| Zero production defects | May reflect poor reporting or low usage rather than quality. |
Useful links
- ISTQB Foundation Level ↗ — monitoring, control and test reporting.
- DORA software delivery metrics ↗ — throughput and instability.
- GitLab test reports ↗ — expose results in pipelines and merge requests.
- GitLab coverage reporting ↗ — coverage trends and diff annotations.
- Playwright reporters ↗ — publish automated results and artefacts.