AI is changing QA from simple code completion into workflows that can understand requirements, inspect applications, use tools and propose test changes. The strongest approach is still human-led: AI accelerates the work while testers control scope, evidence and release decisions.
Where AI helps today
| Approach | Practical QA use | Maturity |
|---|---|---|
| AI assistant | Explain code, draft tests, generate data, summarise failures and improve bug reports. | Use now, with review. |
| Repository-aware assistant | Use project code, conventions and test history to produce more relevant suggestions. | Use now, with access controls. |
| Browser or test agent | Explore a UI, create a plan, run flows, collect evidence and propose automation. | Pilot on controlled environments. |
| MCP-connected agent | Use approved tools such as Playwright, Jira, GitHub, logs or databases through a standard interface. | Use selectively with strict permissions. |
| AI code review | Review test changes, missing cases, unsafe waits, weak assertions and maintainability problems. | Useful as an extra reviewer. |
| Predictive analytics | Prioritise risky changes, tests or components using defect and change history. | Useful when data quality is good. |
MCP for QA
Model Context Protocol (MCP) is an open standard that lets AI applications connect to external data and tools. An MCP server can expose:
- Tools: actions such as opening a browser, querying a test run or creating a ticket.
- Resources: readable context such as requirements, logs, API specifications or test results.
- Prompts: reusable instructions for common workflows.
Example QA flow
- The tester asks an agent to analyse a new checkout requirement.
- The agent reads the requirement and repository guidance.
- Through Playwright MCP, it explores the test environment using the accessibility tree.
- It creates a proposed test plan and identifies missing coverage.
- It generates or updates tests on a separate branch.
- CI runs the real tests. A person reviews the code, trace and results.
Useful MCP connections for QA
Explore pages, perform actions, inspect accessible structure and capture evidence.
Read acceptance criteria, compare existing bugs and draft reports.
Inspect diffs, find affected tests and prepare a reviewed pull request.
Query logs, traces and metrics around a failed test.
Read OpenAPI contracts and propose positive, boundary and negative tests.
Find existing cases, coverage and recent execution results.
Agentic testing
An AI assistant answers a prompt. An agent can plan several steps, call tools, observe results and adjust its next action.
Playwright currently provides specialised test agents for three jobs:
- Planner: explores the application and produces a test plan.
- Generator: turns the plan into Playwright tests.
- Healer: runs tests and proposes repairs for failures.
Agentic workflows are useful for discovery, repetitive setup, coverage suggestions and failure investigation. They should not silently change expected behaviour or weaken assertions just to make a test pass.
Good tasks for an agent
- Explore a defined feature and draft happy-path and negative scenarios.
- Convert an approved manual case into a first automation draft.
- Reproduce a failure and collect console, network, trace and screenshot evidence.
- Compare a code diff with existing test coverage.
- Classify failed tests by likely product, test, data or environment cause.
Other current approaches
Generate Gherkin, API examples or automation drafts from reviewed requirements.
Detect layout, spacing or rendering differences that DOM assertions may miss.
Suggest a replacement when the UI changes. Review every repair to avoid targeting the wrong control.
Create realistic boundary, multilingual and privacy-safe datasets. Validate constraints and bias.
Summarise traces and logs, group similar failures and suggest the most likely cause.
Use code changes and history to recommend which tests should run first.
Retrieve current requirements, standards and past defects before the model answers.
Flag missing assertions, flaky patterns, secrets and risky test changes before human review.
Testing AI systems
When the product contains an LLM or agent, traditional exact-output assertions are often insufficient. Build an evaluation set with representative inputs and expected qualities.
| Area | What to verify |
|---|---|
| Task quality | Correctness, completeness, relevance and required format. |
| Grounding | Claims are supported by the supplied source and citations point to the right content. |
| Safety | Prompt injection, harmful requests, data leakage and unsafe tool use. |
| Reliability | Repeated runs, model changes, long context, unavailable tools and recovery. |
| Fairness | Comparable behaviour across relevant languages, groups and input variations. |
| Operations | Latency, token usage, cost, rate limits, audit trail and fallback. |
- Use deterministic checks for schemas, tool calls, permissions and known facts.
- Use scored rubrics for qualities that need judgement.
- Use human review for high-risk or ambiguous cases.
- Keep a regression dataset and rerun it when prompts, tools, data or models change.
- Calibrate any LLM-as-a-judge against human decisions. Do not assume the judge is correct.
Practical adoption workflow
- Choose one narrow pain point. Example: draft Playwright tests from approved acceptance criteria.
- Define the baseline. Measure current time, review effort, defects found and flaky-test rate.
- Limit context and permissions. Provide only the files and tools needed.
- Require a structured output. Ask for assumptions, scenarios, code and evidence separately.
- Run normal quality gates. Lint, type-check, execute tests and inspect traces.
- Review like external code. Check logic, data, assertions, security and maintainability.
- Measure the result. Keep the approach only if it improves useful outcomes, not just generated volume.
Example prompt
Review the checkout acceptance criteria and the changed files.
Propose:
1. High-risk scenarios
2. Missing negative and boundary tests
3. Playwright tests using existing fixtures and locator conventions
4. Required test data
5. Assumptions that need human confirmation
Do not modify production files.
Do not weaken existing assertions.
Run the relevant tests and report evidence for each result.
Risks and controls
| Risk | Control |
|---|---|
| Incorrect or invented output | Ground with current sources, execute the result and require evidence. |
| Weak generated tests | Mutation test or deliberately break the feature and confirm the test fails. |
| Secret or personal-data exposure | Use approved models, redaction, minimal context and retention controls. |
| Prompt injection through pages or tickets | Treat retrieved text as untrusted data and restrict tool permissions. |
| Unsafe autonomous actions | Use sandbox environments, read-only defaults, approval gates and audit logs. |
| False self-healing | Review locator changes and confirm the original business assertion remains intact. |
| Vendor or model drift | Pin versions where possible and run a stable evaluation suite after changes. |
| Automation bias | Keep release accountability with qualified people. |
Useful tools and links
- Model Context Protocol introduction ↗ — how AI applications connect to tools and data.
- MCP specification ↗ — authoritative protocol reference.
- Playwright MCP ↗ — browser automation for AI agents.
- Playwright Test Agents ↗ — planner, generator and healer workflows.
- Playwright CLI for coding agents ↗ — token-efficient browser control.
- GitHub Copilot cloud agent ↗ — repository tasks and reviewed pull requests.
- GitHub Copilot code review ↗ — AI-assisted pull-request feedback.
- OpenAI evaluations guide ↗ — evaluating model and agent behaviour.
- OWASP GenAI Security Project ↗ — security risks and testing guidance.