Use AI to accelerate test design, generate candidate test code, expand boundary cases, classify failures and maintain supporting artefacts—but keep execution, assertions, release gates and risk decisions deterministic. The highest-value model is human-defined intent plus machine-generated candidates plus automated verification. AI should propose; the test framework, application contracts and reviewers should prove.

Start with one stable service or user journey. Give the AI the production code, approved requirements, existing test conventions and explicit risks. Ask for a test plan before code. Generate a small candidate set, run it in an isolated environment, review every assertion and measure whether it detects seeded defects. Only then add the accepted tests to continuous integration. This approach creates leverage without turning probabilistic output into false confidence.

What AI can and cannot automate

AI is useful where testing contains expensive synthesis: reading unfamiliar code, proposing representative inputs, translating acceptance criteria into test skeletons, identifying untested branches, producing fixtures and summarising traces. It is less reliable when it must decide the correct business outcome without a specification, infer security policy, judge a subtle visual defect or distinguish an intended product change from a regression.

A generated test that passes proves only that the test and current code agree. It does not prove that either is correct. AI may reproduce an implementation defect in the assertion, invent an API, over-mock dependencies or create brittle selectors. The operating principle is therefore: automate test creation only inside an evidence system that can reject weak tests.

Where AI delivers practical value

Unit-test candidates

AI can inspect a function and propose normal, boundary, invalid-input, exception and side-effect scenarios. GitHub’s official Copilot guidance recommends focused tests that follow the project’s framework and patterns, use descriptive names and validate behaviour rather than implementation details. The generated suite still requires review, execution and additional cases for domain rules the model cannot infer.

Test-data and parameter expansion

Once engineers define meaningful categories, AI can suggest values around thresholds, empty states, Unicode, time zones, permissions and malformed structures. Convert approved cases into deterministic parameter sets. Pytest, for example, supports parameterising test functions and fixtures; this is a conventional test capability, not an AI feature. AI helps design the matrix while the framework executes it repeatably.

Integration contract tests

Provide OpenAPI, event schemas, database constraints and error contracts. Ask the model to identify missing status codes, incompatible field changes and failure cases, then encode accepted scenarios in contract tests. Avoid asking it to infer undocumented partner behaviour. Use sandbox responses or provider fixtures and keep external credentials out of prompts.

Browser-test scaffolding

AI can turn a written journey into Playwright test code, suggest accessible locators and explain a failing trace. Playwright also includes a deterministic test generator that records actions and prioritises role, text and test-ID locators. Combining recorded interaction with AI-assisted refactoring is usually safer than asking a model to invent the whole page structure.

Failure triage

Models can cluster similar stack traces, summarise changed files, compare a failure with prior incidents and suggest likely owners. Keep the raw log, screenshot, trace and test result as the source of truth. A model’s diagnosis should be labelled as a hypothesis until an engineer reproduces or proves it.

Maintenance assistance

When a reviewed product change invalidates tests, AI can propose locator, fixture or assertion updates using the change diff and approved acceptance criteria. Do not approve blanket “fix all tests” changes. Require the model to explain why each expected result changed and reject edits that merely make a failing test green.

CTA: Project Supply’s Digital Engineering team can audit your current test architecture and identify where AI will reduce effort without weakening release confidence.

Build the foundation before adding AI

Define the quality model

List the behaviours that matter to customers and the business: correct calculations, access control, data integrity, availability, performance, accessibility and recovery. Assign risk by impact and likelihood. The test portfolio should reflect those risks rather than chase a single coverage percentage.

Stabilise test seams

Make code testable through explicit interfaces, dependency injection, deterministic clocks, isolated state and controllable external services. AI cannot compensate for architecture that requires an entire production environment to test one decision. Better seams improve both human-written and generated tests.

Establish conventions

Document the test framework, folder structure, naming, fixture rules, mocking policy, assertion style and commands. Provide a few high-quality examples. Generated code is far more useful when the model can imitate an approved local pattern instead of generic internet code.

Create a trusted oracle

An oracle defines the expected result. It may be a business rule, schema, reference implementation, snapshot approved by a reviewer or known-good dataset. If the oracle is missing, AI should produce questions or candidate expectations—not mergeable tests. Never let the same unverified model response generate both input and expected output for a high-risk behaviour.

A safe AI testing workflow

Step 1: select a bounded target

Choose a function, endpoint or journey with clear ownership and existing behaviour. Avoid beginning with a highly stateful legacy area. Record current defects, execution time, flaky-test rate and review effort so the pilot has a baseline.

Step 2: assemble controlled context

Provide the smallest relevant code, interface definitions, requirements, approved examples and test conventions. Remove secrets, personal data and proprietary material not permitted by the organisation’s AI policy. State what the model must not assume.

Step 3: request a test plan

Ask for behaviours, equivalence classes, boundaries, error paths, dependencies and missing information. Review the plan with the developer and tester before asking for code. This exposes misunderstood requirements cheaply.

Step 4: generate candidate tests

Request a limited set using the existing framework. Require descriptive names, arrange-act-assert structure, independent execution and comments only where setup is non-obvious. Ask the model to cite the requirement or code path each test covers.

Step 5: run in isolation

Execute formatting, static checks and the target tests in a sandbox. Do not allow an autonomous agent unrestricted production credentials or network access. Capture failures and generated files. A test that does not compile is rejected before human review.

Step 6: validate test effectiveness

Temporarily seed representative defects or use mutation testing where appropriate. Confirm that the new tests fail for the intended reason. Inspect whether assertions are meaningful and mocks preserve the behaviour under test. A test that survives a relevant mutation adds little protection.

Step 7: human review

Review the test and the production code together. Check domain correctness, security cases, fixture realism, determinism and maintenance cost. The reviewer should be able to explain every expected value. Record AI assistance according to organisational policy.

Step 8: merge through normal controls

Use the same pull-request, CI and ownership rules as human-authored code. AI-generated tests should not receive a weaker gate. Keep a small diff and link the acceptance criteria or defect.

Unit and component testing

Ask AI to identify pure logic, validation, state transitions and error handling. Prefer tables of inputs and outputs before implementation. Use parameterisation to cover approved classes without copying near-identical tests. Keep fixtures explicit and small. Mock external systems only at clear boundaries; excessive mocking tests the model’s invented interaction rather than the application.

For legacy code, have AI explain dependencies and propose characterisation tests that capture current behaviour. Mark disputed behaviour for product review instead of silently treating it as correct. Once the behaviour is understood, refactor behind the tests and replace accidental assertions with business-facing ones.

API and integration testing

Generate tests from the API contract, then augment them with business invariants. Validate authentication, authorisation, idempotency, pagination, concurrency, timeouts, retry safety, malformed input and partial dependency failure. Test observable outcomes—stored records, emitted events and response contracts—not private method calls.

Use deterministic service virtualisation for routine CI and a smaller set of real integration tests in a controlled environment. AI can draft fixtures from schemas, but engineers must verify that examples satisfy provider constraints. Refresh recorded fixtures deliberately and scan them for secrets.

End-to-end browser testing

Reserve end-to-end tests for critical journeys that cross meaningful boundaries: signup, checkout, subscription change, permission assignment or recovery. Use Playwright’s locators and auto-waiting rather than fixed sleeps. Generate or record a baseline, then refactor repeated actions into reviewed fixtures or page objects.

On failure, preserve trace evidence. Playwright’s Trace Viewer exposes actions, DOM snapshots, source locations, console output and network requests. AI can summarise this evidence, but the trace remains authoritative. Playwright recommends recording traces on the first retry or retaining them on failure rather than enabling heavy tracing for every successful run.

Testing AI-enabled applications

Separate deterministic and probabilistic layers

Test prompt construction, tool schemas, permissions, parsing and fallback deterministically. Evaluate generated content with datasets, rubrics and multiple runs. Do not force a probabilistic answer into exact-string assertions unless the output contract is intentionally fixed.

Build evaluation datasets

Include normal questions, ambiguous requests, missing context, conflicting sources, prompt injection, sensitive-data attempts and tool failures. Store expected properties such as required citations, prohibited disclosures, valid JSON or correct routing. Use human-reviewed examples for high-impact domains.

Test tool safety

Validate every model-produced tool argument against a schema and the authenticated user’s permissions. Use read-only or sandbox tools during evaluation. Require approval for irreversible actions. OWASP’s guidance for LLM applications highlights risks such as prompt injection; security testing must cover the full application and tool boundary, not only the model response.

CTA: Project Supply’s AI and Data Analytics team can design evaluation datasets, model-quality gates and tool-safety tests for production AI applications.

CI/CD architecture

Pull-request gate

Run fast unit, component, lint and type checks on every change. Allow AI to suggest tests, but require deterministic execution and review before merge. Fail on changed snapshots or generated files that lack approval.

Integration stage

Provision isolated databases and services, apply migrations and run contract tests. Use short-lived credentials. Publish machine-readable reports and retain relevant artefacts. Quarantine is a temporary investigation state, not a permanent home for flaky tests.

End-to-end stage

Run priority journeys against an environment that matches production interfaces. Parallelise carefully, isolate accounts and collect traces on failure. Playwright documents CI setup and supports sharding and retries; configure these to reveal rather than conceal instability.

Post-deployment verification

Run a small smoke suite after deployment and monitor real service indicators. A successful test pipeline does not prove production health when dependencies, data volume and traffic patterns differ. Connect incidents back to missing tests and add the smallest durable regression check.

Metrics that reveal real improvement

Measure escaped defects by severity, change failure rate, time to detect, time to diagnose, flaky-test rate, suite duration, review time and mutation score for selected modules. Track generated tests proposed, accepted, substantially rewritten and later removed. A high generation count with low acceptance is not productivity.

Measure coverage only as a diagnostic. Line coverage can increase while important assertions remain absent. Pair coverage with risk mapping and defect detection. For AI application evaluations, track pass rates by scenario, variance across runs, unsupported claims, tool errors and human escalation.

Governance and security

Approve AI tools and data-use terms. Define which repositories and data may be sent to hosted models. Protect source code, credentials, customer data and test artefacts. Log generated changes and reviewers where required. Keep autonomous execution inside least-privilege sandboxes with network restrictions and spending limits.

Review dependencies suggested by models before installation. Check package origin, maintenance and licence. Models can invent package names or recommend outdated APIs. Pin accepted versions and scan the normal software supply chain.

Common failure modes

Generating tests from code alone

The model mirrors implementation instead of validating requirements. Provide business rules and expected outcomes.

Optimising for coverage

Generated tests execute lines without meaningful assertions. Require defect-detection evidence.

Auto-updating failed assertions

This converts regressions into new expectations. Change an oracle only with an approved product decision.

Creating brittle UI tests

Models may choose unstable selectors or sleeps. Prefer accessible locators, explicit state and trace-based diagnosis.

Granting broad agent access

An agent that can edit code, run commands and access production creates unnecessary risk. Use scoped workspaces and review gates.

CTA: Contact Project Supply for an AI-assisted testing pilot covering one application area, measurable quality gates and a scale-up roadmap.

90-day rollout

Days 1–15: baseline and policy

Select the target, measure quality and effort, document conventions, approve tool access and define acceptance criteria.

Days 16–35: controlled generation

Generate plans and tests for bounded units. Review every output, run mutation checks and record acceptance.

Days 36–55: integration and browser pilot

Add contract and critical-journey candidates. Configure traces, deterministic data and CI artefacts.

Days 56–70: AI application evaluation

If relevant, add prompt-injection, retrieval, tool and output-contract datasets. Establish human escalation.

Days 71–90: scale by evidence

Expand only where defect detection or review speed improved without increasing flakiness or escaped risk