Tests outside CI soon stop mattering
Tests run by hand, whenever someone remembers them, age quietly. After a few weeks half of them fail and nobody knows whether the bugs are in the app or in the tests. CI solves this, but only if the team trusts the results and doesn’t wait hours for them.
In one of my projects, automated regression cut the time spent on regression by 80%. The tests themselves are half of the success. The other half is when they run.
Two levels instead of one huge suite
| level | contents | when | time |
|---|---|---|---|
| Smoke | the most important paths | every pull request | a few minutes |
| Full regression | all scenarios | overnight or before a release | as long as it takes |
The times in the table are indicative. The rule is simple: developers wait for the result on every PR, so smoke has to be short. If it takes too long, the team will start working around it.
An example in GitLab CI
Below are two jobs using Playwright. Smoke tests tagged @smoke run on every merge request, and full regression runs on a schedule:
stages: [smoke, regression]
.playwright:
image: mcr.microsoft.com/playwright:v1.49.0-jammy
before_script:
- npm ci
artifacts:
when: always
paths: [playwright-report/]
smoke:
extends: .playwright
stage: smoke
script:
- npx playwright test --grep @smoke --reporter=html
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
regression:
extends: .playwright
stage: regression
script:
- npx playwright test --reporter=html
rules:
- if: $CI_PIPELINE_SOURCE == "schedule"The image version must match the Playwright version in your project. when: always saves the report even when tests fail, which is exactly when you need it most. In Jenkins the principle is the same: separate jobs, with reports archived as artifacts.
Flaky tests
A test that passes one run and fails the next is worse than no test. It teaches the team to ignore a red pipeline. When you spot one:
- Find the cause. Most often it’s a race with asynchronous loading, a dependency on test order or shared data.
- Don’t add fixed delays. Wait for a specific state, not for time to pass.
- Disable the test while it’s being fixed, but with a ticket and an owner.
Automatic retries hide the problem. If you use them, track which tests only pass on the second attempt.
Data and environment
Tests in CI run in parallel and unattended, so they can’t depend on manually prepared state. Create data through the API before the test, use unique accounts and clean up afterwards. Keep secrets in CI variables, not in the repository.
What to do with the result
A red pipeline has to mean something. Agree that a failed smoke run blocks the merge and that someone reviews the overnight regression in the morning. Without an owner, reports stop being read. A simple dashboard with a trend helps you notice the number of problems growing before it becomes a headache ahead of a release.
Summary
Good regression in CI means short smoke on every change, the full suite on a schedule, zero tolerance for flaky tests and a clear owner for the results. If you want to plug your tests into CI or restore trust in the ones you have, describe your project.