Test Automation Principles

What to automate

Automate checks that are repetitive, stable, important and cheap to express as a clear pass or fail:

• Business rules and calculations
• Regression checks for bugs you have already fixed
• Contract checks between components, such as an API response shape
• A smoke test that runs after each deployment

Leave to humans what needs judgement: whether a screen feels right, new features still changing every week, and one-off checks. See Manual Testing Techniques.

The test pyramid

Tests that exercise a single function are fast and precise. Tests that drive the whole system through the user interface are slow, fragile and hard to diagnose when they fail. The usual advice is a wide base of unit tests, fewer integration tests and only a handful of end-to-end tests for the critical user journeys.

What a healthy suite looks like

• It is fast. If the suite takes too long, people stop running it locally.
• It is deterministic: the same code gives the same result every time. A test that sometimes fails for no reason is called flaky. Fix or delete it, because it teaches people to ignore red builds.
• Tests are independent. They do not depend on each other or on run order.
• It runs automatically. Tests only protect you if they run on every change, usually in CI.

Coverage and brittle tests

• Chasing a coverage percentage encourages meaningless tests.
• Tests that mirror the implementation line by line break on every refactoring.

Try it

Look at a project’s test suite, or at a list of manual checks your team does before a release. Pick three checks and decide for each whether it should be automated, at what level (unit, integration or end-to-end), and why. Then note which one would be most likely to become flaky.