Mid-Level Automation · Pipeline Integration

CI Basics

Tests that only run on your laptop are hobbies, not automation. Learn how to wire your suite into CI so every code change gets validated automatically.

Mid-Level Automation ISTQB CTAL-TAE v2.0 — Chapter 5 ~12 min read + exercise

1 The Hook — Why This Matters

A Dunedin software agency had a talented automation engineer who wrote 150 beautiful Playwright tests. They sat in a repo, untouched, for six months. Why? No one had wired them into the build pipeline. Developers continued merging code without running the suite. When the client finally demanded automated regression, thirty-seven tests failed immediately — not because the application was broken, but because the tests had drifted from six months of UI changes.

Tests in CI are the only tests that matter. Everything else is documentation that might be wrong.

2 The Rule — The One-Sentence Version

Your test suite must run automatically on every meaningful code change, fail the build on failure, and produce artifacts that let you debug without rerunning locally.

Manual test execution doesn't scale. If a developer has to remember to run tests, they won't. CI is your enforcement mechanism and your safety net.

3 The Analogy — Think Of It Like...

Analogy

A fire alarm that only rings if someone remembers to press the button.

You wouldn't trust your life to a manual fire alarm. You shouldn't trust your release to manual test runs. CI is the automatic sensor: it detects smoke (broken code) and triggers the alarm (failed build) without human intervention. No memory required. No discipline required. Just engineering.

4 Watch Me Do It — Step by Step

Here is a complete GitHub Actions workflow that runs a Playwright test suite.

  1. Create the workflow file
    # .github/workflows/test.yml
    name: Test Suite
    on:
      push:
        branches: [main]
      pull_request:
        branches: [main]
    
    jobs:
      unit-tests:
        runs-on: ubuntu-latest
        steps:
          - uses: actions/checkout@v4
          - uses: actions/setup-node@v4
            with:
              node-version: "20"
              cache: "npm"
          - run: npm ci
          - run: npm run test:unit
    
      e2e-tests:
        runs-on: ubuntu-latest
        needs: unit-tests
        steps:
          - uses: actions/checkout@v4
          - uses: actions/setup-node@v4
            with:
              node-version: "20"
              cache: "npm"
          - run: npm ci
          - run: npx playwright install --with-deps
          - run: npx playwright test
          - uses: actions/upload-artifact@v4
            if: failure()
            with:
              name: playwright-report
              path: playwright-report/
  2. Understand the key concepts
    ConceptWhat it is
    WorkflowYAML file defining an automated process
    JobA set of steps running on the same runner
    StepAn individual task (command or action)
    ActionReusable unit (e.g., actions/checkout@v4)
    RunnerThe VM/container executing jobs
  3. Add triggers for your needs
    on:
      push:
        branches: [main, develop]
      pull_request:
        branches: [main]
      schedule:
        - cron: "0 2 * * *"  # nightly at 2am UTC
  4. Cache dependencies
    - uses: actions/cache@v4
      with:
        path: ~/.npm
        key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
Parallelise tests with matrix strategy — Faster feedback
e2e-tests:
  runs-on: ubuntu-latest
  strategy:
    matrix:
      shard: [1, 2, 3, 4]
  steps:
    - uses: actions/checkout@v4
    - uses: actions/setup-node@v4
    - run: npx playwright test --shard=${{ matrix.shard }}/4
    - uses: actions/upload-artifact@v4
      if: failure()
      with:
        name: playwright-report-shard-${{ matrix.shard }}
        path: playwright-report/

This example splits the Playwright suite across four parallel workers. Instead of running 100 tests serially in 10 minutes, it runs ~25 per worker in 2.5 minutes total. Artifacts are uploaded per shard so you can debug which worker found the failure.

Pro tip: NZ tech employers like Trade Me explicitly list "automation-first mindset spanning test pyramid layers" in job postings. Being able to design and troubleshoot CI pipelines is a differentiator between mid-level and junior automation engineers.

5 When to Use It / When NOT to Use It

✅ Run in CI when...

  • Merging to main or develop
  • Opening or updating pull requests
  • Releasing to staging or production
  • Nightly regression validation

❌ Skip CI triggers when...

  • Only documentation changed
  • The change is a README or comment update
  • You've already run identical tests locally

Before you set up CI, ask:

  • What is the order of slowest-to-fastest tests, and should they run in parallel or sequence?
  • Are there dependencies between jobs (e.g., E2E tests depend on unit tests passing)?
  • Where will secrets and credentials live, and how will they be rotated safely?
  • What artifacts must be retained on failure, and for how long?

6 Common Mistakes — Don't Do This

🚫 No artifact upload on failure

I used to think: The console log tells me everything.
Actually: "Element not found" in a headless CI runner is impossible to debug without screenshots. Always upload HTML reports, screenshots, and traces on failure. Use if: failure() to save storage.

🚫 Running E2E tests as the first step

I used to think: E2E tests are the most important, so they should run first.
Actually: E2E tests are the slowest. Run lint -> unit -> integration -> E2E in sequence. Fail fast on cheap checks before spending ten minutes on browser tests.

🚫 Using unpinned action versions

I used to think: @v4 is pinned enough.
Actually: Tags can be moved. A malicious or broken update to actions/checkout@v4 could compromise your pipeline. Use commit SHAs for critical workflows, or at least pin to specific major versions and review changelogs.

When this technique fails

CI pipelines fail when you have not configured proper secrets management, when tests are flaky and block merges constantly, or when the pipeline takes so long that developers start finding workarounds to skip it. A broken CI is worse than no CI because teams lose trust and stop using it.

7 Now You Try — Interview Warm-Up

🎯 Interactive Exercise

Scenario: Your E2E job in GitHub Actions takes 25 minutes. Developers are bypassing the required status check because "it takes too long and blocks the merge." Management wants faster feedback without losing coverage.

What three changes would you propose?

Three changes:

  1. Split the suite: Run smoke tests (critical paths only, ~3 min) as the PR gate. Run the full regression suite nightly or on main merge.
  2. Parallelise with sharding: Use Playwright's --shard option or matrix strategy to split tests across multiple runners.
  3. Optimise test order: Run fast API tests before slow UI tests. Remove redundant or obsolete tests that no longer add value.

8 Self-Check — Can You Actually Do This?

Click each question to reveal the answer. If you got all three, you're ready to practice.

Q1. What is the purpose of needs: unit-tests in a GitHub Actions job?

It makes the e2e-tests job wait until unit-tests completes successfully. This saves CI minutes by not running expensive E2E tests when cheap unit tests already failed.

Q2. Why should you upload artifacts with if: failure() rather than if: always()?

if: failure() only uploads when tests fail, saving storage on successful runs. if: always() uploads every time, which can waste gigabytes of storage and slow down the pipeline.

Q3. What three things should you cache in a CI pipeline?

1) Node modules / pip dependencies, 2) Browser binaries (Playwright), 3) Build artifacts between jobs. Caching avoids redundant downloads and can cut pipeline time by 50% or more.

9 Interview Prep — Mid-Level Q&A

Interviewers ask about pipeline design, failure handling, and how you've troubleshot CI issues. Here are common scenarios.

Q1. "Describe a time a CI pipeline change broke your workflow. How did you fix it?"

I've seen caching issues where a new action version cached differently, causing flaky tests. I diagnosed it by comparing local runs to CI runs, found the cache-key mismatch, and updated the action version. Now I pin action versions and review changelogs before merging.

Q2. "How do you prevent flaky tests from breaking CI?"

I use multiple strategies: add explicit waits instead of implicit ones, isolate test data so there are no shared-state collisions, and run tests multiple times locally before merging. In CI, I use retry logic for genuinely flaky tests and quarantine consistently failing tests in a separate job so they do not block the main pipeline.

Q3. "What is your approach to secrets and credentials in CI?"

Never hardcode secrets in version control. Use CI environment secrets (GitHub Secrets, GitLab CI variables) that are injected at runtime and masked in logs. For local development, use `.env` files (in `.gitignore`) or a secret manager like 1Password. Rotate test credentials quarterly and audit access regularly.

Q4. "How would you reduce a 25-minute CI pipeline to 8 minutes without losing coverage?"

First, parallelise expensive tests (E2E, integration) using matrix strategy or sharding. Second, split the suite: smoke tests (critical paths) on every PR, full regression nightly. Third, cache dependencies and build artifacts between jobs. Fourth, remove redundant tests that duplicate coverage. Finally, run fast checks first (lint, unit) before slow ones (E2E) so failures fail fast.