Testing Professor
Testing ProfessorMulti-Cloud Guidebook · AWS
All scenarios

End-to-End QA Scenario

Nightly k6 Performance Suite on AWS from GitHub Actions

Scheduled GitHub Actions workflow runs distributed k6 on ECS Fargate against staging, streams metrics to CloudWatch, gates on p95/error-rate, and publishes a trend dashboard to S3 + GitHub Pages.

Architecture

GitHub Actions (cron: 0 2 * * *)
   ├─ OIDC assume role
   ├─ aws ecs run-task --launch-type FARGATE --count 10
   │       └─ k6 distributed pods → staging API
   │            ├─ metrics → CloudWatch (embedded metric format)
   │            └─ summary.json → S3 (perf/{date}/)
   ├─ poll task completion (waiter)
   ├─ evaluate thresholds (p95<800ms, error<1%, RPS>=2000)
   │       ├─ PASS → publish HTML trend report to S3 → GitHub Pages
   │       └─ FAIL → open GitHub Issue + SNS → PagerDuty
   └─ post summary to PR / commit status of last green release

Workflow steps

  1. 1

    Schedule

    Workflow with `on: schedule: cron: '0 2 * * *'` plus `workflow_dispatch` for manual runs.

  2. 2

    Assume role

    OIDC role grants `ecs:RunTask`, `ecs:DescribeTasks`, `s3:PutObject` on the perf bucket, `cloudwatch:PutMetricData` and read on the k6 task role.

  3. 3

    Launch grid

    `aws ecs run-task` starts 10 Fargate tasks running a k6 image from ECR, sharding VUs via `K6_SHARD_INDEX`/`K6_SHARD_COUNT`. Target base URL and tokens come from Secrets Manager.

  4. 4

    Stream metrics

    k6 writes CloudWatch EMF lines (`http_req_duration`, `vus`, `iterations`) so a single dashboard shows the run live without a custom exporter.

  5. 5

    Wait + collect

    `aws ecs wait tasks-stopped` blocks until all shards finish; each shard uploads `summary.json` to `s3://qa-perf/{date}/shard-{i}/`.

  6. 6

    Gate

    A `jq` step merges shard summaries and fails the workflow if `p95 > 800`, `error rate > 1%` or `RPS < 2000`. The same step writes a one-line history record to `s3://qa-perf/history.csv`.

  7. 7

    Trend report

    On pass, a Node step renders `history.csv` into a static HTML chart and pushes it to a `gh-pages` branch — anyone can see 30-day p95 trend at a stable URL.

  8. 8

    Alert on regression

    On fail, the workflow opens (or updates) a GitHub Issue with the diff vs. last green run and publishes to an SNS topic that fans out to email + PagerDuty.

Key takeaways

  • Fargate + GitHub Actions = on-demand k6 grid with zero idle cost.
  • CloudWatch EMF avoids running a Prometheus/InfluxDB just for nightly perf.
  • A static GitHub Pages trend page beats a dashboard no one logs into.