The fixture framework

What a golden fixture is

A golden fixture is a synthetic tenant state, hand-built to represent one situation a check must recognize, driven through the real check function and asserted to produce a specific verdict. It is not a mock of the check. It is real input to the real detection logic, with the answer written down in advance.

Because the fixture runs the actual function, it tests the thing that ships: the same code path a live scan takes, minus the network call that collected the data. If the verdict logic is wrong, the fixture fails. If someone later changes the logic and breaks a case, the fixture that covered that case turns red before the change can merge.

The three assertions

Every check that can be fixtured is held to three cases. Together they prove the check is right when things are fine, right when they are not, and honest when it cannot tell.

The gate

Four gates run before any release: the golden fixtures (verdict logic), the collector query contracts (each collector requests exactly the API endpoints and parameters its check reads), the Zero Trust schema test (every check declares a pillar and weight), and the full unit suite. A red run in any of them blocks the release.

The fixture run also emits the numbers above as a machine-readable artifact. This website reads that artifact, and its own build fails if a page count or stat disagrees with it. That is why the counts here cannot drift: the only thing that can change them is a run that passed.

The gates prove they can fail

A green light is only meaningful if red is reachable. Every release gate has a poison self-test: a deliberate failure is injected through the gate's literal invocation shape, and the gate must exit non-zero. A gate that stays green on poisoned input fails the release.

This website holds itself to the same rule. Its build guards (page counts, derived numbers, competitor and retired-term scans, contrast ratios) each have a poison case run in CI on every change: a definition without a page, a red run artifact, a hard-coded stat, a failing color pair. Any guard that cannot fail turns the build red.

From module to page, end to end

The full pipeline behind every number on this site, as text:

1. The module's check definitions and golden fixtures live in the Guerrilla repository.
2. The gating test run drives every fixture through the real check functions. Only a green run emits the artifact (counts, per-check verdicts, module version, git SHA).
3. The site generator reads the artifact and the check definitions and emits one page of data per check, plus the category and baseline crosswalk indexes. It refuses to run on a red or stale artifact.
4. The site build renders those pages and re-verifies everything: page count equals checks tested, no underived numbers, no competitor names, no retired codenames, both themes pass contrast.
5. CI runs the build, the guard poison self-tests, and the accessibility checks on every change. A deploy path that skips them does not exist.

Write a fixture, step by step

Here is a real one from the repository. GROUP-001 checks whether Google Workspace groups are restricted to domain users. It reads one Cloud Identity policy setting, groups_for_business.groups_sharing. A fixture is a JSON file naming the check, the scenario, the expected verdict, and the synthetic auditData the check will see. Drop it in Tests/Fixtures/ and it is picked up automatically.

1. Clean, expect PASS

Sharing is limited to domain users, the recommended state. The check must pass.

{
  "checkId": "GROUP-001",
  "scenario": "clean",
  "expectedStatus": "PASS",
  "auditData": { "CloudIdentityPolicies": { "ByType": {
    "groups_for_business.groups_sharing": [
      { "setting": { "value": { "collaborationCapability": "DOMAIN_USERS_ONLY" } } } ] } } }
}

2. Known-bad, expect FAIL

One field flips to ANYONE_CAN_ACCESS, the exposure the check exists to catch. Nothing else changes. The check must fail.

{
  "checkId": "GROUP-001",
  "scenario": "known-bad",
  "expectedStatus": "FAIL",
  "auditData": { "CloudIdentityPolicies": { "ByType": {
    "groups_for_business.groups_sharing": [
      { "setting": { "value": { "collaborationCapability": "ANYONE_CAN_ACCESS" } } } ] } } }
}

3. Uncollectable, expect Not Assessed

The policy API returned a 403, so the setting is unknown. The check must report Not Assessed, not guess. SKIP is how a fixture writes Not Assessed.

{
  "checkId": "GROUP-001",
  "scenario": "not-assessed",
  "expectedStatus": "SKIP",
  "auditData": { "Errors": { "CloudIdentityPolicies": "Policy API 403" } }
}

That is the whole contract. Three small files, each a real tenant shape with the answer written down. Run pwsh Tests/Invoke-FixtureTests.ps1 and the suite drives your fixtures through the check and confirms each verdict. When they pass, the check is proven, and a maintainer can accept it from you without having your tenant in front of them.

Contribute

The fixture requirement is what lets this project accept a check from a stranger. If you have found a verdict that is wrong, or a check worth adding, or an unusual real tenant shape worth capturing as a fixture, that is a contribution and it is credited. The contribution ladder.