Skip to content
Back to search

modulestf

SkillWorks with Claude CodeGitHub

Review and maintain reusable Terraform modules against a written profile of the module family's rules: pull request review with one verdict, findings-only review of a local change, and a maintainer skill that makes the change.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/modulestf/modulestf with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/modulestf/modulestf'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/modulestf--modulestf/badge.svg)](/listings/modulestf--modulestf)

HTML

<a href="/listings/modulestf--modulestf"><img src="/api/listings/modulestf--modulestf/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

1. Install the skills as the `modulestf` plugin from the [antonbabenko/agent-plugins](https://github.com/antonbabenko/agent-plugins) marketplace. It holds all four skills and the two work in progress skeletons; the Terraform skills link into each other, so they ship together.

   # Claude Code
   /plugin marketplace add antonbabenko/agent-plugins
   /plugin install modulestf@antonbabenko

   # Codex: add the marketplace, then install modulestf from /plugins
   codex plugin marketplace add antonbabenko/agent-plugins

Or copy or symlink all the directories under `.agents/skills/` side by side into your agent's skills directory: `~/.claude/skills/` for Claude Code, `~/.agents/skills/` for Codex. `change-scope` also works alone.

   git clone https://github.com/modulestf/modulestf.git
   mkdir -p ~/.claude/skills
   ln -sfn "$PWD"/modulestf/.agents/skills/* ~/.claude/skills/   # Claude Code
   mkdir -p ~/.agents/skills
   ln -sfn "$PWD"/modulestf/.agents/skills/* ~/.agents/skills/   # Codex

2. Add HashiCorp's [Terraform MCP server](https://github.com/hashicorp/terraform-mcp-server) to your agent under the name `terraform`, so its tools appear as `mcp__terraform__get_latest_provider_version`, `mcp__terraform__search_provider_docs` and `mcp__terraform__get_provider_details`. The skills read provider schemas from it, not from memory. Without it, every rule that needs a provider fact is reported as not run, so a pull request review is inconclusive at best. 3. Log in to GitHub with `gh auth login` as yourself, and install `terraform`, `git`, `jq`, `perl` and [`mtf-facts`](#mtf-facts). A pull request review needs `mtf-facts` for a conclusive verdict.

   brew install modulestf/tap/mtf-facts

Without Homebrew: `go install github.com/modulestf/facts/cmd/mtf-facts@latest`.

4. Ask your agent:

   review terraform-aws-modules/terraform-aws-lambda#780

The skill clones the pull request into a temporary directory, runs `terraform init -backend=false` and `terraform validate` in each example, the ones the change touches first, and shows you a verdict, the findings and the comment it would post. It posts nothing until you confirm the comment and its review event: comment, approve or request changes.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/plugin.json · Manifest · 20 lines
  1. {
  2. "name": "modulestf",
  3. "description": "Review and maintain reusable Terraform modules against a written profile of the module family's rules: pull request review with one verdict, findings-only review of a local change, and a maintainer skill that makes the change.",
  4. "author": {
  5. "name": "Anton Babenko",
  6. "url": "https://github.com/antonbabenko"
  7. },
  8. "homepage": "https://modules.tf",
  9. "repository": "https://github.com/modulestf/modulestf",
  10. "license": "Apache-2.0",
  11. "keywords": [
  12. "terraform",
  13. "terraform-modules",
  14. "terraform-aws-modules",
  15. "code-review",
  16. "pull-request-review",
  17. "iac"
  18. ],
  19. "skills": "./.agents/skills/"
  20. }
README.md · README · 265 lines
  1. # modulestf
  2. Agent skills that review reusable Terraform modules against a written profile of the module family's rules, and help you make the change the review asks for. One profile ships today: [terraform-aws-modules](https://github.com/terraform-aws-modules). They run in your coding agent (Claude Code or Codex) on your machine, with your own GitHub login.
  3. - [The skills](#the-skills)
  4. - [Quick start](#quick-start)
  5. - [Everyday use](#everyday-use)
  6. - [Reading a review](#reading-a-review)
  7. - [Verification ladder](#verification-ladder)
  8. - [Limits](#limits)
  9. - [Advanced use](#advanced-use)
  10. - [Requirements by task](#requirements-by-task)
  11. - [Related repositories](#related-repositories)
  12. ## The skills
  13. | Skill | You give it | You get back |
  14. |-------|-------------|--------------|
  15. | [`terraform-module-pr-review`](.agents/skills/terraform-module-pr-review/SKILL.md) | A pull request or commit URL, or `owner/repo#123` | A verdict and a comment to post, after you confirm |
  16. | [`terraform-module-pr-fix`](.agents/skills/terraform-module-pr-fix/SKILL.md) | A pull request URL or `owner/repo#123` to finish | A reviewed fix pushed to the pull request's branch, after you confirm, or a patch |
  17. | [`terraform-module-pr-merge`](.agents/skills/terraform-module-pr-merge/SKILL.md) | A pull request URL or `owner/repo#123` to merge | A fresh review, then a squash merge after you confirm the pull request and its full head SHA |
  18. | [`terraform-module-reviewer`](.agents/skills/terraform-module-reviewer/SKILL.md) | A module checkout and a diff, a base ref or a list of changed files | A list of findings. Read-only, no verdict |
  19. | [`terraform-module-maintainer`](.agents/skills/terraform-module-maintainer/SKILL.md) | A change to make, review feedback to work through, a module to create or a provider to upgrade | Edits in your checkout, a report of the checks it ran, and reply drafts |
  20. | [`change-scope`](.agents/skills/change-scope/SKILL.md) | A proposed addition, in any language or project | Whether it belongs in the thing being changed, and where it goes if not |
  21. The pull request review calls the reviewer, and both call `change-scope`. The pull request fix calls the pull request review, the maintainer and the reviewer. The pull request merge calls the pull request review. The rules of the module family live in a profile: [`terraform-aws-modules`](.agents/skills/terraform-module-maintainer/profiles/terraform-aws-modules/PROFILE.md).
  22. Two more skills are work in progress and not usable yet. Each refuses every request and points you to the maintainer:
  23. - [`terraform-module-issue`](.agents/skills/terraform-module-issue/SKILL.md): will turn a GitHub issue into a module change and a pull request.
  24. - [`terraform-module-create`](.agents/skills/terraform-module-create/SKILL.md): will create a new module repository from a profile.
  25. How the skills fit together, and which skills come next: [docs/skills-architecture.md](docs/skills-architecture.md).
  26. ## Quick start
  27. 1. Install the skills as the `modulestf` plugin from the [antonbabenko/agent-plugins](https://github.com/antonbabenko/agent-plugins) marketplace. It holds all four skills and the two work in progress skeletons; the Terraform skills link into each other, so they ship together.
  28. ```text
  29. # Claude Code
  30. /plugin marketplace add antonbabenko/agent-plugins
  31. /plugin install modulestf@antonbabenko
  32. # Codex: add the marketplace, then install modulestf from /plugins
  33. codex plugin marketplace add antonbabenko/agent-plugins
  34. ```
  35. Or copy or symlink all the directories under `.agents/skills/` side by side into your agent's skills directory: `~/.claude/skills/` for Claude Code, `~/.agents/skills/` for Codex. `change-scope` also works alone.
  36. ```bash
  37. git clone https://github.com/modulestf/modulestf.git
  38. mkdir -p ~/.claude/skills
  39. ln -sfn "$PWD"/modulestf/.agents/skills/* ~/.claude/skills/ # Claude Code
  40. mkdir -p ~/.agents/skills
  41. ln -sfn "$PWD"/modulestf/.agents/skills/* ~/.agents/skills/ # Codex
  42. ```
  43. 2. Add HashiCorp's [Terraform MCP server](https://github.com/hashicorp/terraform-mcp-server) to your agent under the name `terraform`, so its tools appear as `mcp__terraform__get_latest_provider_version`, `mcp__terraform__search_provider_docs` and `mcp__terraform__get_provider_details`. The skills read provider schemas from it, not from memory. Without it, every rule that needs a provider fact is reported as not run, so a pull request review is inconclusive at best.
  44. 3. Log in to GitHub with `gh auth login` as yourself, and install `terraform`, `git`, `jq`, `perl` and [`mtf-facts`](#mtf-facts). A pull request review needs `mtf-facts` for a conclusive verdict.
  45. ```bash
  46. brew install modulestf/tap/mtf-facts
  47. ```
  48. Without Homebrew: `go install github.com/modulestf/facts/cmd/mtf-facts@latest`.
  49. 4. Ask your agent:
  50. ```text
  51. review terraform-aws-modules/terraform-aws-lambda#780
  52. ```
  53. The skill clones the pull request into a temporary directory, runs `terraform init -backend=false` and `terraform validate` in each example, the ones the change touches first, and shows you a verdict, the findings and the comment it would post. It posts nothing until you confirm the comment and its review event: comment, approve or request changes.
  54. ## Everyday use
  55. Write requests in plain words. The examples show what each skill needs to hear.
  56. ### Review a pull request or a commit
  57. ```text
  58. review https://github.com/terraform-aws-modules/terraform-aws-lambda/pull/780
  59. is terraform-aws-modules/terraform-aws-lambda#780 ready to merge?
  60. what should I reply on terraform-aws-modules/terraform-aws-lambda#780?
  61. review https://github.com/terraform-aws-modules/terraform-aws-lambda/commit/<sha>
  62. ```
  63. You get a verdict, a comment with every finding and its suggested fix, the checks that could not run, and the state of the pull request's checks and threads. The skill proposes one review event (comment, approve or request changes) and asks once. Say "do not post" and it runs render-only: it shows the comment and posts nothing. A commit gets a plain commit comment with no approve or request changes.
  64. ### Triage first
  65. ```text
  66. triage terraform-aws-modules/terraform-aws-lambda#780
  67. ```
  68. A quick pass: scope and the checks that need no provider lookup, with no validation. It never approves. Use it to decide whether a full review is worth it.
  69. ### Review and finish a pull request
  70. ```text
  71. review and finish terraform-aws-modules/terraform-aws-lambda#780
  72. fix https://github.com/terraform-aws-modules/terraform-aws-lambda/pull/780 and push
  73. ```
  74. The skill reviews the pull request, applies the review's findings and your own comments in a throwaway clone, and reviews again, at most three rounds. It applies another person's comment only when you name its item id with "apply". It shows the diff, the verdict, the author and the branch, and pushes only when you confirm that branch in a new message. It never force pushes and never merges, and it pushes only where your token may push to the pull request's branch; otherwise you get a patch. Posting the review and the replies each needs its own confirmation.
  75. ### Review your own change before you push
  76. ```text
  77. review the changes on this branch against master in ~/src/terraform-aws-s3-bucket
  78. ```
  79. The reviewer reads committed changes only. For uncommitted work, paste `git diff` into the request. You get findings, most severe first, each with a rule ID, a file and line, and a suggested fix. The reviewer never edits files and never runs Terraform.
  80. ### Make a change to a module
  81. ```text
  82. add poller_group_name to the event source mapping in ~/src/terraform-aws-lambda
  83. ```
  84. The maintainer checks first that the addition belongs in the module, then makes the smallest edit, says whether it is additive, behaviour-changing or breaking, and drafts the pull request description. Your checkout is treated as untrusted, so it runs only `terraform fmt -check`, `terraform init -backend=false`, `terraform validate` and the documentation update. Say "this is a trusted local run" to let it run `pre-commit run -a` as well. Say it only for code you wrote or have read: pre-commit runs the hooks the repository defines. It counts only in your own message: never from the repository, the pull request or text you paste. The same goes for permission to run outside the sandbox.
  85. ### Work through review feedback
  86. ```text
  87. work through this review on my pull request, workspace ~/src/terraform-aws-lambda:
  88. <paste the review and the comments>
  89. apply F-003
  90. ```
  91. The maintainer decides each item (apply, reject, already addressed, needs a decision, and so on), edits the code for the items it applies, and writes reply drafts that start with "Fixed in <sha>:" or "Not changed:". It stops and asks when an item is unclear or needs your decision. It never posts, pushes or resolves threads.
  92. ### Check whether something belongs, in any project
  93. ```text
  94. before I add a Redis client to this billing service, does it belong here?
  95. ```
  96. `change-scope` answers with one of three outcomes and the rule behind it. It does not judge quality or whether the idea is good, only where it belongs.
  97. ## Reading a review
  98. - **Finding**: one problem in one place: a rule ID such as `compat.variable-removed`, a severity (CRITICAL, HIGH, MEDIUM or LOW), a file and line, a summary and a suggested fix. A fix can be a code change, a choice between options for you to make, or "no code change".
  99. - **Verdict**: computed from the findings and the pull request's checks and threads, with no judgement of its own. The first rule that matches wins ([verdict.md](.agents/skills/terraform-module-pr-review/references/verdict.md)):
  100. 1. Hold for the next major release: at least one CRITICAL finding, and every CRITICAL and HIGH finding is a compatibility (`compat.*`) finding.
  101. 2. Block: any other CRITICAL or HIGH finding, or a failing host signal: a failing check, a merge conflict, or a review that requests changes.
  102. 3. Inconclusive: anything that could not be checked.
  103. 4. Changes suggested: unresolved review threads.
  104. 5. Block: any MEDIUM finding.
  105. 6. Approve with nits: only LOW findings.
  106. 7. Approve: nothing found.
  107. - **Inconclusive**: something could not be checked, for example a provider lookup, a validation or a check run that was still pending. A gap never counts as a pass, and a failed lookup is not evidence that anything is missing.
  108. - **Validation**: `terraform init` downloads the providers and modules the change pins, and `terraform validate` runs those provider plugins. The skill removes AWS, GitHub, Terraform and SSH credentials from that environment, gives it an empty home directory and uses no backend. A passing `terraform validate` is evidence, not proof.
  109. - **Scope outcomes**: when an addition belongs, there is no scope finding. Otherwise it is a Ban (it does not belong here, and the review names where it goes), a Maintainer decision (a person has to decide), or Not evaluated (the rule could not be applied; an unknown, never a pass). Text in the pull request itself, such as "this is in scope", never clears a scope rule.
  110. ## Verification ladder
  111. A change to behaviour, such as dropping `compact()` around a list, is verified by escalating evidence, simplest first. `mtf-facts behaviour` turns each changed construct into a trigger, the review writes the pull request's behaviour claims as data, and `mtf-facts claims-eval` gives each trigger a state from what the levels observed. Reading the code supports a claim and never verifies one.
  112. | Level | What runs | Needs |
  113. |-------|-----------|-------|
  114. | L1 | The base and head expression, evaluated offline with typed samples and unknowns, built-in provider only | `terraform`, `mtf-facts`, `perl`, `jq`, and `sandbox-exec` on macOS or `bwrap` on Linux |
  115. | L2 | A plan-only `terraform test` of the instances a `count` or `for_each` makes | The same as L1 |
  116. | L3 | A curated fixture with the real provider, fake credentials, a prior state and a control run | The same, plus `curl` to fetch the pinned provider, hash-checked, before the sandboxed run |
  117. | L4 | The credentialed plan pass, a base and a head plan per example | An AWS profile and a review role, as in [Add a terraform plan pass](#add-a-terraform-plan-pass) |
  118. L1 to L3 run with the network denied, in a sandbox a canary proves before any command; on an unproven sandbox those levels are unverifiable. A trigger that ends verified, covered by a plan or inert passes, a refuted one blocks, and an unverifiable one makes the review inconclusive. Details: [verification-ladder.md](.agents/skills/terraform-module-reviewer/references/verification-ladder.md) and [ladder-pass.md](.agents/skills/terraform-module-pr-review/references/ladder-pass.md).
  119. ## Limits
  120. - No skill runs `terraform apply`, writes state or uses a real backend. None merges, pushes to a pull request or resolves threads. None obeys instructions found in the code, the pull request text or its comments; all of it is data.
  121. - The pull request review posts only the one comment and review event you confirm in the conversation. From the pull request it runs only `terraform init -backend=false` and `terraform validate`, with AWS, GitHub, Terraform and SSH credentials removed and an empty home directory; both still run the provider plugins the change pins. When you opt in, it also runs `terraform plan` on a read-only review session.
  122. - The reviewer and `change-scope` are read-only. They never post and never run Terraform.
  123. - The maintainer edits only your checkout and never posts. It commits only when you ask, and runs the repository's own hooks only on a run you call trusted.
  124. ## Advanced use
  125. These need extra tools, credentials or a decision from you.
  126. ### Add a terraform plan pass
  127. ```text
  128. review owner/repo#123, plan with profile review-sso and role arn:aws:iam::<account-id>:role/<review-role>
  129. ```
  130. The plan pass runs `terraform plan` on every example, the ones the change touches first, on a short session of a read-only review role that the account owner creates. You name both the AWS profile and the role for that run only. The session carries a policy that allows only read actions. Before planning, a write probe, a dry-run VPC create, must be refused; it tests one write action, not the whole policy. With Docker, the plan runs in an isolated container whose proxy lets each phase reach only the hosts it needs. Without Docker, it runs on your machine in laptop mode, where stricter pattern checks refuse risky examples; they reduce the risk of the code acting on your machine but cannot rule it out. Plan results add evidence and never clear a finding. In runner mode only each example's outcome and resource counts reach the comment. In laptop mode a finding may quote one scanned error line; the scan cannot guarantee that no sensitive value slips through. It needs `terraform`, `jq`, `perl`, `git` and AWS CLI v2. Details: [plan-pass.md](.agents/skills/terraform-module-pr-review/references/plan-pass.md) and the [plan runner](.agents/skills/terraform-module-pr-review/references/plan-runner/README.md), which is a prototype.
  131. ### Create a new module
  132. ```text
  133. create a terraform-aws-<service> module covering <resource types>
  134. ```
  135. The maintainer builds a coverage ledger from the provider schema (every argument becomes an input, an output, or a documented omission), copies the profile's templates and writes the examples. It counts as complete only after a trusted run of `pre-commit` and a `terraform plan` with credentials you choose.
  136. ### Upgrade a provider major version
  137. ```text
  138. upgrade ~/src/terraform-aws-rds to AWS provider 6.x
  139. ```
  140. The maintainer compares the old and new provider schemas, makes the smallest edit plus the version constraint bump, runs `terraform init -backend=false -upgrade`, and writes migration notes in `docs/MIGRATION_<from>_to_<to>.md` for anything that breaks consumers.
  141. ### mtf-facts
  142. [`mtf-facts`](https://github.com/modulestf/facts) is a command-line tool that reads the files alone, with no network and no Terraform. Keep it on your `PATH`, outside the module checkout. A pull request review needs it: it picks the examples to verify and classifies the change for the [verification ladder](#verification-ladder). Without it the classification cannot run, so the review is inconclusive at best. The reviewer also runs its `lint` once and ends the review with a short comparison of where the tool and the review agree. The maintainer can also use its `lint`, `ledger` and `schema-diff` commands. The facts repository has a [walkthrough](https://github.com/modulestf/facts#example-review-a-pull-request) on a real pull request. Design: [docs/facts-engine-design.md](docs/facts-engine-design.md).
  143. ### Review a whole module, not a change
  144. Tell the reviewer that the checkout holds a new module, for example one offered for adoption. It runs a full coverage pass with no base to compare against.
  145. ### Profiles
  146. A profile holds the rules of one module family. Two ship with the skills:
  147. - [`terraform-aws-modules`](.agents/skills/terraform-module-maintainer/profiles/terraform-aws-modules/PROFILE.md), the default: published modules with examples, released from commit titles.
  148. - [`in-repo-modules`](.agents/skills/terraform-module-maintainer/profiles/in-repo-modules/PROFILE.md): in-house modules whose only callers are root configurations in the same repository, with no examples and no releases. A break whose callers all change in the same pull request is MEDIUM, not CRITICAL.
  149. Pick one with `profile: in-repo-modules` in your request, or name your own with `profile-dir: /absolute/path`. A custom profile lives outside any repository. [profile-format.md](.agents/skills/terraform-module-maintainer/references/profile-format.md) says how to make, verify and update one, and which rules each option turns off. The review prints the profile it used and a hash of its files. A module can record its own scope exceptions in `docs/SCOPE.md`; the review reads that file from the base, so an exception must be merged before the change that needs it. To hand the profile to another code reviewer, pin [review-guide.md](.agents/skills/terraform-module-maintainer/profiles/terraform-aws-modules/review-guide.md) by commit; it gives advice, not a verdict.
  150. ### Hosted adapters
  151. A host that runs the review's verify job, plan job, records and render itself pins one commit of this repository and moves them together. This release changes the files they exchange, with no fallback to the older shapes: the render's run file requires `coverage`, and `verify.json` and `plan-hosted.json` gained keys. See [host-pass.md](.agents/skills/terraform-module-pr-review/references/host-pass.md#compatibility).
  152. ### Checks that need permission
  153. When a check cannot run inside your agent's sandbox, the review records it as not run and the verdict is inconclusive at best. Only your own message can allow a run outside the sandbox, for that run or a named set of runs.
  154. ## Requirements by task
  155. | Task | Needs |
  156. |------|-------|
  157. | Every Terraform skill | The five skill directories side by side, a Terraform MCP server |
  158. | Review a pull request or commit | `gh` logged in as a person (a bot or CI identity runs render-only and posts nothing), `git`, `jq`, `perl`, `terraform`, `mtf-facts`, and `sandbox-exec` (macOS) or `bwrap` (Linux) for the ladder |
  159. | Review and finish a pull request | Everything a pull request review needs, `gh` logged in to github.com with a token that can push to the pull request's branch, and `user.name` and `user.email` in your global git configuration |
  160. | Review a local change | `git` |
  161. | Change, fix or upgrade a module | `terraform`; for a trusted run also `pre-commit`, `terraform-docs` and `tflint` |
  162. | Create a module | The same. To finish it: a trusted run and a `terraform plan` you run with your own credentials, not a review role |
  163. | Plan pass | An AWS profile and a review role, AWS CLI v2, `jq`, `perl`; Docker for the isolated runner |
  164. | `change-scope` alone | Nothing else |
  165. | `mtf-facts` comparison in a local review | The `mtf-facts` binary on `PATH`, outside the checkout |
  166. ## Related repositories
  167. - [modulestf/facts](https://github.com/modulestf/facts): `mtf-facts`, the facts engine a pull request review needs.
  168. - [modulestf/terraform-schema-mirror](https://github.com/modulestf/terraform-schema-mirror): provider schemas, every stable version, as release assets.
  169. - [antonbabenko/terraform-skill](https://github.com/antonbabenko/terraform-skill): general Terraform practices. The skills here link to it and do not repeat it.
  170. ## Tests
  171. The tests are shell scripts under `tests/`. Run them all with:
  172. ```bash
  173. for t in tests/*.sh; do bash "$t" || echo "FAILED: $t"; done
  174. ```
  175. What each test checks and what it needs is in [AGENTS.md](AGENTS.md#tests).
  176. ## Contributing and license
  177. Issues are welcome; see [CONTRIBUTING.md](CONTRIBUTING.md). Report security issues as [SECURITY.md](SECURITY.md) says. Licensed under the [Apache License 2.0](LICENSE).
  178. ## History
  179. This repository held modules.tf-lambda, the Cloudcraft-to-Terraform generator (2019-2021). Its code is on the `legacy` branch.
.agents/skills/terraform-module-pr-review/references/plan-runner/smoke.sh · Bundled script · 345 lines · 1 finding marked
  1. #!/bin/bash
  2. # Smoke test of the isolated plan runner. Needs a local Docker runtime and network access
  3. # for the image builds; uses no AWS account and no real credentials.
  4. #
  5. # It plans smoke/example with made-up session credentials and probes the network after
  6. # each phase. Success means:
  7. # - init reaches the registry and download hosts, and not AWS;
  8. # - plan reaches STS through the plan-phase proxy and STS rejects the made-up key
  9. # (InvalidClientTokenId), which proves the path to AWS without any real credential;
  10. # - during plan the registry, GitHub, EC2 public DNS names and S3 bucket hosts are refused;
  11. # - a direct connection that bypasses the proxy fails, and outside names do not resolve;
  12. # - plain HTTP through the proxy is refused;
  13. # - credentials without a session token are refused before anything starts;
  14. # - an oversized file in the output directory is refused;
  15. # - with --base, only the example whose plan is a code-error is re-run at the merge base;
  16. # - run-pass.sh's network check, behind each proxy, finds every path closed and both positive
  17. # controls held, and stops the run before any session when IPv6 is on, a path is open or a
  18. # control fails;
  19. # - no credential value is in the configuration of a plan container (docker inspect);
  20. # - the test pass runs a project's unit tests behind the test proxy, which passes the package
  21. # index and refuses GitHub and AWS, and records passed or failed with no test output;
  22. # - the image tag is derived from the image files unless PLAN_RUNNER_TAG sets it;
  23. # - no container, network or volume of these runs is left behind.
  24. set -uo pipefail
  25. here="$(cd "$(dirname "$0")" && pwd)"
  26. rec="$(mktemp -d "${TMPDIR:-/tmp}/plan-runner-smoke.XXXXXX")"
  27. creds="$rec/fake-credentials.env"
  28. printf '%s\n' AWS_ACCESS_KEY_ID=smoke-test-not-a-key AWS_SECRET_ACCESS_KEY=smoke-test-not-a-secret \
  29. AWS_SESSION_TOKEN=smoke-test-not-a-token > "$creds"
  30. static="$rec/static-credentials.env"
  31. printf '%s\n' AWS_ACCESS_KEY_ID=smoke-test-not-a-key AWS_SECRET_ACCESS_KEY=smoke-test-not-a-secret > "$static"
  32. "$here/run-plan.sh" --build --probe --clone "$here/smoke" --example example \
  33. --record "$rec/record" --credentials-env-file "$creds" > /dev/null
  34. "$here/run-plan.sh" --clone "$here/smoke" --example example \
  35. --record "$rec/static" --credentials-env-file "$static" > "$rec/static.out" 2>&1
  36. static_rc=$?
  37. "$here/run-plan.sh" --clone "$here/smoke" --example attack --record "$rec/attack" \
  38. > "$rec/attack.out" 2>&1
  39. attack_rc=$?
  40. fails=0
  41. check() { # description, command...
  42. local d="$1"; shift
  43. if "$@" > /dev/null 2>&1; then echo "ok $d"; else echo "FAIL $d"; fails=$((fails + 1)); fi
  44. }
  45. r="$rec/record"
  46. check "init succeeded" grep -qx 'init 0' "$r/status"
  47. check "init downloaded the provider through the proxy" grep -q 'CONNECT releases.hashicorp.com:443' "$r/proxy-init.log"
  48. check "init phase refuses AWS" grep -qx 'proxy sts.us-east-1.amazonaws.com: refused' "$r/probe-init.log"
  49. check "init phase passes GitHub" grep -q '^proxy github.com: passed' "$r/probe-init.log"
  50. check "plan reached STS through the plan proxy" grep -q 'CONNECT sts\.[a-z0-9-]*\.amazonaws\.com:443' "$r/proxy-plan.log"
  51. check "STS rejected the made-up key" grep -q 'InvalidClientTokenId' "$r/plan.log"
  52. check "plan phase refuses the registry" grep -qx 'proxy registry.terraform.io: refused' "$r/probe-plan.log"
  53. check "plan phase refuses GitHub" grep -qx 'proxy github.com: refused' "$r/probe-plan.log"
  54. check "plan phase refuses an EC2 public DNS name" grep -qx 'proxy ec2-192-0-2-1.compute-1.amazonaws.com: refused' "$r/probe-plan.log"
  55. check "plan phase refuses an S3 bucket host" grep -qx 'proxy probe-bucket.s3.amazonaws.com: refused' "$r/probe-plan.log"
  56. check "plan phase refuses a regional S3 bucket host" grep -qx 'proxy probe-bucket.s3.us-east-1.amazonaws.com: refused' "$r/probe-plan.log"
  57. check "plan phase refuses a host outside AWS" grep -qx 'proxy example.com: refused' "$r/probe-plan.log"
  58. check "refused plan hosts are listed" grep -qx 'github.com:443' "$r/plan-refused-hosts.txt"
  59. check "plan phase refuses a CONNECT to port 8443" grep -qx 'proxy sts.us-east-1.amazonaws.com:8443: refused' "$r/probe-plan.log"
  60. check "the port 8443 attempt is listed" grep -qx 'sts.us-east-1.amazonaws.com:8443' "$r/plan-refused-hosts.txt"
  61. for phase in init plan; do
  62. check "$phase: direct connection to an address fails" grep -qx 'direct 1.1.1.1:443: closed' "$r/probe-$phase.log"
  63. check "$phase: direct connection to an allowed host fails" grep -qx 'direct registry.terraform.io:443: closed' "$r/probe-$phase.log"
  64. check "$phase: outside names do not resolve" grep -qx 'dns example.com: does not resolve' "$r/probe-$phase.log"
  65. check "$phase: plain HTTP through the proxy is refused" grep -qx 'proxy plain-http registry.terraform.io: exit 0 http 403' "$r/probe-$phase.log"
  66. done
  67. check "credentials without a session token are refused" test "$static_rc" -eq 2
  68. check "the refusal says why" grep -q 'no session token' "$rec/static.out"
  69. a="$rec/attack"
  70. check "attack: the run completed" test "$attack_rc" -eq 0
  71. check "attack: the planted status link was refused" grep -qx 'not a regular file in the work volume, not exported: out/status' "$a/export-refused.txt"
  72. check "attack: the planted plan.log link was refused" grep -qx 'not a regular file in the work volume, not exported: out/plan.log' "$a/export-refused.txt"
  73. check "attack: the record holds no link" test -z "$(find "$a" "$r" -type l -print -quit)"
  74. check "attack: the record holds no password-file content" test -z "$(grep -rl -e 'root:' -e 'User Database' "$a" 2>/dev/null)"
  75. check "attack: the unlisted planted name was not exported" test ! -e "$a/extra"
  76. check "attack: an oversized file was refused" grep -qx 'larger than 5242880 bytes, not exported: out/probe-init.log' "$a/export-refused.txt"
  77. check "attack: the oversized file is not in the record" test ! -e "$a/probe-init.log"
  78. # The plan pass in the runner, on a git fixture holding both smoke examples and one whose init
  79. # fails the same way at the head and at the merge base: a code-error that needs no credentials.
  80. # The fixture is its own origin, so the base fetch stays inside the work volume. The made-up
  81. # credentials fail the pass's sts check, so the pass ends at credentials after init. Ten more
  82. # copies of the failing example make thirteen, past the laptop's cap of twelve, which the
  83. # runner does not have.
  84. repo="$rec/pass-repo"
  85. mkdir -p "$repo/examples/broken"
  86. printf 'terraform {\n required_version = "= 0.0.1"\n}\n' > "$repo/examples/broken/main.tf"
  87. git -C "$repo" init -q
  88. git -C "$repo" remote add origin .
  89. git -C "$repo" add -A
  90. git -C "$repo" -c user.name=smoke -c user.email=smoke@example.com commit -q -m base
  91. basesha="$(git -C "$repo" rev-parse HEAD)"
  92. cp -R "$here/smoke/example" "$repo/examples/sts"
  93. cp -R "$here/smoke/attack" "$repo/examples/attack"
  94. git -C "$repo" add -A
  95. git -C "$repo" -c user.name=smoke -c user.email=smoke@example.com commit -q -m head
  96. mkdir -m 0700 "$rec/pass-tmp"
  97. # A docker wrapper for these runs only: before run-pass.sh removes a runner container, it saves
  98. # that container's docker inspect output, so the check below sees every plan container. With
  99. # $rec/shim-mode it also fakes a fault for the network check: "ipv6" answers that IPv6 is on
  100. # for the run networks, "open-path" points the probe's gateway at the proxy, which listens,
  101. # and "plan-control" gives the plan-phase probe a positive control its proxy refuses.
  102. mkdir "$rec/shim"
  103. cat > "$rec/shim/docker" <<'SHIM'
  104. #!/bin/bash
  105. mode="$(cat "$SMOKE_REC/shim-mode" 2> /dev/null)"
  106. if [ "$1 $2" = "rm -f" ]; then
  107. case "$3" in plan-runner-*) "$SMOKE_DOCKER" container inspect "$3" >> "$SMOKE_REC/inspect.json" 2> /dev/null || true ;; esac
  108. fi
  109. if [ "$mode" = ipv6 ] && [ "$1 $2 $3 $4" = "network inspect -f {{.EnableIPv6}}" ]; then
  110. echo true; echo true; exit 0
  111. fi
  112. if [ "$mode" = open-path ] && [ "$1" = run ]; then
  113. args=() prev=""
  114. for a in "$@"; do
  115. if [ "$prev" = --gateway ]; then args+=(proxy); else args+=("$a"); fi
  116. prev="$a"
  117. done
  118. exec "$SMOKE_DOCKER" "${args[@]}"
  119. fi
  120. if [ "$mode" = plan-control ] && [ "$1" = run ]; then
  121. args=() prev=""
  122. for a in "$@"; do
  123. if [ "$prev" = --allowed ] && [ "$a" = sts.us-east-1.amazonaws.com ]; then args+=(example.com); else args+=("$a"); fi
  124. prev="$a"
  125. done
  126. exec "$SMOKE_DOCKER" "${args[@]}"
  127. fi
  128. exec "$SMOKE_DOCKER" "$@"
  129. SHIM
  130. chmod +x "$rec/shim/docker"
  131. export SMOKE_REC="$rec" SMOKE_DOCKER="$(command -v docker)"
  132. PATH="$rec/shim:$PATH" "$here/run-pass.sh" --clone "$repo" --record "$rec/pass" --tmp "$rec/pass-tmp" --credentials-env-file "$creds" \
  133. --base "$basesha" examples/broken examples/sts examples/attack examples/broken examples/broken \
  134. examples/broken examples/broken examples/broken examples/broken examples/broken examples/broken \
  135. examples/broken examples/broken > /dev/null 2> "$rec/pass.err"
  136. p="$rec/pass"
  137. for ph in init plan; do
  138. nc="$p/network-check-$ph.txt"
  139. check "pass: $ph: the network check found every path closed and both controls held" \
  140. test "$(tail -n 1 "$nc")" = "end network check" -a "$(grep -vcE '^(closed|ok) ' "$nc")" -eq 1 -a "$(grep -c '^ok control ' "$nc")" -eq 2
  141. check "pass: $ph: the proxy was reached directly" grep -qx 'ok control direct proxy:8888' "$nc"
  142. check "pass: $ph: the probe saw one network interface" grep -qx 'closed interfaces: one up' "$nc"
  143. check "pass: $ph: the probe saw no IPv4 default route" grep -qx 'closed route ipv4: no default' "$nc"
  144. check "pass: $ph: an IP-literal CONNECT through the proxy is refused" grep -qx 'closed proxy CONNECT 1.1.1.1:443' "$nc"
  145. check "pass: $ph: external names do not resolve" grep -qx 'closed dns example.com' "$nc"
  146. check "pass: $ph: the gateway, docker0 and the IPv4 metadata address are closed on six ports each" \
  147. test "$(grep -cE '^closed direct [0-9.]+:(22|80|443|2375|2376|8888)$' "$nc")" -ge 19
  148. check "pass: $ph: the IPv6 metadata address is closed" grep -qx 'closed direct fd00:ec2::254:80' "$nc"
  149. done
  150. check "pass: init: the init proxy passes the registry" grep -qx 'ok control proxy CONNECT registry.terraform.io:443' "$p/network-check-init.txt"
  151. check "pass: init: the init proxy refuses AWS" grep -qx 'closed proxy CONNECT sts.us-east-1.amazonaws.com:443' "$p/network-check-init.txt"
  152. check "pass: plan: the plan proxy passes an AWS endpoint" grep -qx 'ok control proxy CONNECT sts.us-east-1.amazonaws.com:443' "$p/network-check-plan.txt"
  153. check "pass: plan: the plan proxy refuses the registry" grep -qx 'closed proxy CONNECT registry.terraform.io:443' "$p/network-check-plan.txt"
  154. check "pass: init phase readied both examples" grep -qx 'init: examples/attack: ready' "$p/plan-pass-init.txt"
  155. check "pass: plan phase checked the credentials through the plan proxy" grep -q 'CONNECT sts\.[a-z0-9-]*\.amazonaws\.com:443' "$p/proxy-plan.log"
  156. check "pass: made-up credentials end the pass" grep -qx 'plan: plan pass ended: credentials' "$p/plan-pass.txt"
  157. check "pass: the host heads each example's block" grep -qx 'host example 2: examples/sts' "$p/plan-pass.txt"
  158. check "pass: the summary line is written" grep -qx 'plan summary: 13 examples, 0 planned, 0 exceptions, 13 other' "$p/plan-pass.txt"
  159. check "pass: the budget is recorded" grep -qx 'plan pass budget: 120 minutes' "$p/plan-pass.txt"
  160. check "pass: no example cap in the runner" test "$(grep -c '^init: examples/broken: code-error$' "$p/plan-pass-init.txt")" -eq 11
  161. check "pass: only the code-error was prepared at the base" test "$(grep -c ' at base: ' "$p/plan-pass-init.txt")" -eq 1
  162. check "pass: the code-error was initialised at the base" grep -qx 'init: examples/broken at base: code-error' "$p/plan-pass-init.txt"
  163. check "pass: the deferred base line was filled in" grep -qx 'base: reproduces at base' "$p/plan-pass.txt"
  164. check "pass: no deferred base line is left" test -z "$(grep -x 'base: deferred' "$p/plan-pass.txt")"
  165. # The init-phase network check asks the init proxy for STS on purpose, to see it refused. So
  166. # what must hold is that no AWS endpoint was reached: tinyproxy wrote no "Established
  167. # connection" line for an amazonaws host, and every request for one has its own "Proxying
  168. # refused on filtered url" line, the same targets the same number of times.
  169. no_aws_reached() { # proxy log
  170. local log
  171. log="$(tr 'A-Z' 'a-z' < "$1")" || return 1
  172. ! printf '%s\n' "$log" | grep 'established connection to host "[^"]*amazonaws' > /dev/null || return 1
  173. [ "$(printf '%s\n' "$log" | sed -nE 's/.*request \(file descriptor [0-9]+\): [a-z]+ ([^ ]*amazonaws[^ ]*) http.*/\1/p' | sort)" = \
  174. "$(printf '%s\n' "$log" | sed -nE 's/.*proxying refused on filtered url "([^"]*amazonaws[^"]*)".*/\1/p' | sort)" ]
  175. }
  176. check "pass: init reached no AWS endpoint" no_aws_reached "$p/proxy-init.log"
  177. check "pass: the init proxy refused the network check's STS request" grep -q 'Proxying refused on filtered url "sts.us-east-1.amazonaws.com:443"' "$p/proxy-init.log"
  178. check "pass: no credential value in the record" test -z "$(grep -rl 'smoke-test-not-a-' "$p")"
  179. check "pass: a plan container was inspected with the credentials file mounted" grep -q '/opt/plan-pass/creds.env' "$rec/inspect.json"
  180. check "pass: no credential value in a plan container's configuration" test -z "$(grep 'smoke-test-not-a-' "$rec/inspect.json")"
  181. # The plan container's credentials loader, the exact line from run-pass.sh, run in the
  182. # runner image on values with a trailing '=' pad and an internal '='.
  183. loader="$(sed -n "s/^load_creds='\(.*\)' # creds-loader$/\1/p" "$here/run-pass.sh")"
  184. printf '%s\n' PLAN_PASS_AWS_ACCESS_KEY_ID=smoke-key PLAN_PASS_AWS_SECRET_ACCESS_KEY=smoke=sec=ret \
  185. PLAN_PASS_AWS_SESSION_TOKEN=smoke-token= OTHER=x > "$rec/loader.env"
  186. chmod 0644 "$rec/loader.env"
  187. loaded="$(docker run --rm --network none --read-only --cap-drop ALL --user 10001:10001 \
  188. -v "$rec/loader.env":/opt/plan-pass/creds.env:ro --entrypoint /bin/bash \
  189. "plan-runner:$("$here/image-tag.sh")" -c "$loader && env | grep -E '^(PLAN_PASS_|OTHER=)' | sort" 2>&1)"
  190. check "loader: the line is found in run-pass.sh" test -n "$loader"
  191. check "loader: a value keeps its trailing pad and internal '='" test "$loaded" = "$(printf '%s\n' \
  192. PLAN_PASS_AWS_ACCESS_KEY_ID=smoke-key PLAN_PASS_AWS_SECRET_ACCESS_KEY=smoke=sec=ret PLAN_PASS_AWS_SESSION_TOKEN=smoke-token=)"
  193. check "pass: the record holds regular files only" test -z "$(find "$p" -mindepth 1 ! -type f -print -quit)"
  194. check "pass: nothing is left under --tmp" test -z "$(find "$rec/pass-tmp" -mindepth 1 -print -quit)"
  195. # The network check fails closed: IPv6 on for a run network, or an open path, stops the run
  196. # before any head code runs and before the first session, with no plan recorded; a plan-phase
  197. # control that fails stops it after init, before the first plan container.
  198. for m in ipv6 open-path plan-control; do
  199. echo "$m" > "$rec/shim-mode"
  200. PATH="$rec/shim:$PATH" "$here/run-pass.sh" --clone "$repo" --record "$rec/net-$m" --tmp "$rec/pass-tmp" \
  201. --credentials-env-file "$creds" examples/sts > /dev/null 2> "$rec/net-$m.err"
  202. echo $? > "$rec/net-$m.rc"
  203. done
  204. rm -f "$rec/shim-mode"
  205. check "network: IPv6 on stops the run" test "$(cat "$rec/net-ipv6.rc")" -eq 2
  206. check "network: the IPv6 refusal says why" grep -qx 'run-pass: refusing to plan: IPv6 is on for a run network' "$rec/net-ipv6.err"
  207. check "network: IPv6 on runs no probe and no plan" test ! -e "$rec/net-ipv6/network-check-init.txt" -a ! -e "$rec/net-ipv6/plan-pass.txt"
  208. check "network: an open path stops the run" test "$(cat "$rec/net-open-path.rc")" -eq 2
  209. check "network: the refusal says why" grep -qx 'run-pass: refusing to plan: the network check found an open path, or could not run' "$rec/net-open-path.err"
  210. check "network: the probe names the open path" grep -qx 'open direct proxy:8888' "$rec/net-open-path/network-check-init.txt"
  211. check "network: an open path runs no head code" test ! -e "$rec/net-open-path/plan-pass-init.txt"
  212. check "network: an open path runs no plan container and records no plan" test ! -e "$rec/net-open-path/proxy-plan.log" -a ! -e "$rec/net-open-path/plan-pass.txt"
  213. q="$rec/net-plan-control"
  214. check "network: a failed plan-phase control stops the run" test "$(cat "$q.rc")" -eq 2
  215. check "network: the plan-phase refusal says why" grep -qx 'run-pass: refusing to plan: the network check found an open path, or could not run' "$q.err"
  216. check "network: the plan-phase probe names the failed control" grep -qx 'fail control proxy CONNECT example.com:443: refused' "$q/network-check-plan.txt"
  217. check "network: the init phase ran before it" grep -qx 'init: examples/sts: ready' "$q/plan-pass-init.txt"
  218. check "network: a failed plan-phase control records no plan" test ! -e "$q/plan-pass.txt"
  219. # --tmp is where the private temporary directory goes: one that cannot be written to stops
  220. # the run before any credential is read or any Docker object is made.
  221. mkdir -m 0500 "$rec/ro-tmp"
  222. "$here/run-pass.sh" --clone "$repo" --record "$rec/ro-pass" --tmp "$rec/ro-tmp" --credentials-env-file "$creds" \
  223. examples/sts > /dev/null 2>&1
  224. ro_rc=$?
  225. check "pass: an unwritable --tmp stops the run" test "$ro_rc" -ne 0
  226. check "pass: an unwritable --tmp made no Docker object" test ! -e "$rec/ro-pass/run-id.txt"
  227. # A profile with no review role stops the run before any credential is read or any Docker
  228. # object is made: there is no plan pass on the profile's own credentials.
  229. "$here/run-pass.sh" --clone "$repo" --record "$rec/norole" --tmp "$rec/pass-tmp" --profile smoke \
  230. examples/sts > "$rec/norole.out" 2>&1
  231. norole_rc=$?
  232. check "pass: a profile with no review role stops the run" test "$norole_rc" -eq 2
  233. check "pass: the refusal names the review role" grep -q 'needs --role-arn' "$rec/norole.out"
  234. check "pass: a profile with no review role made no Docker object" test ! -e "$rec/norole/run-id.txt"
  235. # The host leak scan, on a crafted record: exact values and every evidence pattern go, an
  236. # ordinary line stays.
  237. s="$rec/scan"
  238. mkdir -p "$s"
  239. {
  240. echo 'plan: planned | Plan: 1 to add, 0 to change, 0 to destroy.'
  241. echo 'key AKIAABCDEFGHIJKLMNOP'
  242. echo 'jwt eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIxIn0'
  243. echo '-----BEGIN PRIVATE KEY-----'
  244. echo 'url ?X-Amz-Signature=abc'
  245. echo 'Authorization: AWS4-HMAC-SHA256'
  246. echo 'encoded c21va2UtdGVzdC1ub3QtYS1zZWNyZXQtdmFsdWUtZW5jb2RlZA=='
    • Dismissed after review: Contains a long base64-looking blob — not necessarily malicious, but worth inspecting what it decodes to
    • Line 252 on GitHub
  247. echo 'value smoke-test-exact-value'
  248. echo 'split smoke-test-exa'
  249. echo 'ct-value across two lines'
  250. echo 'id AKIAABCDEF'
  251. echo ' GHIJKLMNOP split too'
  252. echo 'scope: examples/sts'
  253. } > "$s/plan-pass.txt"
  254. echo 'smoke-test-exact-value' > "$rec/scan-values"
  255. scanned="$("$here/record-scan.sh" "$s" "$rec/scan-values")"
  256. check "scan: every crafted leak line is removed, split ones included" test "$scanned" -eq 11
  257. check "scan: the line after a split secret stays" grep -qx 'scope: examples/sts' "$s/plan-pass.txt"
  258. check "scan: the ordinary line stays" grep -qx 'plan: planned | Plan: 1 to add, 0 to change, 0 to destroy.' "$s/plan-pass.txt"
  259. check "scan: no crafted secret is left" test -z "$(grep -E 'AKIA|GHIJKLMNOP|eyJ|BEGIN|Signature|Authorization|c21va2|exact-value|smoke-test-exa|ct-value' "$s/plan-pass.txt")"
  260. # The test pass (../test-pass.md): a passing Pipfile project and a failing requirements-dev.txt
  261. # one, each a git archive as review-run.sh makes it, behind the test proxy, with no credential.
  262. trepo="$rec/tests-repo"
  263. mkdir -p "$trepo"
  264. cp -R "$here/smoke/tests-pass" "$here/smoke/tests-fail" "$trepo/"
  265. git -C "$trepo" init -q
  266. git -C "$trepo" add -A
  267. git -C "$trepo" -c user.name=smoke -c user.email=smoke@example.com commit -q -m tests
  268. git -C "$trepo" archive --format=tar -o "$rec/tests-pass.tar" HEAD:tests-pass
  269. git -C "$trepo" archive --format=tar -o "$rec/tests-fail.tar" HEAD:tests-fail
  270. "$here/run-tests.sh" --archive "$rec/tests-pass.tar" --kind pipenv --python python3 --record "$rec/tp" \
  271. --tmp "$rec/pass-tmp" > "$rec/tp.out" 2>&1
  272. "$here/run-tests.sh" --archive "$rec/tests-fail.tar" --kind requirements --python python3.12 --record "$rec/tf" \
  273. --budget-minutes 5 > "$rec/tf.out" 2>&1
  274. t="$rec/tp" tnc="$rec/tp/network-check-test.txt"
  275. # Only the package index hosts had a connection established through the test proxy.
  276. index_only() { # proxy log
  277. local hosts
  278. hosts="$(tr 'A-Z' 'a-z' < "$1" | sed -nE 's/.*established connection to host "([^"]*)".*/\1/p' | sort -u)" || return 1
  279. [ -n "$hosts" ] && [ -z "$(printf '%s\n' "$hosts" | grep -vxE 'pypi\.org|files\.pythonhosted\.org')" ]
  280. }
  281. check "tests: a passing Pipfile project is passed 2" grep -qx 'passed 2' "$t/tests.txt"
  282. check "tests: the script prints the result" grep -qx 'passed 2' "$rec/tp.out"
  283. check "tests: a failing test under requirements-dev.txt and Python 3.12 is failed 1" grep -qx 'failed 1' "$rec/tf/tests.txt"
  284. check "tests: the network check found every path closed and both controls held" \
  285. test "$(tail -n 1 "$tnc")" = "end network check" -a "$(grep -vcE '^(closed|ok) ' "$tnc")" -eq 1 -a "$(grep -c '^ok control ' "$tnc")" -eq 2
  286. check "tests: the test proxy passes the package index" grep -qx 'ok control proxy CONNECT pypi.org:443' "$tnc"
  287. check "tests: the test proxy refuses GitHub" grep -qx 'closed proxy CONNECT github.com:443' "$tnc"
  288. check "tests: the test proxy refuses AWS" grep -qx 'closed proxy CONNECT sts.us-east-1.amazonaws.com:443' "$tnc"
  289. check "tests: the install reached files.pythonhosted.org through the proxy" grep -q 'Established connection to host "files.pythonhosted.org"' "$t/proxy-test.log"
  290. check "tests: no host beyond the package index was reached" index_only "$t/proxy-test.log"
  291. check "tests: the record holds its four files, regular, and no test output" \
  292. test "$(cd "$t" && find . -type f | LC_ALL=C sort | tr '\n' ' ')" = "./network-check-test.txt ./proxy-test.log ./run-id.txt ./tests.txt " -a -z "$(find "$t" -mindepth 1 ! -type f -print -quit)"
  293. check "tests: run-tests.sh passes no credential to a container" test -z "$(grep -nE -- '--env-file|AWS_|creds' "$here/run-tests.sh")"
  294. check "tests: nothing is left under --tmp" test -z "$(find "$rec/pass-tmp" -mindepth 1 -print -quit)"
  295. # The image tag: derived from the image files unless PLAN_RUNNER_TAG is set, and refused
  296. # when it is not a valid tag.
  297. tag1="$(env -u PLAN_RUNNER_TAG "$here/image-tag.sh")"
  298. tag2="$(env -u PLAN_RUNNER_TAG "$here/image-tag.sh")"
  299. check "image tag: the default is src- and 12 hex digits" test -n "$(printf '%s\n' "$tag1" | grep -Ex 'src-[0-9a-f]{12}')"
  300. check "image tag: the default is the same on every call" test "$tag1" = "$tag2"
  301. check "image tag: PLAN_RUNNER_TAG overrides it" test "$(PLAN_RUNNER_TAG=smoke-x "$here/image-tag.sh")" = smoke-x
  302. check "image tag: an invalid PLAN_RUNNER_TAG is refused" test "$(PLAN_RUNNER_TAG='a b' "$here/image-tag.sh" > /dev/null 2>&1; echo $?)" -eq 2
  303. # Leftovers: each run names everything it creates with its own id, recorded in run-id.txt.
  304. # Only those exact names are inspected here; nothing is removed, and no prefix is matched, so
  305. # another run on the same Docker daemon is neither touched nor counted.
  306. leftover() { # ids...: prints each of those runs' containers, networks and volumes still present
  307. local id n
  308. for id in "$@"; do
  309. for n in "plan-runner-$id" "plan-proxy-$id" "plan-proxy-init-$id" "plan-proxy-plan-$id" "plan-export-$id"; do
  310. docker container inspect "$n" > /dev/null 2>&1 && echo "container $n"
  311. done
  312. for n in "plan-int-$id" "plan-ext-$id"; do
  313. docker network inspect "$n" > /dev/null 2>&1 && echo "network $n"
  314. done
  315. docker volume inspect "plan-work-$id" > /dev/null 2>&1 && echo "volume plan-work-$id"
  316. docker volume inspect "plan-saved-$id" > /dev/null 2>&1 && echo "volume plan-saved-$id"
  317. for n in "test-runner-$id" "test-proxy-$id"; do
  318. docker container inspect "$n" > /dev/null 2>&1 && echo "container $n"
  319. done
  320. for n in "test-int-$id" "test-ext-$id"; do
  321. docker network inspect "$n" > /dev/null 2>&1 && echo "network $n"
  322. done
  323. docker volume inspect "test-work-$id" > /dev/null 2>&1 && echo "volume test-work-$id"
  324. done
  325. return 0
  326. }
  327. ids="$(cat "$r/run-id.txt" "$a/run-id.txt" "$rec/static/run-id.txt" "$p/run-id.txt" \
  328. "$rec/net-ipv6/run-id.txt" "$rec/net-open-path/run-id.txt" "$rec/net-plan-control/run-id.txt" \
  329. "$rec/tp/run-id.txt" "$rec/tf/run-id.txt" 2>/dev/null)"
  330. check "every run recorded its id" test "$(printf '%s\n' "$ids" | grep -c .)" -eq 9
  331. # shellcheck disable=SC2086 # one id per word
  332. check "no container, network or volume of these runs left behind" test -z "$(leftover $ids)"
  333. echo "record: $r"
  334. [ "$fails" -eq 0 ] && echo "smoke passed" || echo "smoke failed: $fails"
  335. exit "$fails"

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.