Skip to content
Back to search

career-ops

MarketplaceWorks with Claude CodeGitHubIncomplete

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026. Its scripts were only partly scanned.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/career-ops-hq/career-ops with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/career-ops-hq/career-ops'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/career-ops-hq-career-ops/badge.svg)](/listings/career-ops-hq-career-ops)

HTML

<a href="/listings/career-ops-hq-career-ops"><img src="/api/listings/career-ops-hq-career-ops/badge.svg" alt="findsafeskills safety rating" /></a>

Install

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

**Fastest way, one command:**

npx @santifer/career-ops init

> ๐Ÿ’ก `npx` ships with [Node.js](https://nodejs.org): it runs the installer once, > without installing anything globally. No Node yet? Install it first. > (Already using a Claude Code / Codex CLI? Then you already have it.)

This clones the latest release into `./career-ops` and installs dependencies. Then:

cd career-ops
claude   # or codex / qwen / opencode / agy / grok โ€” open your AI CLI here

**On first launch, career-ops walks you through setup (your CV, profile and target roles) just by chatting. Nothing to edit by hand.**

<details> <summary><b>Prefer to set it up manually? (git clone)</b></summary>

git clone https://github.com/career-ops-hq/career-ops.git
cd career-ops && npm install
npx playwright install chromium   # only needed for PDF generation
# On a non-Debian/Ubuntu Linux distro (Fedora, Arch, ...), Chromium's system
# libraries aren't installed by the line above โ€” install them yourself with
# your distro's package manager if PDF generation fails to launch the browser
# (Playwright's own docs list the required libraries per platform).

# 2. Open your AI CLI in this directory. On first launch it walks you
#    through setup (your CV, profile and target roles) by chatting
claude   # or codex / opencode / qwen / agy / grok

# Then ask your CLI to adapt the system to you:
# "Change the archetypes to backend engineering roles"
# "Translate the modes to English"
# "Add these 5 companies to portals.yml"
# "Update my profile with this CV I'm pasting"

# 3. Start using
# Paste a job URL or JD text to trigger auto-pipeline.
# Every other mode, and how to ask for it in your CLI: see Usage below.

LinkedIn job alerts: paste the email. No Google account. See [docs/LINKEDIN_JOBS.md](docs/LINKEDIN_JOBS.md).

</details>

Global install

npm i -g @santifer/career-ops

This installs the `career-ops` binary globally, so you can run `career-ops init [folder]` without `npx`. `init` is its only command: it sets up a workspace, and everything after that happens inside your AI coding CLI (see Usage below), not through this binary.

**Which one should you use?** - `npx @santifer/career-ops init`: best for first use; creates a dedicated project folder. - `npm i -g @santifer/career-ops`: if you'd rather type `career-ops init` than the `npx` form. It adds no other commands.

Other ways to run it

- **From any folder, as a Claude Code plugin:** [docs/CLAUDE_PLUGIN.md](docs/CLAUDE_PLUGIN.md). - **Without an AI CLI:** standalone Gemini, Ollama and OpenAI-compatible scripts evaluate a JD on their own. See [docs/SCRIPTS.md](docs/SCRIPTS.md#geminieval--ollamaeval--openaieval).

> **The system is designed to be customized by your AI coding CLI itself.** Modes, archetypes, scoring weights, negotiation scripts -- just ask it to change them. It reads the same files it uses, so it knows exactly what to edit.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/marketplace.json ยท Manifest ยท 20 lines
  1. {
  2. "name": "career-ops",
  3. "metadata": {
  4. "description": "Open-source AI job search that runs locally in your AI coding CLI: scan job portals, evaluate listings into a structured A-H report with a 1-5 score, tailor your CV, track applications",
  5. "version": "1.40.0"
  6. },
  7. "owner": {
  8. "name": "santifer",
  9. "url": "https://santifer.io"
  10. },
  11. "plugins": [
  12. {
  13. "name": "career-ops",
  14. "source": "./",
  15. "description": "Open-source AI job search that runs locally in your AI coding CLI: scan job portals, evaluate listings into a structured A-H report with a 1-5 score, tailor your CV, track applications",
  16. "category": "productivity",
  17. "version": "1.40.0"
  18. }
  19. ]
  20. }
README.md ยท README ยท 634 lines
  1. <p align="center"><picture><source media="(prefers-color-scheme: dark)" srcset="docs/wordmark-dark.svg"><img src="docs/wordmark-light.svg" alt="career-ops" width="250" height="56"></picture></p>
  2. <p align="center">The open-source AI job search agent.</p>
  3. <p align="center">
  4. <a href="README.md">English</a> |
  5. <a href="README.es.md">Espaรฑol</a> |
  6. <a href="README.de.md">Deutsch</a> |
  7. <a href="README.fr.md">Franรงais</a> |
  8. <a href="README.pt-BR.md">Portuguรชs (Brasil)</a> |
  9. <a href="README.ko-KR.md">ํ•œ๊ตญ์–ด</a> |
  10. <a href="README.ja.md">ๆ—ฅๆœฌ่ชž</a> |
  11. <a href="README.cn.md">็ฎ€ไฝ“ไธญๆ–‡</a> |
  12. <a href="README.zh-TW.md">็น้ซ”ไธญๆ–‡</a> |
  13. <a href="README.ua.md">ะฃะบั€ะฐั—ะฝััŒะบะฐ</a> |
  14. <a href="README.ru.md">ะ ัƒััะบะธะน</a> |
  15. <a href="README.pl.md">Polski</a> |
  16. <a href="README.da.md">Dansk</a> |
  17. <a href="README.ta.md">เฎคเฎฎเฎฟเฎดเฏ</a> |
  18. <a href="README.ar.md">ุงู„ุนุฑุจูŠุฉ</a> |
  19. <a href="README.hi.md">เคนเคฟเคจเฅเคฆเฅ€</a> |
  20. <a href="README.tr.md">Tรผrkรงe</a> |
  21. <a href="README.vi.md">Tiแบฟng Viแป‡t</a> |
  22. <a href="README.id.md">Bahasa Indonesia</a>
  23. </p>
  24. <!-- The non-breaking spaces (&nbsp;) and word joiners (&#8288;) below decide where each line wraps on a phone. Keep them when editing. -->
  25. <p align="center"><a href="https://github.com/santifer"><img src="docs/avatar-santifer.png" width="48" height="48" align="middle" alt="Santiago Fernรกndez de Valderrama Aparicio"></a>&nbsp;&nbsp;Created by <a href="https://github.com/santifer"><strong>@santifer</strong></a>, with&nbsp;&nbsp;<a href="https://github.com/career-ops-hq/career-ops#builders"><picture><source media="(prefers-color-scheme: dark)" srcset="docs/builders-stack-dark.svg"><img src="docs/builders-stack-light.svg" alt="" height="32" align="middle"></picture></a>&nbsp;&nbsp;<a href="https://github.com/career-ops-hq/career-ops#builders">700+ contributors</a></p>
  26. <p align="center">
  27. <strong>Months of sending CVs into&nbsp;silence.</strong><br>
  28. So I built the filter I&nbsp;needed.
  29. </p>
  30. <p align="center">
  31. <strong>740&nbsp;listings&nbsp;evaluated. 68&nbsp;applied. 12&nbsp;interviews.&nbsp;1&nbsp;offer.</strong><br>
  32. I was its first user. I got the&nbsp;job. Then&nbsp;I&nbsp;open-&#8288;sourced&nbsp;it.
  33. </p>
  34. <p align="center">
  35. Paste a job. On your machine, it tells you if it's still&nbsp;open and if it&nbsp;fits.<br>
  36. It tailors your CV and drafts your&nbsp;answers. <strong>You&nbsp;press&nbsp;Submit.</strong>
  37. </p>
  38. <p align="center">
  39. <img src="docs/demo.gif" alt="The author's own search in the career-ops dashboard: scored listings, a red count of the ones marked do not apply, then a full evaluation report" width="800">
  40. </p>
  41. <p align="center"><sub>My own search, partway through, with&nbsp;the&nbsp;UI&nbsp;in&nbsp;Spanish.</sub></p>
  42. <p align="center">
  43. Companies use AI to filter candidates.<br>
  44. <strong>I just gave candidates AI&nbsp;to&nbsp;<em>choose</em>&nbsp;companies.</strong>
  45. </p>
  46. <p align="center">
  47. Six months later, <strong>I left that job.</strong><br>
  48. Now we build career-ops, so&nbsp;you&nbsp;can&nbsp;land&nbsp;yours.
  49. </p>
  50. <p align="center"><a href="#your-turn">Try it on one job&nbsp;โ†“</a> ยท <a href="https://santifer.io/career-ops-system">Read the full&nbsp;story&nbsp;โ†’</a></p>
  51. <p align="center"><sub>Prefer not to use a terminal? <a href="https://career-ops.org/waitlist?ref=readme-en">Join the waitlist for a web version&nbsp;โ†’</a></sub></p>
  52. <br>
  53. <p align="center"><a href="HIRED.md"><img src="docs/hired-badge.svg" alt="Hired with career-ops: public stories on the record"></a></p>
  54. <p align="center">The people who came after me wrote down how they got hired.</p>
  55. <p align="center">
  56. <a href="HIRED.md"><img src="docs/hired-wall.svg" alt="Hired stories you can open and read" width="800"></a>
  57. </p>
  58. <p align="center"><sub>Every card is a public issue you can open. Landed yours? <a href="https://github.com/career-ops-hq/career-ops/issues/new?template=i-got-hired.yml">Leave your card โ†’</a></sub></p>
  59. <p align="center">โญ <strong>If career-ops helped you, a star helps the next person find&nbsp;it.</strong></p>
  60. <br>
  61. <p align="center">
  62. <a href="https://wired.com.gr/article/to-ai-ergaleio-pou-fernei-epanastasi-ston-tropo-pou-psachnoume-douleia/" rel="noopener noreferrer nofollow"><picture><source media="(prefers-color-scheme: dark)" srcset="docs/press/wired-dark.svg"><img src="docs/press/wired.svg" alt="WIRED" height="32"></picture></a>
  63. &nbsp;&nbsp;&nbsp;&nbsp;
  64. <a href="https://www.businessinsider.com/how-i-built-tool-filter-job-listings-landed-head-ai-2026-4" rel="noopener noreferrer nofollow"><picture><source media="(prefers-color-scheme: dark)" srcset="docs/press/business-insider-dark.svg"><img src="docs/press/business-insider.svg" alt="Business Insider" height="32"></picture></a>
  65. </p>
  66. <p align="center">
  67. <a href="https://vercel.com/open-source-program"><img alt="Vercel OSS Program" src="https://vercel.com/oss/program-badge-2026.svg" height="32"></a>
  68. </p>
  69. <p align="center">
  70. <a href="https://trendshift.io/repositories/25195" rel="noopener noreferrer"><img src="https://trendshift.io/api/badge/repositories/25195" alt="GitHub Trending: #1 repository of the day (Trendshift)" width="250" height="55"></a>
  71. &nbsp;&nbsp;&nbsp;&nbsp;
  72. <a href="https://www.producthunt.com/products/santifer-io?utm_source=badge-featured&utm_medium=badge" rel="noopener noreferrer"><img src="docs/press/producthunt.svg" alt="Featured on Product Hunt" width="206" height="54"></a>
  73. </p>
  74. ## Your turn
  75. Paste one job you were about to apply to tonight. If it comes back <em>do not apply</em>, you just got the night back. If it comes back with a plan, you know what to do next.
  76. ```bash
  77. npx @santifer/career-ops init
  78. ```
  79. <p align="center"><sub>Open source. Local. In the AI CLI you already use. <a href="docs/RUNNING_ON_A_BUDGET.md">Free and local models included.</a></sub></p>
  80. ## You don't have to search alone
  81. The silence after you hit send is not about you. Enough people saw the same thing to write the practice down, in six lines:
  82. > Apply better to fewer. Signal over volume. Evidence over keywords. A human decides. Local-first. Dignity on both sides of the table.
  83. <p align="center"><a href="https://career-ops.org/manifesto?utm_source=readme"><img src="https://img.shields.io/badge/dynamic/regex?url=https%3A%2F%2Fraw.githubusercontent.com%2Fcareer-ops-hq%2Fcareer-ops%2Fmain%2FSIGNATURES.md&search=n%3A(%5Cd%2B)%5Cs*%24&replace=%241%20signatories&label=THE%20CAREEROPS%20MANIFESTO&color=DD7627&style=for-the-badge&labelColor=2b3137" alt="The CareerOps manifesto: live count of signatories"></a></p>
  84. <p align="center">A few of them, in their own words.</p>
  85. <p align="center">
  86. <a href="https://career-ops.org/manifesto?utm_source=readme"><img src="docs/manifesto-wall.svg" alt="Three signatures of the CareerOps manifesto, in their own words" width="800"></a>
  87. </p>
  88. <p align="center"><sub>Every signature is a commit you can audit. <a href="https://career-ops.org/manifesto?utm_source=readme">Read them all, or add yours โ†’</a> ยท <a href="https://career-ops.org/manifesto?utm_source=readme#what-is-ai-job-search">What is AI job search?</a></sub></p>
  89. <br>
  90. Hiring will not fix itself. The people going through it can, and they are already in the room, comparing notes and fixing each other's setups.
  91. <a id="builders"></a>
  92. <p align="center"><a href="CONTRIBUTING.md"><img src="https://warpchart.dev/api/builders?lang=en" alt="Animated wall of contributors' GitHub avatars, filling in order of arrival since April 2026, with the next square left open." width="100%"></a></p>
  93. [This is where the fix gets written](CONTRIBUTING.md). A room, not a club. **Let's build together.**
  94. <p align="center"><a href="https://discord.gg/8pRpHETxa4"><img src="docs/discord-badge.svg" alt="Discord: live member count of the career-ops server"></a></p>
  95. ## Sponsors
  96. career-ops is free for candidates, forever. These companies sponsor the project:
  97. <p align="center">
  98. <a href="https://serpapi.com/career-ops-org" title="SerpApi"><img src="docs/sponsors/serpapi.svg" alt="SerpApi" height="48"></a>
  99. </p>
  100. <p align="center"><strong>SerpApi</strong> ยท Build a portfolio project with live search data. SerpApi gives developers structured JSON/Markdown from Google Search, Maps, Shopping, and other engines through a simple API call.</p>
  101. > Sponsorship buys clearly labeled visibility, never influence: no amount of money changes the roadmap or places anything in the product. Sponsors never appear in evaluations, rankings or recommendations.
  102. ## What career-ops does for you
  103. Paste a job. It tells you whether that night is worth it.
  104. - **Still open?** It checks that the posting is still live before you write a word.
  105. - **Not you?** It scores the role against your real CV and tells you to skip a weak fit. You can override it.
  106. - **Worth it?** It drafts the CV, the cover letter and the answers. You read them. You send them.
  107. - **Who do I talk to?** It finds the person and drafts the note. It never sends it.
  108. - **Where does it all go?** Every application stays on your machine. Nothing is uploaded to us.
  109. - **What should I learn?** After a run of noes, it names the gap.
  110. The first runs are rough. It does not know you yet. Talk to it: your CV, what you want, what you refuse. Think of it as a recruiter's first week.
  111. On first launch it asks for all that in chat. Nothing to configure by hand.
  112. ## What career-ops will not do
  113. - **Auto-submit an application.** It drafts the answer to every field; you review and click Submit. The script never POSTs (`prepare-application.mjs`).
  114. - **Send an email.** Drafts only. There is no mail transport anywhere in this codebase.
  115. - **Phone home.** No telemetry, no backend of ours. Your CV goes from your machine to the AI provider you chose, and nowhere else. The only public ledger is this repo: `HIRED.md` and its issues.
  116. - **Push you to apply below 4.0/5.** It will tell you not to. You can override it, and it will say so.
  117. It reformulates your CV; it must never fabricate it. A check in code stops a PDF whose numbers or facts appear in neither your CV nor your article digest, but it cannot judge every rewording yet. Read every CV before you send it. Details in the [FAQ](#faq).
  118. ## Features
  119. | Feature | Description |
  120. | ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------- |
  121. | **A-H Evaluation** | Role summary, CV match (with how much each requirement matters for this posting, and whether that weight came from the JD's own wording, its structure, or an estimate, labelled per requirement; an estimate can never be top-band), level strategy, comp research, personalization, interview prep (STAR+R) -- plus a Block G posting-legitimacy check (is it still open, is it a repost: observations, not verdicts), and a Work-Auth signal that flags an explicit no-sponsorship JD as a hard blocker |
  122. | **Human-in-the-Loop** | AI evaluates and recommends, you decide and act. The system never submits an application -- you always have the final call <!-- hitl: absolute guarantee. Do not add "automatically", "by itself", "without your permission" or any other hedge when translating this row. --> |
  123. | **ATS PDF Generation** | ATS-readable CVs tailored to each JD from your own experience, in Space Grotesk + DM Sans design |
  124. | **Cover Letter Generator** | Research-backed cover letters with keyword mirroring, four interactive angle prompts (why/problems/approach/tone), draft-in-chat approval gate, and A4 PDF via the same HTML + Playwright pipeline as CVs. Auto-drafts on every evaluation; complete and generate on demand via `/career-ops cover` |
  125. | **Beyond the CV** | Company research ([`deep`](modes/deep.md)) surfaces AI strategy, recent moves, engineering culture, and the angle your profile should take. Contact discovery ([`contacto`](modes/contacto.md)) identifies the hiring manager, recruiter, or team peer worth reaching out to and drafts a โ‰ค300-character LinkedIn message tuned to each contact type. Formal application email drafts ([`email`](modes/email.md)) turn an evaluated report or pasted JD into a subject line, body, and attachment checklist without sending, submitting, or clicking anything. Applications get you in the queue; research gets you a conversation. |
  126. | **Pattern Analysis** | Rejection patterns and per-ATS-channel advance rates (`analyze-patterns.mjs`), lifetime funnel stats (`stats.mjs`), repost detection, a possible sign of a ghost job (`detect-reposts.mjs`) |
  127. <details>
  128. <summary><b>Everything else it does</b></summary>
  129. | Feature | Description |
  130. | ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------- |
  131. | **Auto-Pipeline** | Paste a URL, get a full evaluation + PDF + tracker entry |
  132. | **Interview Story Bank** | Accumulates STAR+Reflection stories across evaluations -- 5-10 master stories you can adapt to the behavioral questions you get |
  133. | **Negotiation Scripts** | Salary negotiation frameworks, geographic discount pushback, competing offer leverage |
  134. | **Application Email Drafts** | Formal recruiter/referral/cold application emails from a report or pasted JD, with subject line, attachment checklist, source-backed fit points, and a profile-driven contact block. Draft-only -- career-ops never sends, submits, or clicks anything. |
  135. | **Portal Scanner** | 100+ companies pre-configured (Anthropic, OpenAI, ElevenLabs, Retool, n8n...) + custom web-search queries across Ashby, Greenhouse, Lever, Wellfound |
  136. | **Funded Company Discovery** | Review-first `npm run company:funded` script surfaces recently funded companies and source diagnostics from structured public feeds without editing your data |
  137. | **Batch Processing** | Parallel evaluation with headless CLI workers (`claude -p` / `opencode run`) |
  138. | **Dashboard TUI** | Terminal UI to browse, filter, and sort your pipeline |
  139. | **Pipeline Integrity** | Automated merge, dedup, status normalization, health checks |
  140. | **Interview Suite** | Time-blocked prep plans, practice sessions with feedback, post-interview debriefs ([`interview/`](modes/interview/README.md)), and a company red-flag detector ([`interview-redflag`](modes/interview-redflag.md)) |
  141. | **Offer Stage** | Contract reading companion -- clause walk plus a lawyer question list ([`offer-prep`](modes/offer-prep.md)) -- and a desired/advertised/actual salary-gap analyzer (`salary-gap.mjs`) |
  142. | **Follow-ups & Replies** | Follow-up cadence calculator and seeded reminders (`followup-cadence.mjs`, `followup-seed.mjs`); turn them off with `follow_ups.enabled: false` in `config/profile.yml`; employer reply classification into tracker updates ([`reply-watch`](modes/reply-watch.md)) |
  143. | **Plugin System** | Opt-in integrations (Gmail, Notion, Apify + a community registry), disabled by default -- see [docs/PLUGINS.md](docs/PLUGINS.md) |
  144. | **LinkedIn job alerts** | Paste a LinkedIn alert email on the dashboard or with `node linkedin-import.mjs`. No Google account. See [docs/LINKEDIN_JOBS.md](docs/LINKEDIN_JOBS.md) |
  145. </details>
  146. ## How It Works
  147. <p align="center"><picture><source media="(prefers-color-scheme: dark)" srcset="docs/how-it-works-dark.svg"><img src="docs/how-it-works-light.svg" alt="How it works: you paste a job URL or description. Archetype Detection classifies it as LLMOps, Agentic, PM, SA, FDE or Transformation. The A-H Evaluation reads cv.md and covers match, gaps, comp research, STAR stories and legitimacy. It produces a report (reports/*.md), the tailored CV in the format set by cv.output_format โ€” a PDF by default โ€” and a tracker entry (data/applications.md)." width="760"></picture></p>
  148. ## Quick Start
  149. **Fastest way, one command:**
  150. ```bash
  151. npx @santifer/career-ops init
  152. ```
  153. > ๐Ÿ’ก `npx` ships with [Node.js](https://nodejs.org): it runs the installer once,
  154. > without installing anything globally. No Node yet? Install it first.
  155. > (Already using a Claude Code / Codex CLI? Then you already have it.)
  156. This clones the latest release into `./career-ops` and installs dependencies. Then:
  157. ```bash
  158. cd career-ops
  159. claude # or codex / qwen / opencode / agy / grok โ€” open your AI CLI here
  160. ```
  161. **On first launch, career-ops walks you through setup (your CV, profile and target roles) just by chatting. Nothing to edit by hand.**
  162. <details>
  163. <summary><b>Prefer to set it up manually? (git clone)</b></summary>
  164. ```bash
  165. git clone https://github.com/career-ops-hq/career-ops.git
  166. cd career-ops && npm install
  167. npx playwright install chromium # only needed for PDF generation
  168. # On a non-Debian/Ubuntu Linux distro (Fedora, Arch, ...), Chromium's system
  169. # libraries aren't installed by the line above โ€” install them yourself with
  170. # your distro's package manager if PDF generation fails to launch the browser
  171. # (Playwright's own docs list the required libraries per platform).
  172. # 2. Open your AI CLI in this directory. On first launch it walks you
  173. # through setup (your CV, profile and target roles) by chatting
  174. claude # or codex / opencode / qwen / agy / grok
  175. # Then ask your CLI to adapt the system to you:
  176. # "Change the archetypes to backend engineering roles"
  177. # "Translate the modes to English"
  178. # "Add these 5 companies to portals.yml"
  179. # "Update my profile with this CV I'm pasting"
  180. # 3. Start using
  181. # Paste a job URL or JD text to trigger auto-pipeline.
  182. # Every other mode, and how to ask for it in your CLI: see Usage below.
  183. ```
  184. LinkedIn job alerts: paste the email. No Google account. See [docs/LINKEDIN_JOBS.md](docs/LINKEDIN_JOBS.md).
  185. </details>
  186. ### Global install
  187. ```bash
  188. npm i -g @santifer/career-ops
  189. ```
  190. This installs the `career-ops` binary globally, so you can run `career-ops init [folder]` without `npx`. `init` is its only command: it sets up a workspace, and everything after that happens inside your AI coding CLI (see Usage below), not through this binary.
  191. **Which one should you use?**
  192. - `npx @santifer/career-ops init`: best for first use; creates a dedicated project folder.
  193. - `npm i -g @santifer/career-ops`: if you'd rather type `career-ops init` than the `npx` form. It adds no other commands.
  194. ### Other ways to run it
  195. - **From any folder, as a Claude Code plugin:** [docs/CLAUDE_PLUGIN.md](docs/CLAUDE_PLUGIN.md).
  196. - **Without an AI CLI:** standalone Gemini, Ollama and OpenAI-compatible scripts evaluate a JD on their own. See [docs/SCRIPTS.md](docs/SCRIPTS.md#geminieval--ollamaeval--openaieval).
  197. > **The system is designed to be customized by your AI coding CLI itself.** Modes, archetypes, scoring weights, negotiation scripts -- just ask it to change them. It reads the same files it uses, so it knows exactly what to edit.
  198. See [docs/SETUP.md](docs/SETUP.md) for the full setup guide (including [scanning through an outbound proxy](docs/SETUP.md#scanning-through-an-outbound-proxy)), [docs/SUPPORTED_CLIS.md](docs/SUPPORTED_CLIS.md) for how to run career-ops in each supported CLI, and its headless command where it has one, [docs/RUNNING_ON_A_BUDGET.md](docs/RUNNING_ON_A_BUDGET.md) for instructions on running career-ops cheaply using custom or local models (and [docs/FREE_TIER.md](docs/FREE_TIER.md) for running it at zero cost on Antigravity CLI's free tier), [docs/AUTOMATION.md](docs/AUTOMATION.md) for scheduling recurring scans and a zero-token triage-to-shortlist recipe, [docs/KEEPING_PORTALS_LIVE.md](docs/KEEPING_PORTALS_LIVE.md) for keeping your company list current, [docs/BACKUP_STRATEGIES.md](docs/BACKUP_STRATEGIES.md) for backing up your CV, tracker and reports, [docs/APPLY_AUTOFILL.md](docs/APPLY_AUTOFILL.md) for details on the ATS auto-fill flow, [docs/LINKEDIN_JOIN.md](docs/LINKEDIN_JOIN.md) for cross-referencing a LinkedIn connections export against the companies in your funnel, [docs/LINKEDIN_ALERTS.md](docs/LINKEDIN_ALERTS.md) for adding LinkedIn job-alert emails on the scan schedule, and [docs/FAQ.md](docs/FAQ.md) for answers to common setup questions, including [how story provenance prevents invented numbers](docs/FAQ.md#why-does-career-ops-refuse-to-use-a-number-from-my-story-bank). Design principles live in [ARCHITECTURE.md](ARCHITECTURE.md); runtime flows in [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md).
  199. ## Usage
  200. Paste a job URL or description into your AI CLI. career-ops checks that a linked posting is still live (pasted text has no link to check), scores it against your CV, writes a report, tailors your CV (a PDF by default) and adds a row to your tracker. That covers most days. Everything else is a mode you ask for by name. If a posting can't be read, the pipeline stops and asks you to paste the text or share a screenshot.
  201. ### Asking for a mode
  202. | CLI | How you ask for `scan` |
  203. | --- | --- |
  204. | Claude Code, OpenCode, Antigravity CLI (`agy`), Grok Build CLI | `/career-ops scan` |
  205. | Pi | `/skill:career-ops scan` |
  206. | Codex, Qwen, Cursor, or any other supported CLI | Plain language: "Run the career-ops scan mode" |
  207. `/career-ops` with nothing after it shows the command menu in the CLIs that use it. Plain language works in every supported CLI. As a Claude Code plugin, the command is `/career-ops:career-ops`.
  208. All of them share one router, `.agents/skills/career-ops/SKILL.md`, and the same `modes/*.md` files, so every CLI routes a request the same way (browser-heavy modes also need Playwright tools). Slash commands are not guaranteed in Codex: ask in plain language, or run one prompt headless with `codex exec "Evaluate this JD with career-ops auto-pipeline: https://company.com/jobs/123"` ([docs/CODEX.md](docs/CODEX.md)). How to run it in each supported CLI, headless commands where a CLI has one, and per-CLI setup notes are in [docs/SUPPORTED_CLIS.md](docs/SUPPORTED_CLIS.md).
  209. ### The everyday commands
  210. | Command | What it does |
  211. | --- | --- |
  212. | *paste a URL or JD* | Full pipeline: liveness check (URLs only), A-G evaluation plus H at high scores, report, tailored CV (PDF by default), tracker row |
  213. | `scan` | Scan your portals for new postings and add them to the inbox (`data/pipeline.md`) |
  214. | `pipeline` | Work through the URLs waiting in the inbox (the standard and premium tiers may skip obvious mismatches) |
  215. | `apply` | Fill an application form with you, field by field. It stops before Submit |
  216. | `tracker` | Where every application stands |
  217. | `followup` | Which applications are due a follow-up, with drafts |
  218. <details>
  219. <summary><b>All commands, by stage of the search</b></summary>
  220. `oferta`, `ofertas` and `contacto` keep their original Spanish names: offer, offers, contact.
  221. **Find roles**
  222. | Command | What it does |
  223. | --- | --- |
  224. | [`scan`](modes/scan.md) | Scan portals and discover new offers |
  225. | [`discover`](modes/discover.md) | Resolve a list of company names to scannable ATS boards. Previews by default; adds them to `portals.yml` only with `--write` (zero-token) |
  226. | [`pipeline`](modes/pipeline.md) | Process pending URLs from the inbox (`data/pipeline.md`) |
  227. | [`triage`](modes/triage.md) | Fast first-pass go/no-go score from `modes/_brief.md`, before a full evaluation (writes no files) |
  228. | [`batch`](modes/batch.md) | Evaluate many postings with headless workers (parallel with Claude, one at a time with other CLIs) |
  229. | [`titles`](modes/titles.md) | Suggest adjacent job titles from your CV to broaden the search |
  230. **Evaluate a role**
  231. | Command | What it does |
  232. | --- | --- |
  233. | *paste a URL or JD* | [Auto-pipeline](modes/auto-pipeline.md): evaluation + report + tailored CV (PDF by default) + tracker entry |
  234. | [`oferta`](modes/oferta.md) | Evaluation only, Blocks A to G plus H at high scores (no PDF) |
  235. | [`ofertas`](modes/ofertas.md) | Compare and rank several offers |
  236. | [`deep`](modes/deep.md) | Deep research prompt about the company |
  237. | [`eu-swe`](modes/regional/eu-swe.md) | Calibrate a European software-engineering application before CV, apply or interview |
  238. **Apply**
  239. | Command | What it does |
  240. | --- | --- |
  241. | [`pdf`](modes/pdf.md) | ATS-optimized CV as a PDF |
  242. | [`ats`](modes/ats.md) | ATS-friendliness check of a generated CV: score and fixable issues |
  243. | [`text`](modes/text.md) | Tailored CV as markdown (mirrors `cv.md`, no PDF) |
  244. | [`latex`](modes/latex.md) | Export the CV as LaTeX/Overleaf `.tex` |
  245. | [`latex-tex`](modes/latex-tex.md) | Tailor your own `resume.tex` in place (opt-in; `cv.md` stays the default) |
  246. | [`cover`](modes/cover.md) | Cover letter, from a pasted JD or an evaluated report (`cover {slug}`) |
  247. | [`email`](modes/email.md) | Formal application email draft. Draft-only: never sends, submits or clicks |
  248. | [`contacto`](modes/contacto.md) | Find the hiring manager, recruiter or a peer, and draft a LinkedIn message |
  249. | [`apply`](modes/apply.md) | Live application assistant (Playwright from this checkout). Does not submit |
  250. **Interview**
  251. | Command | What it does |
  252. | --- | --- |
  253. | [`interview-prep`](modes/interview-prep.md) | Company-specific interview prep document |
  254. | [`interview/plan`](modes/interview/plan.md) | Time-blocked prep plan for an upcoming interview |
  255. | [`interview/practice`](modes/interview/practice.md) | Practice interview, one question at a time, with feedback |
  256. | [`interview/debrief`](modes/interview/debrief.md) | Post-interview debrief: close gaps, predict the next round |
  257. | [`interview-redflag`](modes/interview-redflag.md) | Analyze an employer's warning signs before you join |
  258. **After you apply**
  259. | Command | What it does |
  260. | --- | --- |
  261. | [`tracker`](modes/tracker.md) | Application status overview |
  262. | [`followup`](modes/followup.md) | Follow-up cadence: flag overdue applications, generate drafts |
  263. | [`reply-watch`](modes/reply-watch.md) | Classify employer replies and suggest tracker updates |
  264. | [`outcome`](modes/outcome.md) | Record an application's outcome and archive its artifacts |
  265. | [`offer-prep`](modes/offer-prep.md) | Read a received offer or contract with you: clause walk and questions for a lawyer (not legal advice) |
  266. **Your profile and CV**
  267. | Command | What it does |
  268. | --- | --- |
  269. | [`interview`](modes/interview.md) | Profile setup: an interactive interview about your experience that builds your CV and profile (not interview practice) |
  270. | [`intake`](modes/intake.md) | Build or enrich your profile from files in `documents/` (master CV, LinkedIn export). Nothing is written without your confirmation |
  271. | [`master-profile`](modes/master-profile.md) | Import, review and validate your Master Career Profile, and prepare selected facts as CV bullets |
  272. | [`add`](modes/add.md) | Add a project, paper or role to your CV (fetch, preview, confirm) |
  273. | [`expand`](modes/expand.md) | Discover competencies missing from your CV, from your profile links |
  274. | [`retro`](modes/retro.md) | Review this session and keep what you taught it as rules (writes only the ones you accept) |
  275. **Improve your targeting**
  276. | Command | What it does |
  277. | --- | --- |
  278. | [`patterns`](modes/patterns.md) | Analyze rejection patterns and improve targeting |
  279. | [`calibrate`](modes/calibrate.md) | Check whether your evaluation scores predict your real outcomes. Advisory: never changes scoring |
  280. | [`upskill`](modes/upskill.md) | Skill-gap analysis across your evaluated reports |
  281. | [`training`](modes/training.md) | Evaluate a course or certification against your North Star |
  282. | [`project`](modes/project.md) | Evaluate a portfolio project idea |
  283. **System**
  284. | Command | What it does |
  285. | --- | --- |
  286. | [`agent-inbox`](modes/agent-inbox.md) | Queue requests for the next session, and work through them then (`data/agent-inbox.md`). `inbox` works too |
  287. | [`update`](modes/update.md) | Update career-ops system files, with a diff preview and compatibility check |
  288. </details>
  289. ## Pre-configured Portals
  290. The scanner comes with **100+ companies** ready to scan and **35+ search queries** across major job boards. Copy `templates/portals.example.yml` to `portals.yml` and add your own:
  291. **AI Labs:** Anthropic, OpenAI, Mistral, Cohere, LangChain, Pinecone
  292. **Voice AI:** ElevenLabs, PolyAI, Parloa, Hume AI, Deepgram, Vapi, Bland AI
  293. **AI Platforms:** Retool, Airtable, Vercel, Temporal, Glean, Arize AI
  294. **Contact Center:** Ada, LivePerson, Sierra, Decagon, Talkdesk, Genesys
  295. **Enterprise:** Salesforce, Twilio, Gong, Dialpad
  296. **LLMOps:** Langfuse, Weights & Biases, Lindy, Cognigy, Speechmatics
  297. **Automation:** n8n, Zapier, Make.com
  298. **European:** Factorial, Attio, Tinybird, Clarity AI, Travelperk
  299. **Job boards searched:** 55+ provider modules cover ATS APIs, board-wide feeds, XML/RSS feeds, markdown feeds, and local parsers. See [Supported job boards](docs/SUPPORTED_JOB_BOARDS.md) for the full table.
  300. Keep the list current: `portals.yml` is designed to stay live, not be a one-time setup file. See [Keeping portals.yml live](docs/KEEPING_PORTALS_LIVE.md) for the house rule that adds new companies automatically and the commands to validate the file after every change.
  301. By default `node scan.mjs` (a.k.a. `npm run scan`) trusts what each ATS feed returns. Some companies leave stale postings in their public API even after the role is closed, so those expired entries can leak into `pipeline.md`. Pass `--verify` to launch Playwright after the API pass and drop expired postings before they hit the pipeline:
  302. ```bash
  303. node scan.mjs --verify # zero-token discovery + Playwright liveness check
  304. ```
  305. The verification is sequential and only runs against new offers (after dedup), so the cost stays bounded.
  306. ## Dashboard TUI
  307. The built-in terminal dashboard lets you browse your pipeline visually:
  308. ```bash
  309. npm run serve:dashboard # launch the TUI
  310. npm run build:dashboard # optional: build the standalone binary
  311. ```
  312. Features: 10 filter tabs, 8 sort modes, grouped/flat view, lazy-loaded previews, inline status changes.
  313. There is also an **experimental web UI** (alpha and opt-in: nothing runs unless you start it): see [`web/README.md`](web/README.md).
  314. ## Project Structure
  315. ```
  316. career-ops/
  317. โ”œโ”€โ”€ AGENTS.md # Canonical agent instructions (all CLIs)
  318. โ”œโ”€โ”€ CLAUDE.md # Claude Code wrapper (imports AGENTS.md)
  319. โ”œโ”€โ”€ CODEX.md # Codex wrapper (imports AGENTS.md)
  320. โ”œโ”€โ”€ OPENCODE.md # OpenCode wrapper (imports AGENTS.md)
  321. โ”œโ”€โ”€ GEMINI.md # Legacy no-op guard to avoid Antigravity duplicate context
  322. โ”œโ”€โ”€ cv.md # Your CV (create this)
  323. โ”œโ”€โ”€ article-digest.md # Your proof points (optional)
  324. โ”œโ”€โ”€ config/
  325. โ”‚ โ””โ”€โ”€ profile.example.yml # Template for your profile
  326. โ”œโ”€โ”€ modes/ # Skill modes
  327. โ”‚ โ”œโ”€โ”€ _shared.md # Shared context (system file; local edit kept unless apply --force)
  328. โ”‚ โ”œโ”€โ”€ _profile.md # Your archetypes and framing (customize this)
  329. โ”‚ โ”œโ”€โ”€ oferta.md # Single evaluation
  330. โ”‚ โ”œโ”€โ”€ pdf.md # PDF generation
  331. โ”‚ โ”œโ”€โ”€ cover.md # Cover letter generation
  332. โ”‚ โ”œโ”€โ”€ email.md # Formal application email drafts
  333. โ”‚ โ”œโ”€โ”€ scan.md # Portal scanner
  334. โ”‚ โ”œโ”€โ”€ batch.md # Batch processing
  335. โ”‚ โ””โ”€โ”€ ...
  336. โ”œโ”€โ”€ templates/
  337. โ”‚ โ”œโ”€โ”€ cv-template.html # ATS-optimized CV template
  338. โ”‚ โ”œโ”€โ”€ portals.example.yml # Scanner config template
  339. โ”‚ โ””โ”€โ”€ states.yml # Canonical statuses
  340. โ”œโ”€โ”€ batch/
  341. โ”‚ โ”œโ”€โ”€ batch-prompt.md # Self-contained worker prompt
  342. โ”‚ โ””โ”€โ”€ batch-runner.sh # Orchestrator script
  343. โ”œโ”€โ”€ dashboard/ # Go TUI pipeline viewer
  344. โ”œโ”€โ”€ data/ # Your tracking data (gitignored)
  345. โ”œโ”€โ”€ reports/ # Evaluation reports (gitignored)
  346. โ”œโ”€โ”€ output/ # Generated PDFs (gitignored)
  347. โ”œโ”€โ”€ fonts/ # Space Grotesk + DM Sans
  348. โ”œโ”€โ”€ docs/ # Setup, customization, budget guide, architecture
  349. โ””โ”€โ”€ examples/ # Sample CV, report, proof points
  350. ```
  351. ## External Data Directory (Optional)
  352. By default, user-layer data (such as `cv.md`, `portals.yml`, and `data/` / `reports/` / `output/` folders) lives inside the project root folder.
  353. To separate your personal data from the code (making it easier to switch branches, pull updates, or test multiple profiles), you can configure an external data directory using the following precedence:
  354. 1. **Environment Variables:** Set the `CAREER_OPS_ROOT` or `CAREER_OPS_DATA_DIR` environment variable:
  355. ```bash
  356. export CAREER_OPS_ROOT=~/my-career-data
  357. ```
  358. 2. **Marker File:** Create a `.career-ops-data` file in the repository root containing the path to your data directory.
  359. 3. **Default:** Defaults to the repository root.
  360. Once resolved, all user files are resolved and written relative to that folder, while prompt files and scripts continue to resolve relative to the repository.
  361. - **Tracker Override:** You can also set `CAREER_OPS_TRACKER` to override the applications tracker file path directly.
  362. - **Writes:** All write operations (such as merges) canonically target `{DATA_ROOT}/data/applications.md`.
  363. - **Scanner config:** `portals.yml` is read and validated at `{DATA_ROOT}/portals.yml`, so `node validate-portals.mjs` checks the same file `scan.mjs` reads.
  364. - **Generated documents:** tailored CVs and cover letters are written under `{DATA_ROOT}/output/`, and the PDF manifest that links them to a report lives at `{DATA_ROOT}/data/pdf-index.tsv`. While `CAREER_OPS_TRACKER` is unset, the tracker workspace that bounds those writes is the data root, not the checkout.
  365. - **PDFs under a tracker override:** `generate-pdf.mjs` resolves `CAREER_OPS_TRACKER` before it derives the workspace, so with the override set the workspace is the folder that holds that tracker (or the folder above it, when the tracker sits in a `data/` folder). The CV's HTML and every PDF must then sit inside that workspace, and the manifest moves to its `data/pdf-index.tsv`. Cover letters still target `{DATA_ROOT}/output/`, so they are refused when that folder falls outside the tracker's workspace.
  366. - **Code layer stays put:** `node_modules/`, `providers/`, `modes/` and the scripts themselves always resolve against the repository, never the data root.
  367. The Go dashboard TUI, Node.js scripts, and AI agent modes all automatically respect this resolution hierarchy.
  368. ## Tech Stack
  369. ![Claude Code](https://img.shields.io/badge/Claude_Code-000?style=flat&logo=anthropic&logoColor=white)
  370. ![Node.js](https://img.shields.io/badge/Node.js-339933?style=flat&logo=node.js&logoColor=white)
  371. ![Playwright](https://img.shields.io/badge/Playwright-2EAD33?style=flat&logo=playwright&logoColor=white)
  372. ![Go](https://img.shields.io/badge/Go-00ADD8?style=flat&logo=go&logoColor=white)
  373. ![Bubble Tea](https://img.shields.io/badge/Bubble_Tea-FF75B5?style=flat&logo=go&logoColor=white)
  374. - **Agent**: AI coding CLI with shared skills and modes (`AGENTS.md` + CLI wrapper)
  375. - **PDF**: Playwright + HTML template
  376. - **Cover letters**: HTML template + Playwright (A4 PDF, same pipeline as CVs)
  377. - **Scanner**: Playwright + Greenhouse API + WebSearch
  378. - **Dashboard**: Go + Bubble Tea + Lipgloss (Catppuccin Mocha theme)
  379. - **Data**: Markdown tables + YAML config + TSV batch files
  380. <p align="center">
  381. <a href="https://github.com/career-ops-hq/career-ops/releases/latest"><img src="docs/npm-badge.svg" alt="Latest release"></a>
  382. </p>
  383. <p align="center">
  384. <a href="https://claude.com/claude-code"><img src="https://img.shields.io/badge/Built_with-Claude_Code-000?style=for-the-badge&logo=anthropic&logoColor=white" alt="Built with Claude Code"></a>
  385. </p>
  386. <p align="center">
  387. <sub>Also runs on any agent-skill-standard CLI. See <a href="docs/SUPPORTED_CLIS.md">Supported CLIs</a>.</sub><br>
  388. <img src="https://img.shields.io/badge/Claude_Code-000?style=flat&logo=anthropic&logoColor=white" alt="Claude Code">
  389. <img src="https://img.shields.io/badge/OpenCode-111827?style=flat&logo=terminal&logoColor=white" alt="OpenCode">
  390. <img src="https://img.shields.io/badge/Antigravity_CLI-4285F4?style=flat&logo=google&logoColor=white" alt="Antigravity CLI">
  391. <img src="https://img.shields.io/badge/Codex-412991?style=flat&logo=openai&logoColor=white" alt="Codex">
  392. <img src="https://img.shields.io/badge/Qwen-615CED?style=flat" alt="Qwen">
  393. <img src="https://img.shields.io/badge/Kimi-FF4B4B?style=flat" alt="Kimi">
  394. <img src="https://img.shields.io/badge/GitHub_Copilot-000?style=flat&logo=githubcopilot&logoColor=white" alt="GitHub Copilot">
  395. <img src="https://img.shields.io/badge/Grok_Build_CLI-000?style=flat&logo=x&logoColor=white" alt="Grok Build CLI">
  396. <img src="https://img.shields.io/badge/Pi-4B3F72?style=flat" alt="Pi">
  397. <img src="https://img.shields.io/badge/Oh_My_Pi-4B3F72?style=flat" alt="Oh My Pi">
  398. <br>
  399. <img src="https://img.shields.io/badge/Node.js-339933?style=flat&logo=node.js&logoColor=white" alt="Node.js">
  400. <img src="https://img.shields.io/badge/Go-00ADD8?style=flat&logo=go&logoColor=white" alt="Go">
  401. <img src="https://img.shields.io/badge/Playwright-2EAD33?style=flat&logo=playwright&logoColor=white" alt="Playwright">
  402. <img src="https://img.shields.io/badge/Bubble_Tea-FF75B5?style=flat&logo=go&logoColor=white" alt="Bubble Tea">
  403. <img src="https://img.shields.io/badge/License-MIT-blue.svg" alt="MIT">
  404. <a href="TRADEMARK.md"><img src="https://img.shields.io/badge/Trademark-Policy-blue.svg" alt="Trademark Policy"></a>
  405. </p>
  406. ## Also Open Source
  407. - **[cv-santiago](https://github.com/santifer/cv-santiago)** -- The portfolio website (santifer.io) with AI chatbot, LLMOps dashboard, and case studies. If you need a portfolio to showcase alongside your job search, fork it and make it yours.
  408. ## FAQ
  409. **What is career-ops?**
  410. career-ops ([career-ops.org](https://career-ops.org), also known as **careerops**) is an open-source AI job search that runs locally in your AI coding CLI (Claude Code, Codex, OpenCode and others) and leaves every decision to you. It evaluates job offers against your CV, generates ATS-tailored PDFs, finds the right person to contact, and tracks everything in one place: you always have the final call. It is the first reference implementation of the [CareerOps Manifesto](https://career-ops.org/manifesto).
  411. **Can the tailored CV invent things?**
  412. It must not, and the prompts say so: reformulate, never fabricate. `generate-pdf` blocks a CV whose numbers or facts are missing from your sources, unless you pass `--skip-fact-check`. The `pdf` mode also runs `cv-title-check.mjs`, which compares each job title in the tailored CV with the title cv.md gives the same job (matched by company and dates) and asks you to keep or revert any that differ. It warns rather than blocks, because reframing a title is sometimes your call. Nothing judges rewording yet: [#1411](https://github.com/career-ops-hq/career-ops/issues/1411) (fail-closed faithfulness check) tracks that. Until then, read every CV before you send it. The [Legal Disclaimer](LEGAL_DISCLAIMER.md) says the same in longer words.
  413. **Can I run career-ops for free, or on a cheaper / local model?**
  414. Yes. career-ops is CLI-agnostic and runs on free and local models (OpenRouter free models, Ollama or any OpenAI-compatible endpoint), so you are not tied to a paid subscription. See [docs/RUNNING_ON_A_BUDGET.md](docs/RUNNING_ON_A_BUDGET.md) for the full setup.
  415. **I pay for Claude Pro/Max but career-ops is burning API credits. Why?**
  416. Because an `ANTHROPIC_API_KEY` in your environment takes precedence over your logged-in subscription: the CLI uses the key and bills per token. Run `echo $ANTHROPIC_API_KEY`, and if it prints anything, remove it from your shell profile, restart the terminal and run `/login`. Batch mode is the exception, since `claude -p` workers do not use the interactive login: run `claude setup-token` once and export the result as `CLAUDE_CODE_OAUTH_TOKEN`. Full walkthrough in [docs/RUNNING_ON_A_BUDGET.md](docs/RUNNING_ON_A_BUDGET.md#2b-already-paying-for-a-subscription-make-sure-you-are-using-it).
  417. **Which AI CLIs does career-ops work with?**
  418. career-ops runs on any major AI coding CLI (Claude Code, Codex, Gemini / Antigravity, OpenCode, Grok, Qwen and more) through the open Agent Skill Standard, so it is never locked to a single vendor. Use the CLI you already have.
  419. **How do I install career-ops on Windows?**
  420. career-ops runs on Windows. Platform-specific setup and the known sharp edges (Git Bash discovery, line endings, Task Scheduler) are in [docs/WINDOWS.md](docs/WINDOWS.md). If skills fail to load with a symlink error during install, the fix is in [docs/FAQ.md](docs/FAQ.md). Full steps are in [docs/SETUP.md](docs/SETUP.md). On NixOS, use the flake dev shell instead of `npx playwright install`; see [docs/NIXOS.md](docs/NIXOS.md).
  421. **Does career-ops auto-apply to jobs for me?**
  422. No. career-ops is a filter, not a spray-and-pray auto-applier. The AI evaluates, ranks and drafts; you review and decide. It drafts and fills; it never submits. You press Submit. That human-in-the-loop design is the whole point.
  423. **Is career-ops free and open source?**
  424. Yes. career-ops is free and open source, and for the candidate it always will be. It is the first reference implementation of the [CareerOps Manifesto](https://career-ops.org/manifesto). Read it, and if it says what you believe, sign it.
  425. ## About the Author
  426. I'm [Santiago Fernรกndez de Valderrama Aparicio](https://santifer.io/about) (santifer), a former founder: I built and sold a business that still runs with my name on it. I built career-ops to manage my own job search, and it worked: it landed me a Head of Applied AI role. Six months later I left that role to focus on building career-ops full time.
  427. Curious how a repo this size is maintained with a fleet of AI agents and a human deciding every merge? Read [Agentic maintenance: how career-ops is run by a fleet of AI agents](https://santifer.io/ai-agent-fleet).
  428. My portfolio and other open source projects โ†’ [santifer.io](https://santifer.io)
  429. Wikidata: [Santiago Fernรกndez de Valderrama Aparicio](https://www.wikidata.org/wiki/Q138710224) ยท [career-ops](https://www.wikidata.org/wiki/Q139007988).
  430. ## Disclaimer
  431. **career-ops is a local, open-source tool, NOT a hosted service.** By using this software, you acknowledge:
  432. 1. **You control your data.** Your CV, contact info, and personal data stay on your machine and are sent directly to the AI provider you choose (Anthropic, OpenAI, etc.). We do not collect, store, or have access to any of your data.
  433. 2. **You control the AI.** The default prompts instruct the AI not to auto-submit applications, but AI models can behave unpredictably. If you modify the prompts or use different models, you do so at your own risk. **Always review AI-generated content for accuracy before submitting.**
  434. 3. **You comply with third-party ToS.** You must use this tool in accordance with the Terms of Service of the career portals you interact with (Greenhouse, Lever, Workday, LinkedIn, etc.). Do not use this tool to spam employers or overwhelm ATS systems.
  435. 4. **No guarantees.** Evaluations are recommendations, not truth. AI models may hallucinate skills or experience. The authors are not liable for employment outcomes, rejected applications, account restrictions, or any other consequences.
  436. See [LEGAL_DISCLAIMER.md](LEGAL_DISCLAIMER.md) for full details. This software is provided under the [MIT License](LICENSE) "as is", without warranty of any kind.
  437. ## Contributors
  438. <a href="https://github.com/career-ops-hq/career-ops/graphs/contributors">
  439. <img src="https://contrib.rocks/image?repo=career-ops-hq/career-ops&v=2" />
  440. </a>
  441. Every person who has shipped code, docs, translations or tests is listed in
  442. [CONTRIBUTORS.md](CONTRIBUTORS.md), including non-code contributions, which
  443. the graph above cannot show.
  444. Got hired using career-ops? [Share your story!](https://github.com/career-ops-hq/career-ops/issues/new?template=i-got-hired.yml)
  445. ## License & Trademark
  446. The code is licensed under [MIT](LICENSE). The "career-ops" name and
  447. brand are governed by the [Trademark Policy](TRADEMARK.md), permissive
  448. for community use, reserved for commercial product naming and
  449. endorsement.
  450. ## Star history
  451. <p align="center">
  452. <a href="https://warpchart.dev/hq">
  453. <picture>
  454. <source media="(prefers-color-scheme: dark)" srcset="https://warpchart.dev/api/chart?theme=dark&v=3">
  455. <img alt="Live star telemetry of career-ops-hq/career-ops" src="https://warpchart.dev/api/chart?theme=light&v=3" loading="lazy">
  456. </picture>
  457. </a>
  458. </p>
  459. <p align="center"><sub>Created and maintained by <a href="https://santifer.io">Santiago Fernรกndez de Valderrama Aparicio</a> (<a href="https://github.com/santifer">@santifer</a>)</sub></p>
  460. ## Let's Connect
  461. [![Website](https://img.shields.io/badge/santifer.io-000?style=for-the-badge&logo=safari&logoColor=white)](https://santifer.io)
  462. [![LinkedIn](https://img.shields.io/badge/LinkedIn-0A66C2?style=for-the-badge&logo=linkedin&logoColor=white)](https://linkedin.com/in/santifer)
  463. [![X](https://img.shields.io/badge/X-000?style=for-the-badge&logo=x&logoColor=white)](https://x.com/santifer)
  464. [![Discord](https://img.shields.io/badge/Discord-5865F2?style=for-the-badge&logo=discord&logoColor=white)](https://discord.gg/8pRpHETxa4)
  465. [![Email](https://img.shields.io/badge/Email-EA4335?style=for-the-badge&logo=gmail&logoColor=white)](mailto:hi@santifer.io)
batch/batch-runner.sh ยท Bundled script ยท 2,601 lines ยท 1 finding marked
  1. #!/usr/bin/env bash
  2. set -euo pipefail
  3. # career-ops batch runner โ€” standalone orchestrator for headless agent workers
  4. # Reads batch-input.tsv, delegates each offer to a worker, tracks state in
  5. # batch-state.tsv for resumability.
  6. #
  7. # Supported CLIs (--cli flag):
  8. # claude โ€” claude -p with --dangerously-skip-permissions (default)
  9. # opencode โ€” opencode run (falls back to ollama launch opencode if not in PATH)
  10. # gemini โ€” gemini -p
  11. # qwen โ€” qwen -p
  12. # grok โ€” grok -p with --always-approve
  13. #
  14. # Only claude supports --strict-mcp-config, the rate-limit/session retry loop,
  15. # and --parallel > 1; other CLIs run sequentially with a single attempt.
  16. SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
  17. PROJECT_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
  18. # Under Git Bash / MSYS, pwd gives a POSIX path (/c/...). Node is a native
  19. # Windows binary and gets it unconverted when MSYS path conversion is off
  20. # (MSYS_NO_PATHCONV=1), so every `node "$PROJECT_DIR/..."` call failed with
  21. # MODULE_NOT_FOUND (#4910). cygpath -m yields C:/... which both bash and node
  22. # resolve, as for jd_file below.
  23. case "$(uname -s)" in
  24. MINGW*|MSYS*|CYGWIN*) command -v cygpath >/dev/null 2>&1 && PROJECT_DIR="$(cygpath -m "$PROJECT_DIR")" ;;
  25. esac
  26. # The user layer (profile, reports, tracker additions) lives under the data
  27. # root, which CAREER_OPS_ROOT / CAREER_OPS_DATA_DIR / a .career-ops-data marker
  28. # can place outside the checkout. Every path here used to be built from
  29. # PROJECT_DIR, so under an external data root workers were scored against the
  30. # checkout's template profile, and their reports and tracker lines were looked
  31. # for in directories reserve-report-num.mjs and merge-tracker.mjs never read
  32. # (#5115). path-resolver.mjs owns the precedence, so ask it instead of
  33. # re-deriving it in bash. It prints nothing when the data root is the checkout,
  34. # which keeps DATA_ROOT spelled exactly like PROJECT_DIR in the default layout.
  35. DATA_ROOT="$PROJECT_DIR"
  36. if [[ -f "$PROJECT_DIR/path-resolver.mjs" ]]; then
  37. if ! resolved_data_root="$(node --input-type=module -e '
  38. import { realpathSync } from "node:fs";
  39. import { dirname } from "node:path";
  40. import { pathToFileURL } from "node:url";
  41. const resolver = process.argv[1];
  42. const { getCareerOpsRoot } = await import(pathToFileURL(resolver).href);
  43. const real = (p) => { try { return realpathSync(p); } catch { return p; } };
  44. const root = getCareerOpsRoot();
  45. if (real(root) !== real(dirname(resolver))) {
  46. process.stdout.write(process.platform === "win32" ? root.replace(/\\/g, "/") : root);
  47. }
  48. ' "$PROJECT_DIR/path-resolver.mjs")"; then
  49. echo "ERROR: could not resolve the career-ops data root through path-resolver.mjs" >&2
  50. exit 1
  51. fi
  52. if [[ -n "$resolved_data_root" ]]; then
  53. DATA_ROOT="$resolved_data_root"
  54. fi
  55. fi
  56. BATCH_DIR="$SCRIPT_DIR"
  57. INPUT_FILE="$BATCH_DIR/batch-input.tsv"
  58. STATE_FILE="$BATCH_DIR/batch-state.tsv"
  59. PROMPT_FILE="$BATCH_DIR/batch-prompt.md"
  60. # The profile follows CAREER_OPS_PROFILE as well as the data root, like every
  61. # other reader does through path-resolver.mjs (#5144). Building it from the
  62. # data root here left a run reading the data root's profile for spend_tier and
  63. # for the prompt's personalization while the .mjs readers used the override.
  64. # The .md prompt files still name config/profile.yml; the script-generated
  65. # runtime block and the injected personalization use $PROFILE_FILE.
  66. PROFILE_FILE="$DATA_ROOT/config/profile.yml"
  67. PROFILE_RESOLVED=false
  68. if [[ -f "$PROJECT_DIR/path-resolver.mjs" ]]; then
  69. if ! resolved_profile_file="$(node --input-type=module -e '
  70. import { pathToFileURL } from "node:url";
  71. const resolver = process.argv[1];
  72. const { getProfilePath } = await import(pathToFileURL(resolver).href);
  73. if (typeof getProfilePath !== "function") {
  74. process.stderr.write("WARN: path-resolver.mjs has no getProfilePath(), so CAREER_OPS_PROFILE is ignored in this run. Update the install to use the override.\n");
  75. } else {
  76. const profile = getProfilePath();
  77. process.stdout.write(process.platform === "win32" ? profile.replace(/\\/g, "/") : profile);
  78. }
  79. ' "$PROJECT_DIR/path-resolver.mjs")"; then
  80. echo "ERROR: could not resolve the career-ops profile path through path-resolver.mjs" >&2
  81. exit 1
  82. fi
  83. if [[ -n "$resolved_profile_file" ]]; then
  84. PROFILE_FILE="$resolved_profile_file"
  85. PROFILE_RESOLVED=true
  86. fi
  87. elif [[ -n "${CAREER_OPS_PROFILE:-}" ]]; then
  88. echo "WARN: no path-resolver.mjs next to this script, so CAREER_OPS_PROFILE is ignored in this run." >&2
  89. fi
  90. if [[ -n "${CAREER_OPS_PROFILE:-}" && "$PROFILE_RESOLVED" == "true" && ! -f "$PROFILE_FILE" ]]; then
  91. # Before the override was honored this read the data root's profile anyway.
  92. # Say it, instead of silently running with no profile at all.
  93. echo "WARN: no profile at $PROFILE_FILE (CAREER_OPS_PROFILE); the prompt gets no profile and spend_tier is not taken from it." >&2
  94. fi
  95. LOGS_DIR="$BATCH_DIR/logs"
  96. DISCARD_LOG="$LOGS_DIR/discard.log"
  97. TRACKER_DIR="$DATA_ROOT/batch/tracker-additions"
  98. REPORTS_DIR="$DATA_ROOT/reports"
  99. # The runner's own bookkeeping (input, state, logs, locks) stays beside the
  100. # script. reserve-report-num.mjs and merge-tracker.mjs read batch-state.tsv
  101. # there too, and a data root's own copy beside it (resolveBatchStatePaths), and
  102. # both skip report numbers recorded as failed. The variable still names the file for every worker, and
  103. # learn.mjs reads it to tell a batch worker. Built from PROJECT_DIR, which is
  104. # the form node can open under Git Bash (see above).
  105. export CAREER_OPS_BATCH_STATE="$PROJECT_DIR/batch/batch-state.tsv"
  106. # Workers run unattended, so nobody can confirm a change to the user's standing
  107. # rules. modes/_shared.md tells an agent that sees this flag not to write them.
  108. export CAREER_OPS_HEADLESS=1
  109. LOCK_FILE="$BATCH_DIR/batch-runner.pid"
  110. PAUSE_FILE="$BATCH_DIR/batch-runner.paused"
  111. THROTTLE_FILE="$BATCH_DIR/batch-runner.throttled"
  112. STATE_LOCK_DIR="$BATCH_DIR/.batch-state.lock"
  113. STATE_LOCK_PID_FILE="$STATE_LOCK_DIR/pid"
  114. STATE_LOCK_FLOCK_FILE="$BATCH_DIR/.batch-state.flock"
  115. STATE_LOCK_FLOCK=0
  116. STATE_LOCK_TIMEOUT_SECONDS=30
  117. MAIN_PID="${BASHPID:-$$}"
  118. # Defaults
  119. PARALLEL=""
  120. DEFAULT_CLAUDE_PARALLEL=3
  121. DRY_RUN=false
  122. RETRY_FAILED=false
  123. RESUME_PAUSED=false
  124. START_FROM=0
  125. SCOPE_COMPANY=""
  126. MAX_RETRIES=2
  127. MIN_SCORE=0
  128. SKIP_PDF=false
  129. MODE="full"
  130. MODEL="" # explicit override; otherwise resolved from the profile's spend_tier
  131. RESOLVED_MODEL=""
  132. RESOLVED_SPEND_TIER=""
  133. CLI=claude
  134. RATE_LIMIT_SLEEP=300
  135. WORKER_TIMEOUT=1800
  136. THIN_ACTION=webfetch # webfetch | skip | browser-extract: what to do when the JD is still empty after the prefetch and the ATS API fallback
  137. BATCH_PAUSED=false
  138. EVIDENCE_FRESHNESS_JSON="" # set once per run by check_evidence_freshness
  139. STATUS_ONLY=false
  140. STATUS_FULL=true
  141. WATCH_MODE=false
  142. LIMIT=0 # internal sentinel only; explicit --limit must be positive
  143. # Return success for non-negative integer or decimal strings.
  144. is_decimal_number() {
  145. [[ "$1" =~ ^[0-9]+([.][0-9]+)?$ ]]
  146. }
  147. usage() {
  148. cat <<'USAGE'
  149. career-ops batch runner โ€” process job offers in batch via headless agent workers
  150. Defaults to claude; other CLIs via --cli. For claude the model is resolved from
  151. spend_tier in config/profile.yml (or in the file CAREER_OPS_PROFILE names)
  152. unless --model overrides it.
  153. Usage: batch-runner.sh [OPTIONS]
  154. Options:
  155. --cli NAME Agent CLI to use: claude (default), opencode, gemini, qwen, grok
  156. --model NAME Model for the CLI (e.g. qwen2.5:32b for opencode/ollama).
  157. For claude, overrides the tier-resolved model (otherwise
  158. the spend_tier in config/profile.yml, or in the file
  159. CAREER_OPS_PROFILE names: economy/standard/premium;
  160. default standard).
  161. --parallel N Number of parallel workers (default: 3 for claude, 1 otherwise).
  162. On a rate limit, the rest of the batch runs on 1 worker.
  163. --dry-run Show what would be processed, don't execute
  164. --retry-failed Only retry offers marked as "failed" in state
  165. --resume-paused Resume offers paused by a Claude session/rate limit
  166. --start-from N Start from offer ID N (skip earlier IDs)
  167. --company A,B Only process offers whose company matches (case- and
  168. punctuation-insensitive substring of the notes' company
  169. half or of the URL, leaving out a shared ATS domain;
  170. comma-separated). Applies before --limit
  171. --limit N Max offers: positive integer; omit for no limit
  172. --max-retries N Max retry attempts per offer (default: 2)
  173. --min-score N Skip PDF/tracker for offers scoring below N (default: 0 = off)
  174. --skip-pdf Skip PDF generation entirely (write โŒ in tracker PDF column)
  175. --worker-timeout N Stop a worker after N seconds and record it as failed
  176. (default: 1800; 0 = no limit)
  177. --mode MODE Evaluation mode: full (default) or match (condensed single-pass CV match)
  178. --single-pass Alias for --mode match
  179. --thin-action MODE What to do when no usable JD text could be fetched (JS-rendered
  180. page, empty shell): 'webfetch' (default) spawns the worker and
  181. lets it try WebFetch; 'skip' records the offer as skipped with
  182. a needs-playwright marker and spawns no worker; 'browser-extract'
  183. renders the page with browser-extract.mjs (headless Chromium,
  184. zero model tokens, needs 'npx playwright install chromium') and
  185. hands the recovered text to the worker
  186. --rate-limit-sleep N Maximum adaptive retry delay in seconds; 0 pauses immediately
  187. (default: 300; range: 0โ€“2147483647; claude only)
  188. --status Show batch progress and a per-job table, then exit
  189. --summary With --status, print the totals only (the table is one row per offer)
  190. --full With --status, print the per-job table (the default)
  191. --watch Live-refresh progress (always with the per-job table) until the run completes
  192. -h, --help Show this help
  193. Files:
  194. batch-input.tsv Input offers (id, url, source, notes)
  195. batch-state.tsv Processing state (auto-managed)
  196. batch-prompt.md Prompt template for workers
  197. logs/ Per-offer logs
  198. tracker-additions/ Tracker lines for post-batch merge
  199. With a data root outside the checkout (CAREER_OPS_ROOT, CAREER_OPS_DATA_DIR
  200. or a .career-ops-data marker), config/profile.yml, reports/ and
  201. batch/tracker-additions/ are read and written there. The other files stay
  202. beside this script. CAREER_OPS_PROFILE, when set, names the profile read for
  203. spend_tier and injected into the prompt, as it does for the other scripts.
  204. Examples:
  205. # Dry run to see pending offers
  206. ./batch-runner.sh --dry-run
  207. # Process all pending
  208. ./batch-runner.sh
  209. # Retry only failed offers
  210. ./batch-runner.sh --retry-failed
  211. # Process 2 at a time starting from ID 10 (claude only)
  212. ./batch-runner.sh --parallel 2 --start-from 10
  213. # Local LLM via OpenCode (free, runs sequentially)
  214. ./batch-runner.sh --cli opencode --model qwen2.5:32b
  215. USAGE
  216. }
  217. # Parse arguments
  218. while [[ $# -gt 0 ]]; do
  219. case "$1" in
  220. --cli) CLI="$2"; shift 2 ;;
  221. --parallel) PARALLEL="$2"; shift 2 ;;
  222. --dry-run) DRY_RUN=true; shift ;;
  223. --retry-failed) RETRY_FAILED=true; shift ;;
  224. --resume-paused) RESUME_PAUSED=true; shift ;;
  225. --start-from) START_FROM="$2"; shift 2 ;;
  226. --company)
  227. [[ $# -ge 2 && -n "$2" && "$2" != -* ]] || { echo "ERROR: --company requires an argument"; exit 1; }
  228. SCOPE_COMPANY="$2"; shift 2 ;;
  229. --limit)
  230. if [[ $# -lt 2 ]] || ! [[ "$2" =~ ^[0-9]+$ ]] || [[ "$2" =~ ^0+$ ]]; then
  231. echo "ERROR: --limit must be a positive integer; omit --limit for no limit."
  232. exit 1
  233. fi
  234. # Remove leading zeroes before Bash arithmetic (which otherwise reads octal).
  235. LIMIT="${2#"${2%%[!0]*}"}"
  236. # Compare decimal strings before arithmetic can wrap an oversized integer.
  237. if [[ ${#LIMIT} -gt 19 ]] || { [[ ${#LIMIT} -eq 19 ]] && [[ "$LIMIT" > 9223372036854775807 ]]; }; then
  238. echo "ERROR: --limit must be at most 9223372036854775807; omit --limit for no limit."
  239. exit 1
  240. fi
  241. shift 2
  242. ;;
  243. --max-retries) MAX_RETRIES="$2"; shift 2 ;;
  244. --min-score) MIN_SCORE="$2"; shift 2 ;;
  245. --skip-pdf) SKIP_PDF=true; shift ;;
  246. --mode)
  247. [[ $# -ge 2 ]] || { echo "ERROR: --mode requires an argument (full or match)"; exit 1; }
  248. MODE="$2"
  249. if [[ "$MODE" == "match" || "$MODE" == "single-pass" ]]; then
  250. MODE="match"
  251. fi
  252. shift 2
  253. ;;
  254. --single-pass)
  255. MODE="match"
  256. shift
  257. ;;
  258. --rate-limit-sleep)
  259. [[ $# -ge 2 ]] || { echo "ERROR: --rate-limit-sleep requires an argument"; exit 1; }
  260. RATE_LIMIT_SLEEP="$2"
  261. shift 2
  262. ;;
  263. --model) MODEL="$2"; shift 2 ;;
  264. --thin-action)
  265. [[ $# -ge 2 ]] || { echo "ERROR: --thin-action requires an argument (skip, webfetch or browser-extract)"; exit 1; }
  266. case "$2" in
  267. skip|webfetch|browser-extract) THIN_ACTION="$2" ;;
  268. *) echo "ERROR: --thin-action must be 'skip', 'webfetch' or 'browser-extract' (got '$2')."; exit 1 ;;
  269. esac
  270. shift 2
  271. ;;
  272. --worker-timeout)
  273. if [[ $# -lt 2 ]] || ! [[ "$2" =~ ^[0-9]+$ ]]; then
  274. echo "ERROR: --worker-timeout must be a non-negative integer (seconds)."
  275. exit 1
  276. fi
  277. # Strip leading zeroes, then compare as a decimal string: Bash arithmetic
  278. # wraps past 2^63-1, and a wrapped value at or below 0 disables the watchdog.
  279. WORKER_TIMEOUT="${2#"${2%%[!0]*}"}"
  280. WORKER_TIMEOUT="${WORKER_TIMEOUT:-0}"
  281. if [[ ${#WORKER_TIMEOUT} -gt 19 ]] || { [[ ${#WORKER_TIMEOUT} -eq 19 ]] && [[ "$WORKER_TIMEOUT" > 9223372036854775807 ]]; }; then
  282. echo "ERROR: --worker-timeout must be at most 9223372036854775807 seconds."
  283. exit 1
  284. fi
  285. shift 2
  286. ;;
  287. --status) STATUS_ONLY=true; shift ;;
  288. --summary) STATUS_FULL=false; shift ;;
  289. --full) STATUS_FULL=true; shift ;;
  290. --watch) WATCH_MODE=true; shift ;;
  291. -h|--help) usage; exit 0 ;;
  292. *) echo "Unknown option: $1"; usage; exit 1 ;;
  293. esac
  294. done
  295. if [[ "$MODE" != "full" && "$MODE" != "match" ]]; then
  296. echo "ERROR: Unknown --mode '$MODE'. Supported: full, match."
  297. exit 1
  298. fi
  299. MATCH_OUTPUT_LANGUAGE=""
  300. if [[ "$MODE" == "match" ]]; then
  301. PROMPT_FILE="$BATCH_DIR/batch-prompt-match.md"
  302. SKIP_PDF=true
  303. # #5297: a match worker never sees the user profile, so targeting and
  304. # compensation cannot move the score. language.output is the one setting it
  305. # gets, resolved here once per run with the same reader the evaluators use.
  306. if [[ -f "$PROFILE_FILE" ]]; then
  307. MATCH_OUTPUT_LANGUAGE="$(node --input-type=module -e '
  308. import { readFileSync } from "node:fs";
  309. import { pathToFileURL } from "node:url";
  310. const { parseOutputLanguage } = await import(pathToFileURL(process.argv[1]).href);
  311. process.stdout.write(parseOutputLanguage(readFileSync(process.argv[2], "utf-8")));
  312. ' "$PROJECT_DIR/profile-language.mjs" "$PROFILE_FILE" 2>/dev/null)" || {
  313. MATCH_OUTPUT_LANGUAGE=""
  314. echo "WARN: could not read language.output from $PROFILE_FILE; match workers will write in en." >&2
  315. }
  316. fi
  317. [[ -n "$MATCH_OUTPUT_LANGUAGE" ]] || MATCH_OUTPUT_LANGUAGE="en"
  318. fi
  319. if [[ -z "$PARALLEL" ]]; then
  320. if [[ "$CLI" == "claude" ]]; then PARALLEL=$DEFAULT_CLAUDE_PARALLEL; else PARALLEL=1; fi
  321. fi
  322. if ! [[ "$RATE_LIMIT_SLEEP" =~ ^[0-9]+$ ]]; then
  323. echo "ERROR: --rate-limit-sleep must be a non-negative integer (seconds)."
  324. exit 1
  325. fi
  326. # Normalize decimal input before shell arithmetic (08 must not mean octal),
  327. # and bound it before conversion so oversized input cannot wrap around.
  328. RATE_LIMIT_SLEEP=$(printf '%s' "$RATE_LIMIT_SLEEP" | sed 's/^0*//')
  329. RATE_LIMIT_SLEEP=${RATE_LIMIT_SLEEP:-0}
  330. if [[ ${#RATE_LIMIT_SLEEP} -gt 10 ]] || { [[ ${#RATE_LIMIT_SLEEP} -eq 10 ]] && [[ "$RATE_LIMIT_SLEEP" > 2147483647 ]]; }; then
  331. echo "ERROR: --rate-limit-sleep must not exceed 2147483647 seconds."
  332. exit 1
  333. fi
  334. if ! is_decimal_number "$MIN_SCORE"; then
  335. echo "ERROR: --min-score must be a non-negative number."
  336. exit 1
  337. fi
  338. # Lock file to prevent double execution
  339. # noclobber makes the create atomic, so two runners started together cannot
  340. # both own the lock (and both stop the shared PDF painter on exit).
  341. acquire_lock() {
  342. if ! (set -C; printf '%s\n' "$MAIN_PID" > "$LOCK_FILE") 2>/dev/null; then
  343. local old_pid
  344. old_pid=$(cat "$LOCK_FILE" 2>/dev/null || true)
  345. if [[ -n "$old_pid" ]] && kill -0 "$old_pid" 2>/dev/null; then
  346. echo "ERROR: Another batch-runner is already running (PID $old_pid)"
  347. echo "If this is stale, remove $LOCK_FILE"
  348. exit 1
  349. fi
  350. echo "WARN: Stale lock file found (PID $old_pid not running). Removing."
  351. rm -f "$LOCK_FILE"
  352. if ! (set -C; printf '%s\n' "$MAIN_PID" > "$LOCK_FILE") 2>/dev/null; then
  353. echo "ERROR: Another batch-runner took the lock first"
  354. exit 1
  355. fi
  356. fi
  357. }
  358. release_lock() {
  359. if [[ "${BASHPID:-$$}" != "$MAIN_PID" ]]; then
  360. return
  361. fi
  362. # Stop the painter this run started. A copied runner in a fixture has no
  363. # lib/pdf-server.mjs, and a run that never painted has no pid file. Either
  364. # miss must not fail the batch.
  365. if [[ "${PDF_SERVER_OWNER:-}" == "1" ]]; then
  366. PDF_SERVER_MODULE="$PROJECT_DIR/lib/pdf-server.mjs" PDF_SERVER_ROOT="$PROJECT_DIR" \
  367. node --input-type=module -e 'import { pathToFileURL } from "node:url"; const m = await import(pathToFileURL(process.env.PDF_SERVER_MODULE).href); await m.stopSharedPdfServer(process.env.PDF_SERVER_ROOT);' \
  368. >/dev/null 2>&1 || true
  369. fi
  370. rm -f "$LOCK_FILE"
  371. }
  372. trap release_lock EXIT
  373. # Validate prerequisites
  374. check_prerequisites() {
  375. if [[ ! -f "$INPUT_FILE" ]]; then
  376. echo "ERROR: $INPUT_FILE not found. Add offers first."
  377. exit 1
  378. fi
  379. if [[ ! -f "$PROMPT_FILE" ]]; then
  380. echo "ERROR: $PROMPT_FILE not found."
  381. exit 1
  382. fi
  383. if [[ "$THIN_ACTION" == "browser-extract" && ! -f "$PROJECT_DIR/browser-extract.mjs" ]]; then
  384. echo "ERROR: --thin-action browser-extract needs browser-extract.mjs (and 'npx playwright install chromium')."
  385. exit 1
  386. fi
  387. # Resolve the binary the configured CLI actually needs. opencode can run
  388. # natively or fall back to ollama, so check for either.
  389. local cli_cmd
  390. case "$CLI" in
  391. claude) cli_cmd="claude" ;;
  392. opencode) command -v opencode &>/dev/null && cli_cmd="opencode" || cli_cmd="ollama" ;;
  393. gemini) cli_cmd="gemini" ;;
  394. qwen) cli_cmd="qwen" ;;
  395. grok) cli_cmd="grok" ;;
  396. *) echo "ERROR: Unknown --cli '$CLI'. Supported: claude, opencode, gemini, qwen, grok"; exit 1 ;;
  397. esac
  398. if ! command -v "$cli_cmd" &>/dev/null; then
  399. local level="ERROR"
  400. [[ "$DRY_RUN" == "true" ]] && level="WARN"
  401. echo "$level: '$cli_cmd' not found in PATH (required for --cli $CLI)."
  402. if [[ "$CLI" == "opencode" ]]; then
  403. echo " Install opencode (https://opencode.ai) or Ollama (https://ollama.ai) with an opencode model."
  404. fi
  405. [[ "$DRY_RUN" == "true" ]] || exit 1
  406. fi
  407. # Parallelism, the rate-limit retry loop, and --strict-mcp-config are
  408. # claude-only; local models run one at a time.
  409. if [[ "$CLI" != "claude" && "$PARALLEL" -gt 1 ]]; then
  410. echo "WARN: --parallel >1 is not supported for --cli $CLI (local models run sequentially). Resetting to 1."
  411. PARALLEL=1
  412. fi
  413. mkdir -p "$LOGS_DIR" "$TRACKER_DIR" "$REPORTS_DIR"
  414. }
  415. # Status/watch mode only needs prior batch state, not worker prerequisites.
  416. check_status_prerequisites() {
  417. if [[ ! -f "$STATE_FILE" ]]; then
  418. echo "No state file found at $STATE_FILE"
  419. exit 0
  420. fi
  421. }
  422. # Initialize state file if it doesn't exist
  423. init_state() {
  424. if [[ ! -f "$STATE_FILE" ]]; then
  425. printf 'id\turl\tstatus\tstarted_at\tcompleted_at\treport_num\tscore\terror\tretries\n' > "$STATE_FILE"
  426. fi
  427. }
  428. # A lock held longer than this is assumed abandoned, independent of the
  429. # PID-liveness check below. This exists because `kill -0 $pid` is unreliable
  430. # on Git Bash/Windows (MSYS PIDs from $!/$BASHPID don't reliably map to real
  431. # Windows process IDs), so a genuinely-dead lock holder can otherwise never
  432. # be recovered there and every other worker times out waiting for it.
  433. STATE_LOCK_STALE_AGE_SECONDS=15
  434. acquire_state_lock() {
  435. if [[ "${STATE_LOCK_DISABLED:-0}" -eq 1 ]]; then
  436. return 0
  437. fi
  438. local waited=0
  439. local owner_file=""
  440. local create_misses=0
  441. local max_waits=$((STATE_LOCK_TIMEOUT_SECONDS * 10))
  442. while true; do
  443. if mkdir "$STATE_LOCK_DIR" 2>/dev/null; then
  444. # A unique metadata path binds cleanup to this acquisition, not its successor.
  445. if owner_file=$(mktemp "$STATE_LOCK_DIR/owner.XXXXXXXXXXXX") &&
  446. printf '%s\t%s\n' "${BASHPID:-$$}" "$(date +%s)" > "$owner_file"; then
  447. # mkdir may have belonged to a generation reclaimed before mktemp.
  448. # Register first, then decline if another owner is still registered.
  449. local competing_owner=false registered
  450. for registered in "$STATE_LOCK_PID_FILE" "$STATE_LOCK_DIR"/owner.*; do
  451. if [[ -f "$registered" && "$registered" != "$owner_file" ]]; then
  452. competing_owner=true
  453. break
  454. fi
  455. done
  456. if [[ "$competing_owner" == "true" ]]; then
  457. rm -f "$owner_file" 2>/dev/null || true
  458. rmdir "$STATE_LOCK_DIR" 2>/dev/null || true
  459. continue
  460. fi
  461. STATE_LOCK_OWNED_FILE="$owner_file"
  462. STATE_LOCK_OWNED=1
  463. return 0
  464. fi
  465. if [[ -n "$owner_file" ]]; then rm -f "$owner_file" 2>/dev/null || true; fi
  466. rmdir "$STATE_LOCK_DIR" 2>/dev/null || true
  467. echo "ERROR: Failed to initialize state lock metadata at $STATE_LOCK_DIR" >&2
  468. return 1
  469. fi
  470. if [[ ! -d "$STATE_LOCK_DIR" ]]; then
  471. # mkdir can fail because the lock exists and the holder can release it
  472. # before this check (a wide window on Windows/Git Bash): retry mkdir a few
  473. # times before treating the directory as impossible to create.
  474. if (( create_misses < 3 )); then
  475. create_misses=$((create_misses + 1))
  476. continue
  477. fi
  478. if (( PARALLEL <= 1 )); then
  479. echo "WARN: State lock creation failed. Falling back to lock-free operation (single-worker mode)." >&2
  480. STATE_LOCK_DISABLED=1
  481. STATE_LOCK_OWNED=0
  482. return 0
  483. fi
  484. echo "ERROR: Failed to create state lock directory $STATE_LOCK_DIR" >&2
  485. return 1
  486. fi
  487. create_misses=0
  488. local lock_file="" candidate
  489. # Accept a dead lock left by the older fixed-name protocol as well.
  490. for candidate in "$STATE_LOCK_PID_FILE" "$STATE_LOCK_DIR"/owner.*; do
  491. if [[ -f "$candidate" ]]; then lock_file="$candidate"; break; fi
  492. done
  493. if [[ -n "$lock_file" ]]; then
  494. local lock_pid lock_epoch
  495. lock_pid=$(cut -f1 "$lock_file" 2>/dev/null || true)
  496. lock_epoch=$(cut -f2 "$lock_file" 2>/dev/null || true)
  497. local stale=false
  498. local stale_reason=""
  499. # SAFETY INVARIANT: never treat the lock as stale while kill -0
  500. # positively confirms the recorded PID is still running โ€” a
  501. # confirmed-alive owner may still write update_state_unlocked's
  502. # rewrite of STATE_FILE, and reclaiming under it would let two
  503. # processes rewrite $STATE_FILE.tmp concurrently (real data loss).
  504. # The age-based fallback below only ever fires when the PID check
  505. # could NOT confirm liveness (empty/missing PID, or kill -0 itself
  506. # reported not-running) โ€” it narrows, but does not replace, the PID
  507. # check. This intentionally leaves one Windows/Git-Bash edge case
  508. # unhandled: a `kill -0` FALSE POSITIVE (reports alive for a PID
  509. # that Windows has actually reused for an unrelated process). That
  510. # gap is accepted because the alternative โ€” reclaiming while any
  511. # chance remains the owner is genuinely alive โ€” risks silent
  512. # concurrent-write corruption, which is worse than this lock
  513. # occasionally timing out (recoverable via retry) in that rare case.
  514. local pid_confirmed_alive=false
  515. if [[ -n "$lock_pid" ]] && kill -0 "$lock_pid" 2>/dev/null; then
  516. pid_confirmed_alive=true
  517. fi
  518. if [[ "$pid_confirmed_alive" == "false" ]]; then
  519. if [[ -n "$lock_pid" ]]; then
  520. stale=true
  521. stale_reason="PID $lock_pid not running"
  522. elif [[ "$lock_epoch" =~ ^[0-9]+$ ]]; then
  523. local now age
  524. now=$(date +%s)
  525. age=$((now - lock_epoch))
  526. if (( age >= STATE_LOCK_STALE_AGE_SECONDS )); then
  527. stale=true
  528. stale_reason="lock age ${age}s >= ${STATE_LOCK_STALE_AGE_SECONDS}s (no PID recorded to check liveness against)"
  529. fi
  530. fi
  531. fi
  532. if [[ "$stale" == "true" ]]; then
  533. rm -f "$lock_file"
  534. if rmdir "$STATE_LOCK_DIR" 2>/dev/null; then
  535. echo "WARN: Recovered stale state lock ($stale_reason)." >&2
  536. continue
  537. fi
  538. fi
  539. fi
  540. if (( waited >= max_waits )); then
  541. echo "ERROR: Timed out waiting for state lock at $STATE_LOCK_DIR" >&2
  542. echo "If no batch-runner worker is active, remove the stale lock directory." >&2
  543. return 1
  544. fi
  545. sleep 0.1
  546. ((waited += 1))
  547. done
  548. }
  549. release_state_lock() {
  550. if [[ "${STATE_LOCK_OWNED:-0}" -ne 1 ]]; then
  551. return
  552. fi
  553. rm -f "$STATE_LOCK_OWNED_FILE" 2>/dev/null || true
  554. rmdir "$STATE_LOCK_DIR" 2>/dev/null || true
  555. STATE_LOCK_OWNED=0
  556. }
  557. run_with_state_lock() {
  558. if [[ "$STATE_LOCK_FLOCK" == "1" ]]; then
  559. run_with_state_flock "$@" || return $?
  560. return 0
  561. fi
  562. acquire_state_lock || return $?
  563. local status=0
  564. if "$@"; then
  565. status=0
  566. else
  567. status=$?
  568. fi
  569. release_state_lock
  570. return "$status"
  571. }
  572. # The mkdir lock is only as atomic as mkdir(1). On Ubuntu 26.04 mkdir is uutils
  573. # coreutils, and before uutils 0.11.0 it checks that the directory is absent,
  574. # then creates it, and exits 0 when a racing mkdir made it in between. Two
  575. # workers both took the lock, reserved the same report number and rewrote
  576. # batch-state.tsv together, with nothing printed (#6362). flock(1) leaves the
  577. # excluding to the kernel. main() calls this once, before any worker starts,
  578. # so every worker takes the same lock. Without flock (stock macOS, Git Bash)
  579. # mkdir(1) is atomic and the mkdir lock stays, as it does on a file system
  580. # that refuses flock.
  581. select_state_lock() {
  582. command -v flock >/dev/null 2>&1 || return 0
  583. if { flock -x -n 9; } 2>/dev/null 9>>"$STATE_LOCK_FLOCK_FILE"; then
  584. STATE_LOCK_FLOCK=1
  585. else
  586. echo "WARN: flock cannot lock $STATE_LOCK_FLOCK_FILE; using the mkdir state lock." >&2
  587. fi
  588. }
  589. # Polls every 0.1s up to STATE_LOCK_TIMEOUT_SECONDS, like the mkdir lock
  590. # (BusyBox flock has no -w). The lock lives on fd 9, open only inside this
  591. # function, so the workers, their CLI and the PDF painter never inherit it. A
  592. # program started inside the critical section does inherit it, and holds the
  593. # lock until it exits (9>&- would not stop that: bash saves fd 9 to another
  594. # descriptor that the program still gets), so start nothing long-lived there.
  595. # The lock file is never removed: a worker waiting on the old file and one
  596. # locking a new one would both get in.
  597. run_with_state_flock() {
  598. local status=1 waited=0
  599. local max_waits=$((STATE_LOCK_TIMEOUT_SECONDS * 10))
  600. {
  601. while true; do
  602. if flock -x -n 9; then
  603. status=0
  604. "$@" || status=$?
  605. break
  606. fi
  607. if (( waited >= max_waits )); then
  608. echo "ERROR: Timed out waiting for state lock at $STATE_LOCK_FLOCK_FILE" >&2
  609. break
  610. fi
  611. sleep 0.1
  612. waited=$((waited + 1))
  613. done
  614. } 9>>"$STATE_LOCK_FLOCK_FILE"
  615. return "$status"
  616. }
  617. # Get status of an offer from state file
  618. get_status() {
  619. local id="$1"
  620. if [[ ! -f "$STATE_FILE" ]]; then
  621. echo "none"
  622. return
  623. fi
  624. local status
  625. status=$(awk -F'\t' -v id="$id" '$1 == id { print $3 }' "$STATE_FILE")
  626. echo "${status:-none}"
  627. }
  628. # Get retry count for an offer
  629. get_retries() {
  630. local id="$1"
  631. if [[ ! -f "$STATE_FILE" ]]; then
  632. echo "0"
  633. return
  634. fi
  635. local retries
  636. retries=$(awk -F'\t' -v id="$id" '$1 == id { print $9 }' "$STATE_FILE")
  637. echo "${retries:-0}"
  638. }
  639. # Read spend_tier from $PROFILE_FILE (config/profile.yml, or the file
  640. # CAREER_OPS_PROFILE names). Defaults to "standard" if the key is absent or
  641. # invalid.
  642. read_spend_tier() {
  643. local raw=""
  644. if [[ -f "$PROFILE_FILE" ]]; then
  645. raw=$(
  646. awk -F: '
  647. /^[[:space:]]*spend_tier[[:space:]]*:/ {
  648. value = substr($0, index($0, ":") + 1)
  649. print value
  650. exit
  651. }
  652. ' "$PROFILE_FILE"
  653. )
  654. raw="${raw%%#*}"
  655. raw="${raw//$'\r'/}"
  656. raw="${raw#"${raw%%[![:space:]]*}"}"
  657. raw="${raw%"${raw##*[![:space:]]}"}"
  658. case "$raw" in
  659. \"*\") raw="${raw#\"}"; raw="${raw%\"}" ;;
  660. \'*\') raw="${raw#\'}"; raw="${raw%\'}" ;;
  661. esac
  662. raw="$(printf '%s' "$raw" | tr '[:upper:]' '[:lower:]')"
  663. fi
  664. case "$raw" in
  665. economy|standard|premium)
  666. printf '%s\n' "$raw"
  667. ;;
  668. "")
  669. printf '%s\n' "standard"
  670. ;;
  671. *)
  672. echo "WARN: Invalid spend_tier \"$raw\" in ${PROFILE_FILE#"$DATA_ROOT/"}; falling back to standard." >&2
  673. printf '%s\n' "standard"
  674. ;;
  675. esac
  676. }
  677. # Tier -> model mapping. Keep in sync with the table in modes/_shared.md.
  678. spend_tier_to_model() {
  679. case "$1" in
  680. economy) echo "claude-haiku-5-5" ;;
  681. premium) echo "claude-opus-5-5" ;;
  682. standard|*) echo "claude-sonnet-5-5" ;;
  683. esac
  684. }
  685. # Run one worker CLI with stdin closed, so a CLI that waits for input reads EOF
  686. # instead of hanging on the runner's terminal. With WORKER_TIMEOUT > 0 a
  687. # watchdog sends TERM at the limit and KILL 10 s later, and the worker exits
  688. # 124. Background jobs ignore SIGINT in a non-interactive shell, so Ctrl+C is
  689. # forwarded to the worker by hand.
  690. run_worker() {
  691. "$@" </dev/null &
  692. local pid=$!
  693. local watchdog=""
  694. if (( WORKER_TIMEOUT > 0 )); then
  695. (
  696. for (( s = 0; s < WORKER_TIMEOUT; s++ )); do
  697. sleep 1
  698. kill -0 "$pid" 2>/dev/null || exit 0
  699. done
  700. kill -TERM "$pid" 2>/dev/null
  701. for (( s = 0; s < 10; s++ )); do
  702. sleep 1
  703. kill -0 "$pid" 2>/dev/null || exit 124
  704. done
  705. kill -KILL "$pid" 2>/dev/null
  706. exit 124
  707. ) &
  708. watchdog=$!
  709. fi
  710. trap 'kill -TERM "$pid" 2>/dev/null; exit 130' INT TERM
  711. local rc=0
  712. wait "$pid" || rc=$?
  713. trap - INT TERM
  714. if [[ -n "$watchdog" ]]; then
  715. local watchdog_rc=0
  716. wait "$watchdog" || watchdog_rc=$?
  717. if (( watchdog_rc == 124 )); then
  718. echo "batch-runner: worker timed out after ${WORKER_TIMEOUT}s"
  719. rc=124
  720. fi
  721. fi
  722. return "$rc"
  723. }
  724. # Resolve the model to pass to the worker CLI. --model always wins.
  725. # spend_tier maps to Claude model names, so it only applies to --cli claude;
  726. # other CLIs use --model verbatim, or their own default when it is unset.
  727. resolve_worker_model() {
  728. if [[ -n "$MODEL" ]]; then
  729. RESOLVED_MODEL="$MODEL"
  730. RESOLVED_SPEND_TIER="override"
  731. return 0
  732. fi
  733. if [[ "$CLI" != "claude" ]]; then
  734. RESOLVED_MODEL=""
  735. RESOLVED_SPEND_TIER="cli-default"
  736. return 0
  737. fi
  738. RESOLVED_SPEND_TIER="$(read_spend_tier)"
  739. RESOLVED_MODEL="$(spend_tier_to_model "$RESOLVED_SPEND_TIER")"
  740. }
  741. # Append a one-line, auditable record of a pre-screen-gate discard to
  742. # batch/logs/discard.log (see modes/batch.md โ€” Pre-screen gate). Format:
  743. # {ISO8601 timestamp}\t{job id}\t{url}\t{reason}
  744. #
  745. # Callers write the "skipped" state first and call this only when that state
  746. # (or its recovery record) was written, so the same id, url and reason are on
  747. # disk. A failed append is therefore reported, not fatal: the record can be
  748. # rebuilt from state instead of aborting the run.
  749. log_discard() {
  750. local id="$1" url="$2" reason="$3"
  751. local ts
  752. ts="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
  753. if ! { mkdir -p "$LOGS_DIR" && printf '%s\t%s\t%s\t%s\n' "$ts" "$id" "$url" "$reason" >> "$DISCARD_LOG"; } 2>/dev/null; then
  754. echo " โš ๏ธ Could not append to $DISCARD_LOG for #$id; its skipped state record keeps the reason" >&2
  755. fi
  756. }
  757. # The mode a plain redirect gives a new file (0644 under umask 022). mktemp
  758. # creates 0600, and the state file keeps the mode it has always had.
  759. STATE_FILE_MODE=$(printf '%o' "$(( 0666 & ~0$(umask) ))")
  760. # Update or insert state for an offer.
  761. # Caller must hold the state lock (run_with_state_lock) while this runs.
  762. update_state_unlocked() {
  763. local id="$1" url="$2" status="$3" started="$4" completed="$5" report_num="$6" score="$7" error="$8" retries="$9"
  764. # batch-state.tsv is tab-separated with one row per line -- a literal tab,
  765. # newline, or carriage return inside $error (e.g. from a worker's raw error
  766. # text, or JSON.parse unescaping \n/\r/\t in a caller upstream) would split
  767. # into extra columns or extra rows and corrupt every row after it. Collapse
  768. # them to spaces centrally here so every caller is protected, not just the
  769. # one that happened to trigger this.
  770. error=${error//$'\r'/ }
  771. error=${error//$'\n'/ }
  772. error=${error//$'\t'/ }
  773. if [[ ! -f "$STATE_FILE" ]]; then
  774. init_state
  775. fi
  776. # A tmp file of its own per rewrite, beside the state file so mv stays a
  777. # rename. With one shared $STATE_FILE.tmp, two rewrites that overlapped
  778. # truncated and renamed each other's file: rows doubled and mv failed (#6362).
  779. local tmp
  780. tmp=$(mktemp "$STATE_FILE.tmp.XXXXXX") || return 1
  781. chmod "$STATE_FILE_MODE" "$tmp" 2>/dev/null || true
  782. local found=false
  783. # Write header
  784. head -1 "$STATE_FILE" > "$tmp"
  785. # Process existing lines. Don't split a row with `IFS=$'\t' read`: tab is IFS
  786. # whitespace, so the two tabs around an empty cell (an empty error from a
  787. # worker that died without output, or an empty completed_at/score written by
  788. # reconcile_recovery_records) collapse into one and every later cell shifts
  789. # left -- `retries` became the error text and the retry counter was lost.
  790. # Read the line raw and split on \x1f after translating the tabs, as
  791. # reconcile_recovery_records does.
  792. while IFS= read -r sline; do
  793. IFS=$'\x1f' read -r sid surl sstatus sstarted scompleted sreport sscore serror sretries <<< "${sline//$'\t'/$'\x1f'}"
  794. [[ "$sid" == "id" ]] && continue # skip header
  795. if [[ "$sid" == "$id" ]]; then
  796. printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n' \
  797. "$id" "$url" "$status" "$started" "$completed" "$report_num" "$score" "$error" "$retries" >> "$tmp"
  798. found=true
  799. else
  800. printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n' \
  801. "$sid" "$surl" "$sstatus" "$sstarted" "$scompleted" "$sreport" "$sscore" "$serror" "$sretries" >> "$tmp"
  802. fi
  803. done < "$STATE_FILE"
  804. if [[ "$found" == "false" ]]; then
  805. printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n' \
  806. "$id" "$url" "$status" "$started" "$completed" "$report_num" "$score" "$error" "$retries" >> "$tmp"
  807. fi
  808. if ! mv -f "$tmp" "$STATE_FILE"; then
  809. rm -f "$tmp"
  810. return 1
  811. fi
  812. }
  813. update_state() {
  814. run_with_state_lock update_state_unlocked "$@"
  815. }
  816. # Durable last-resort records of state transitions that could NOT be written
  817. # into $STATE_FILE (state-lock exhausted its retries). ONE FILE PER RECORD:
  818. # each failed transition gets its own uniquely-named file via mktemp
  819. # (O_CREAT|O_EXCL โ€” atomic creation, guaranteed-unique name), so no two
  820. # workers ever write to the same file and no shared-file truncate/append
  821. # race can exist on any platform. That matters here because recovery writes
  822. # are CORRELATED, not independent: they fire exactly when the state lock is
  823. # jammed, which makes all parallel workers fail (and try to record) at the
  824. # same moment โ€” a shared recovery file is racing precisely when it is
  825. # needed most (PR #2417 review). This mechanism must also never depend on
  826. # the state lock that just failed, and it doesn't: creation is the only
  827. # synchronization. reconcile_recovery_records() (start of the next run,
  828. # single-threaded, before any worker spawns) merges each record into
  829. # $STATE_FILE and deletes its file only on success โ€” there is no
  830. # rewrite-and-rename step, so nothing here needs cross-filesystem atomicity.
  831. RECOVERY_DIR="$BATCH_DIR/batch-state-recovery.d"
  832. append_recovery_record() {
  833. local id="$1" url="$2" status="$3" started="$4" completed="$5" report_num="$6" score="$7" error="$8" retries="$9"
  834. # Same rationale as update_state_unlocked: a literal tab/newline/CR in
  835. # $error would split this single-line record into extra fields or rows,
  836. # corrupting the record and the state row it later reconciles into.
  837. error=${error//$'\r'/ }
  838. error=${error//$'\n'/ }
  839. error=${error//$'\t'/ }
  840. # \x1f is used as the in-memory delimiter when this record is read back
  841. # (see reconcile_recovery_records); strip it here too so a stray unit
  842. # separator in $error can't inject a field boundary on the read side.
  843. error=${error//$'\x1f'/ }
  844. mkdir -p "$RECOVERY_DIR" || return 1
  845. local rec
  846. rec=$(mktemp "$RECOVERY_DIR/rec-XXXXXX") || return 1
  847. printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n' \
  848. "$id" "$url" "$status" "$started" "$completed" "$report_num" "$score" "$error" "$retries" > "$rec"
  849. }
  850. # A recovery record is ALWAYS older than whatever $STATE_FILE holds for the
  851. # same id: the record is only written when the lock was unreachable, so any
  852. # row present for that id now was written afterwards by a path that did reach
  853. # the lock. Merging it blindly therefore rolls a finished offer backwards --
  854. # the score and completed_at are overwritten, and since rate_limited and
  855. # failed are NOT terminal, the offer re-enters pending selection in main()
  856. # and gets evaluated a second time. Cost: one lost evaluation plus one
  857. # duplicate run, on a path that by construction only fires when something
  858. # already went wrong.
  859. #
  860. # Completed and human-held states must not be rolled back by stale records.
  861. # Keep these protected states in sync with the pending-selection guard.
  862. recovery_record_is_superseded() {
  863. local current="$1"
  864. [[ "$current" == "completed" || "$current" == "skipped" || "$current" == "needs_confirmation" ]]
  865. }
  866. # Merge one recovery record, but only into a row that has not already reached
  867. # a completed or human-held state. The status read happens INSIDE the lock: a
  868. # check-then-write gap would let a worker finish between the two and
  869. # reintroduce the same rollback. get_status is a lock-free reader (plain awk
  870. # over $STATE_FILE), so calling it here does not re-enter the non-reentrant
  871. # mkdir lock.
  872. #
  873. # Exit codes: 0 merged ยท 3 superseded (record is stale, caller should drop
  874. # it) ยท anything else is a real failure. 3 avoids colliding with the 1 that
  875. # acquire_state_lock returns when the lock itself is unreachable.
  876. reconcile_one_unlocked() {
  877. local id="$1"
  878. local current
  879. current=$(get_status "$id")
  880. if recovery_record_is_superseded "$current"; then
  881. echo " Superseded: offer id=$id already '$current' in state โ€” discarding stale recovery record (would have rolled it back to '$3')."
  882. return 3
  883. fi
  884. update_state_unlocked "$@"
  885. }
  886. reconcile_one() {
  887. run_with_state_lock reconcile_one_unlocked "$@"
  888. }
  889. # Merge any recovery records left by a prior run into $STATE_FILE. Runs
  890. # single-threaded at the very start of main(), before any worker is spawned,
  891. # so there is no lock contention here โ€” this is the one place these records
  892. # are guaranteed a clean shot at the lock. Each record file is deleted only
  893. # after its transition lands in $STATE_FILE (or is found to be superseded);
  894. # genuine failures leave the file in place for the run after that.
  895. reconcile_recovery_records() {
  896. [[ -d "$RECOVERY_DIR" ]] || return 0
  897. local -a rec_files=()
  898. local f
  899. for f in "$RECOVERY_DIR"/rec-*; do
  900. [[ -f "$f" ]] || continue
  901. rec_files+=("$f")
  902. done
  903. if (( ${#rec_files[@]} == 0 )); then
  904. rmdir "$RECOVERY_DIR" 2>/dev/null || true
  905. return 0
  906. fi
  907. echo "=== Reconciling ${#rec_files[@]} recovery record(s) from a prior interrupted run ==="
  908. local merged=0 superseded=0 still_failed=0
  909. local rid rurl rstatus rstarted rcompleted rreport rscore rerror rretries
  910. local rline
  911. local rc
  912. for f in "${rec_files[@]}"; do
  913. rline=""
  914. # Don't split with `IFS=$'\t' read`: tab is IFS *whitespace*, so a run of
  915. # tabs around an empty interior field (e.g. an empty completed_at on a
  916. # non-terminal record) collapses to one delimiter and every later field
  917. # shifts left, corrupting the merge. Read the line raw, then split on \x1f
  918. # (a non-whitespace unit separator that preserves empty fields) after
  919. # translating the on-disk tabs to it. Same rationale as process_offer.
  920. IFS= read -r rline < "$f" || true
  921. IFS=$'\x1f' read -r rid rurl rstatus rstarted rcompleted rreport rscore rerror rretries <<< "${rline//$'\t'/$'\x1f'}"
  922. if [[ -z "$rid" ]]; then
  923. echo " WARN: discarding unreadable recovery record $f" >&2
  924. rm -f "$f"
  925. continue
  926. fi
  927. rc=0
  928. reconcile_one "$rid" "$rurl" "$rstatus" "$rstarted" "$rcompleted" "$rreport" "$rscore" "$rerror" "$rretries" || rc=$?
  929. if (( rc == 0 )); then
  930. rm -f "$f"
  931. merged=$((merged + 1))
  932. elif (( rc == 3 )); then
  933. # Stale by definition, not a failure โ€” the row it targets already
  934. # finished. Dropping the file is correct; keeping it would retry the
  935. # same rollback on every subsequent run.
  936. rm -f "$f"
  937. superseded=$((superseded + 1))
  938. else
  939. still_failed=$((still_failed + 1))
  940. fi
  941. done
  942. echo " Merged: $merged | Superseded: $superseded | Still unrecovered: $still_failed"
  943. if (( still_failed > 0 )); then
  944. echo " WARN: $still_failed record(s) could not be merged even single-threaded โ€” check $STATE_LOCK_DIR for a genuinely stuck lock. Unmerged records remain in $RECOVERY_DIR." >&2
  945. else
  946. rmdir "$RECOVERY_DIR" 2>/dev/null || true
  947. fi
  948. }
  949. # Retry wrapper around update_state. Bare `update_state ...` calls under
  950. # `set -e` will silently kill the entire background worker subshell if a
  951. # single lock-timeout failure propagates โ€” this wrapper retries a few times
  952. # and, if it still fails, falls back to append_recovery_record so the
  953. # transition is never actually lost (only delayed until the next run's
  954. # reconcile step), then logs a clear warning and returns non-zero so the
  955. # CALLER can still decide whether to skip side effects that assumed success.
  956. # Return 1 means the transition is durable in recovery; 2 means neither write
  957. # succeeded, so a human gate must stop and surface the persistence failure
  958. # (found under --parallel 5 on Git Bash/Windows: ~47 of 50 jobs silently
  959. # dropped in one run from exactly this before the retry+recovery-log fix).
  960. update_state_retrying() {
  961. local attempt=0
  962. local max_attempts=3
  963. while (( attempt < max_attempts )); do
  964. if update_state "$@"; then
  965. return 0
  966. fi
  967. attempt=$((attempt + 1))
  968. if (( attempt < max_attempts )); then
  969. echo " โš ๏ธ State update failed (attempt $attempt/$max_attempts), retrying in 2s..." >&2
  970. sleep 2
  971. fi
  972. done
  973. if append_recovery_record "$@"; then
  974. echo " โš ๏ธ State update failed after $max_attempts attempts โ€” offer id=$1 status=$3 recorded to $RECOVERY_DIR for reconciliation on next run." >&2
  975. else
  976. echo " โŒ State update failed after $max_attempts attempts AND recovery-record write also failed โ€” offer id=$1 status=$3 was NOT recorded anywhere. It will be retried as pending next run." >&2
  977. return 2
  978. fi
  979. return 1
  980. }
  981. is_rate_limit_log() {
  982. local log_file="$1"
  983. grep -Eiq '(rate limit|rate_limit|too many requests|429|quota exceeded|try again later|temporarily unavailable)' "$log_file"
  984. }
  985. is_session_limit_log() {
  986. local log_file="$1"
  987. grep -Eiq '(session limit|resets [0-9:]+[ap]m|usage limit|limit[[:space:]]+reached)' "$log_file"
  988. }
  989. # Print a delay or "pause". Only standalone delta-seconds headers are trusted;
  990. # never feed worker text into shell arithmetic. awk compares before formatting,
  991. # so even an arbitrarily long integer header safely requests a pause.
  992. rate_limit_delay() {
  993. local log_file="$1" retry="$2" base delay hint step
  994. if (( RATE_LIMIT_SLEEP == 0 )); then
  995. echo pause
  996. return
  997. fi
  998. base=30
  999. (( base <= RATE_LIMIT_SLEEP )) || base=$RATE_LIMIT_SLEEP
  1000. # A log that exists and is empty has no header to find, and awk would print
  1001. # nothing for it: skip the two processes. A missing log still reaches awk.
  1002. hint=""
  1003. [[ -e "$log_file" && ! -s "$log_file" ]] || hint=$(LC_ALL=C awk -v cap="$RATE_LIMIT_SLEEP" '
  1004. tolower($0) ~ /^[[:space:]]*retry-after:[[:space:]]*[0-9]+[[:space:]]*$/ {
  1005. value=$0
  1006. sub(/^[^:]*:[[:space:]]*/, "", value)
  1007. if (value + 0 > cap) over=1
  1008. if (value + 0 > largest) largest=value + 0
  1009. found=1
  1010. }
  1011. END {
  1012. if (over) print "pause"
  1013. else if (found) printf "%.0f\n", largest
  1014. }
  1015. ' "$log_file")
  1016. if [[ "$hint" == pause ]]; then
  1017. echo pause
  1018. return
  1019. fi
  1020. delay=$base
  1021. if [[ -n "$hint" ]]; then
  1022. (( hint <= delay )) || delay=$hint
  1023. else
  1024. # Saturate before doubling, avoiding exponentiation/overflow even on resume.
  1025. for (( step=0; step<retry && delay<RATE_LIMIT_SLEEP; step++ )); do
  1026. if (( delay > RATE_LIMIT_SLEEP / 2 )); then
  1027. delay=$RATE_LIMIT_SLEEP
  1028. else
  1029. delay=$((delay * 2))
  1030. fi
  1031. done
  1032. fi
  1033. # Upward-only jitter never undercuts Retry-After or the base delay.
  1034. delay=$((delay + delay * (RANDOM % 21) / 100))
  1035. (( delay <= RATE_LIMIT_SLEEP )) || delay=$RATE_LIMIT_SLEEP
  1036. echo "$delay"
  1037. }
  1038. mark_paused_rate_limit() {
  1039. local id="$1" url="$2" started_at="$3" report_num="$4" retries="$5" log_file="$6"
  1040. local completed_at
  1041. completed_at=$(date -u +%Y-%m-%dT%H:%M:%SZ)
  1042. local error_msg
  1043. error_msg=$(tail -5 "$log_file" 2>/dev/null | tr '\n' ' ' | cut -c1-200 || echo "session/rate limit reached")
  1044. update_state_retrying "$id" "$url" "paused_rate_limit" "$started_at" "$completed_at" "$report_num" "-" "$error_msg" "$retries" || true
  1045. printf '%s\t%s\t%s\n' "$id" "$report_num" "$error_msg" > "$PAUSE_FILE"
  1046. BATCH_PAUSED=true
  1047. }
  1048. reserve_report_num_unlocked() {
  1049. local id="$1" url="$2" started="$3" retries="$4"
  1050. # Use the shared, cross-process-atomic reservation system (O_CREAT|O_EXCL
  1051. # sentinel files in reserve-report-num.mjs) instead of the old bash-native
  1052. # max(existing report files, batch-state.tsv numbers)+1 scan. The bash-native
  1053. # version had zero visibility into reservations made by any OTHER process
  1054. # calling `node reserve-report-num.mjs` directly -- e.g. an interactively
  1055. # dispatched Agent evaluating one offer with a browser tool while a batch
  1056. # run is in flight. Both could independently compute the same "next" number
  1057. # and collide on disk. Found 2026-07-30: two separate collisions (report
  1058. # 049, report 051) in one batch run for exactly this reason -- routing every
  1059. # caller through the same node script means they all share one real lock.
  1060. local report_num=""
  1061. report_num=$(node "$PROJECT_DIR/reserve-report-num.mjs" 2>/dev/null | tr -d '[:space:]') || report_num=""
  1062. # No number means the allocator failed (lock timeout, unwritable reports/,
  1063. # exhausted retries). Returning success with an empty number let the caller
  1064. # launch a paid worker with a blank "Report number:" and write an empty
  1065. # report_num cell into batch-state.tsv; the retry wrapper below is what
  1066. # handles a failed reservation, so fail here and let it.
  1067. if [[ ! "$report_num" =~ ^[0-9]+$ ]]; then
  1068. echo " โŒ reserve-report-num.mjs returned no report number for offer id=$id" >&2
  1069. return 1
  1070. fi
  1071. update_state_unlocked "$id" "$url" "processing" "$started" "-" "$report_num" "-" "-" "$retries"
  1072. printf '%s\n' "$report_num"
  1073. }
  1074. # Release a report-number reservation via the shared atomic system. Safe to
  1075. # call even if the number was never actually reserved this way (e.g. a
  1076. # resumed/paused offer) -- the underlying script no-ops on a missing sentinel.
  1077. release_report_num() {
  1078. local report_num="$1"
  1079. [[ -n "$report_num" && "$report_num" != "-" ]] || return 0
  1080. node "$PROJECT_DIR/reserve-report-num.mjs" --release "$report_num" >/dev/null 2>&1 || true
  1081. }
  1082. reserve_report_num() {
  1083. run_with_state_lock reserve_report_num_unlocked "$@"
  1084. }
  1085. # Succeed if reports/ holds a real report for this number. The reservation
  1086. # sentinel ({num}-RESERVED.md) never counts: reserve-report-num.mjs creates it
  1087. # before the worker runs, so counting it would mark a no-op worker completed.
  1088. #
  1089. # A plain glob loop, not `compgen -G`: compgen exists only when bash is built
  1090. # with programmable completion, and some distro builds (NixOS among them) omit
  1091. # it from non-interactive shells. There, compgen fails, the check sees no
  1092. # report, and every worker is recorded as failed even though its report is on
  1093. # disk. merge-tracker.mjs then refuses those reports as possibly fabricated.
  1094. has_report_file() {
  1095. local report_num="$1" f
  1096. for f in "$REPORTS_DIR/${report_num}"-*.md; do
  1097. [[ -e "$f" ]] || continue
  1098. [[ "$f" == *-RESERVED.md ]] && continue
  1099. return 0
  1100. done
  1101. return 1
  1102. }
  1103. # Retry wrapper โ€” same rationale as update_state_retrying above. A bare
  1104. # `x=$(reserve_report_num ...)` under `set -e` kills the worker subshell
  1105. # silently on a single lock-timeout; this retries and logs instead.
  1106. reserve_report_num_retrying() {
  1107. local attempt=0
  1108. local max_attempts=3
  1109. local result=""
  1110. while (( attempt < max_attempts )); do
  1111. if result=$(reserve_report_num "$@"); then
  1112. printf '%s\n' "$result"
  1113. return 0
  1114. fi
  1115. attempt=$((attempt + 1))
  1116. if (( attempt < max_attempts )); then
  1117. echo " โš ๏ธ Report-number reservation failed (attempt $attempt/$max_attempts), retrying in 2s..." >&2
  1118. sleep 2
  1119. fi
  1120. done
  1121. echo " โŒ Report-number reservation failed after $max_attempts attempts for offer id=$1 โ€” leaving it pending for next run." >&2
  1122. return 1
  1123. }
  1124. # Process a single offer
  1125. process_offer() {
  1126. local id="$1" url="$2" source="$3" notes="$4"
  1127. local started_at
  1128. started_at=$(date -u +%Y-%m-%dT%H:%M:%SZ)
  1129. local retries
  1130. retries=$(get_retries "$id")
  1131. local report_num
  1132. if ! report_num=$(reserve_report_num_retrying "$id" "$url" "$started_at" "$retries"); then
  1133. return 1
  1134. fi
  1135. local date
  1136. date=$(date +%Y-%m-%d)
  1137. # Use mktemp instead of a predictable path: a fixed name like
  1138. # /tmp/batch-jd-${id}.txt is guessable, so an attacker on a shared machine
  1139. # could pre-create it as a symlink and redirect or clobber the write.
  1140. # Always project-local, never ${TMPDIR}: the worker CLI has to be able to read
  1141. # this file, and CLIs that auto-reject external_directory (/tmp/*) โ€” opencode
  1142. # does โ€” cannot read anything outside the project. Honouring TMPDIR here would
  1143. # re-open that hole whenever TMPDIR is the usual /tmp. mktemp still randomizes
  1144. # the name, so the symlink-guessability property above is unchanged.
  1145. local jd_tmp_dir="$PROJECT_DIR/batch/tmp"
  1146. mkdir -p "$jd_tmp_dir"
  1147. local jd_file
  1148. jd_file="$(mktemp "$jd_tmp_dir/batch-jd-${id}.XXXXXX")"
  1149. # The worker is a native process. Under Git Bash / MSYS the path above is a
  1150. # POSIX one (/tmp/... or /c/...) that a Windows binary cannot open, so every
  1151. # worker read "JD source unavailable" even when curl had filled the file.
  1152. # cygpath -m yields C:/... which both bash and the worker resolve.
  1153. case "$(uname -s)" in
  1154. MINGW*|MSYS*|CYGWIN*) command -v cygpath >/dev/null 2>&1 && jd_file="$(cygpath -m "$jd_file")" ;;
  1155. esac
  1156. # Pre-populate $jd_file with a static curl fetch so the worker reads HTML
  1157. # directly instead of always falling through to WebFetch (#2492). WebFetch is
  1158. # unreliable on JS-rendered boards (Phenom, Workday, iCIMS) because it hits
  1159. # the rendered JS shell rather than the actual JD text. curl returns the raw
  1160. # HTML in a single round-trip; for static boards that is exactly the JD.
  1161. # For JS-rendered boards the file will be thin (JS shell only), which the
  1162. # sufficiency check below detects โ€” the file is then truncated to 0 bytes so
  1163. # the worker's Step-1 WebFetch fallback fires exactly as designed.
  1164. # If curl is absent or fails, $jd_file stays empty and WebFetch fires too.
  1165. # Minimum visible word count to treat a fetched page as a real JD rather than
  1166. # a JS shell. A JS app shell (Workday, Phenom, iCIMS) has near-zero visible
  1167. # words after HTML stripping; a real JD has hundreds. 80 is a conservative
  1168. # lower bound โ€” any genuine posting has at least a title, summary, and a few
  1169. # requirements, which together exceed 80 stripped words.
  1170. local prefetch_min_words=80
  1171. local jd_prefetch_words=0
  1172. local jd_rescue_closed_note=""
  1173. local jd_raw_file=""
  1174. local curl_status=0
  1175. if command -v curl >/dev/null 2>&1; then
  1176. # Reject loopback, link-local, and private-network destinations before curl
  1177. # connects. --proto/--proto-redir restrict schemes but not destination IPs,
  1178. # so a malicious offer URL could reach cloud metadata (169.254.169.254) or
  1179. # internal services without this guard.
  1180. local _url_safe
  1181. local current_url="$url"
  1182. local redirect_count=0
  1183. local redirect_headers
  1184. local redirect_location
  1185. while :; do
  1186. _url_safe=$(node -e "
  1187. try {
  1188. const u = new URL(process.argv[1]);
  1189. const h = u.hostname.toLowerCase().replace(/\.\$/, '');
  1190. const blocked =
  1191. h === 'localhost' || h === 'localhost.localdomain' ||
  1192. h.endsWith('.local') || h.endsWith('.internal') ||
  1193. h.includes(':') ||
  1194. /^127\./.test(h) || /^169\.254\./.test(h) ||
  1195. /^10\./.test(h) || /^172\.(1[6-9]|2[0-9]|3[01])\./.test(h) ||
  1196. /^192\.168\./.test(h) || /^0\./.test(h);
  1197. process.stdout.write(blocked ? '0' : '1');
  1198. } catch (e) { process.stdout.write('0'); }
  1199. " "$current_url" 2>/dev/null)
  1200. if [[ "$_url_safe" != "1" ]]; then
  1201. echo " โ„น๏ธ JD prefetch: blocked โ€” private/loopback destination ($current_url)"
  1202. : > "$jd_file"
  1203. break
  1204. else
  1205. # Headers are read by curl/bash only โ€” never by the worker โ€” so this one
  1206. # stays on ${TMPDIR:-/tmp} and the expression stays self-contained (the
  1207. # prefetch block is extracted and run in isolation by
  1208. # tests/batch-runner-jd-prefetch.test.mjs, where $jd_tmp_dir is not set).
  1209. redirect_headers="$(mktemp "${TMPDIR:-/tmp}/batch-jd-headers.XXXXXX")"
  1210. curl_status=0
  1211. curl --silent --show-error --location --max-redirs 0 \
  1212. --max-time 20 --connect-timeout 5 \
  1213. --fail --compressed \
  1214. --proto '=http,https' --proto-redir 'https,http' --max-filesize 5000000 \
  1215. --user-agent "Mozilla/5.0 (compatible; career-ops/batch)" \
  1216. --header "Accept: text/html,application/xhtml+xml,*/*;q=0.8" \
  1217. --dump-header "$redirect_headers" \
  1218. --output "$jd_file" \
  1219. -- "$current_url" 2>/dev/null || curl_status=$?
  1220. redirect_location=""
  1221. if [[ "$curl_status" -eq 47 ]]; then
  1222. redirect_location="$(awk 'tolower($0) ~ /^location:[[:space:]]*/ { value=$0; sub(/^[^:]*:[[:space:]]*/, "", value) } END { print value }' "$redirect_headers")"
  1223. fi
  1224. rm -f "$redirect_headers"
  1225. if [[ "$curl_status" -eq 47 && -n "$redirect_location" ]]; then
  1226. if [[ "$redirect_count" -ge 10 ]]; then
  1227. : > "$jd_file"
  1228. echo " โ„น๏ธ JD prefetch: too many redirects โ€” worker will WebFetch"
  1229. break
  1230. fi
  1231. current_url="$(node -e "
  1232. try { process.stdout.write(new URL(process.argv[2], process.argv[1]).href); }
  1233. catch (e) { process.stdout.write(''); }
  1234. " "$current_url" "$redirect_location" 2>/dev/null)"
  1235. if [[ -z "$current_url" ]]; then
  1236. : > "$jd_file"
  1237. break
  1238. fi
  1239. redirect_count=$((redirect_count + 1))
  1240. continue
  1241. fi
  1242. if [[ "$curl_status" -ne 0 ]]; then
  1243. : > "$jd_file"
  1244. fi
  1245. break
  1246. fi
  1247. done
  1248. # Keep the bytes curl downloaded: the strip below discards <script> blocks,
  1249. # which is where a JS-rendered board keeps its JD (schema.org JobPosting,
  1250. # hydration payload), so jd-rescue.mjs reads this copy, not the stripped text.
  1251. jd_raw_file="${jd_file}.raw"
  1252. cp "$jd_file" "$jd_raw_file" 2>/dev/null || true
  1253. # Strip HTML tags and count visible words to distinguish a real JD (hundreds
  1254. # of words) from a JS shell (near zero visible text).
  1255. jd_prefetch_words=$(node -e "
  1256. const fs = require('fs');
  1257. try {
  1258. const text = fs.readFileSync(process.argv[1], 'utf-8')
  1259. .replace(/<script[\s\S]*?<\/script>/gi, ' ')
  1260. .replace(/<style[\s\S]*?<\/style>/gi, ' ')
  1261. .replace(/<[^>]+>/g, ' ')
  1262. .replace(/&(nbsp|#160|#xa0);/gi, ' ')
  1263. .replace(/\s+/g, ' ')
  1264. .trim();
  1265. fs.writeFileSync(process.argv[1], text);
  1266. process.stdout.write(String(text.split(' ').filter(Boolean).length));
  1267. } catch (e) { process.stdout.write('0'); }
  1268. " "$jd_file" 2>/dev/null) || jd_prefetch_words=0
  1269. # Ensure jd_prefetch_words is always a non-negative integer. A non-integer
  1270. # (e.g. empty string, "NaN") would cause bash arithmetic to fail or
  1271. # miscompare. Strip everything that is not a digit and default to 0.
  1272. jd_prefetch_words="${jd_prefetch_words//[^0-9]/}"
  1273. jd_prefetch_words="${jd_prefetch_words:-0}"
  1274. # Zero-token JD rescue: read the JD out of the structured data the board
  1275. # already shipped in the bytes above. It only replaces the text when the page
  1276. # is thin or the rescued JD is longer, and it fails open (any failure leaves
  1277. # the stripped text as is, and the checks below decide as before).
  1278. if [[ -f "$BATCH_DIR/jd-rescue.mjs" && -s "$jd_raw_file" ]]; then
  1279. _rs_rc=0
  1280. _rs_line=$(node "$BATCH_DIR/jd-rescue.mjs" "$url" "$jd_raw_file" "$jd_file" "$jd_prefetch_words" "${current_url:-$url}" 2>/dev/null) || _rs_rc=$?
  1281. if [[ "$_rs_rc" -eq 3 ]]; then
  1282. # The ATS says the posting is closed: terminal, even though the page text may look like a JD.
  1283. jd_rescue_closed_note="posting closed (jd-rescue): $(printf '%s' "${_rs_line#*$'\t'}" | tr -c '[:print:]' ' ' | sed 's/^closed://; s/ */ /g')"
  1284. : > "$jd_file"
  1285. jd_prefetch_words=0
  1286. elif [[ "$_rs_rc" -eq 4 ]]; then
  1287. # The download landed on a different page than the offer URL: its text is not this posting's.
  1288. : > "$jd_file"
  1289. jd_prefetch_words=0
  1290. elif [[ "$_rs_rc" -eq 0 ]]; then
  1291. IFS=$'\t' read -r _rs_words _rs_method <<< "$_rs_line"
  1292. _rs_words="${_rs_words//[^0-9]/}"
  1293. _rs_words="${_rs_words:-0}"
  1294. if [[ "$_rs_words" -ge "$prefetch_min_words" ]]; then
  1295. echo " โ„น๏ธ JD rescue: recovered ${_rs_words} words via ${_rs_method} (visible text had ${jd_prefetch_words})"
  1296. jd_prefetch_words="$_rs_words"
  1297. fi
  1298. fi
  1299. fi
  1300. rm -f "$jd_raw_file"
  1301. if [[ "$jd_prefetch_words" -lt "$prefetch_min_words" ]]; then
  1302. : > "$jd_file"
  1303. echo " โ„น๏ธ JD prefetch: thin content (${jd_prefetch_words} words) โ€” worker will WebFetch"
  1304. else
  1305. echo " โ„น๏ธ JD prefetch: ${jd_prefetch_words} words written to JD file"
  1306. fi
  1307. fi
  1308. # ATS API fallback (#4246). When the static fetch above left $jd_file empty
  1309. # (thin JS shell, curl failure, curl absent), ask the ATS's own public API
  1310. # through fetch-jd.mjs before the worker falls back to WebFetch, which hits
  1311. # the same unrendered shell on Ashby and embedded Greenhouse boards.
  1312. # The call lives here and not in the worker prompt on purpose: the offer URL
  1313. # is untrusted, and here it travels as one argv element ("$url"), never as
  1314. # text inside a shell command the worker composes. fetch-jd.mjs only talks to
  1315. # the known ATS API hosts it resolves from the URL, so a private or unknown
  1316. # host is a miss (exit 1), not a request.
  1317. local jd_closed_note="$jd_rescue_closed_note"
  1318. local jd_api_words=0
  1319. if [[ ! -s "$jd_file" && -z "$jd_closed_note" && -f "$PROJECT_DIR/fetch-jd.mjs" ]]; then
  1320. local jd_api_tmp jd_api_rc=0
  1321. jd_api_tmp="$(mktemp "${TMPDIR:-/tmp}/batch-jd-api.XXXXXX")"
  1322. node "$PROJECT_DIR/fetch-jd.mjs" "$url" > "$jd_api_tmp" 2>/dev/null || jd_api_rc=$?
  1323. if [[ "$jd_api_rc" -eq 3 ]]; then
  1324. # Exit 3: the posting's own data says it is closed (validThrough in the past,
  1325. # closure text on the page, an ATS delisting flag). Do not fall back to
  1326. # WebFetch, which could read the JD from the page and score a closed role.
  1327. # The evidence is page text, so keep it to one short printable line.
  1328. jd_closed_note="posting closed (fetch-jd): $(head -c 160 "$jd_api_tmp" | tr -c '[:print:]' ' ' | sed 's/^CLOSED_POSTING //; s/ */ /g')"
  1329. elif [[ "$jd_api_rc" -eq 0 ]]; then
  1330. jd_api_words=$(wc -w < "$jd_api_tmp" 2>/dev/null) || jd_api_words=0
  1331. jd_api_words="${jd_api_words//[^0-9]/}"
  1332. jd_api_words="${jd_api_words:-0}"
  1333. if grep -q 'Job description truncated here: the posting is longer than' "$jd_api_tmp" 2>/dev/null; then
  1334. # fetch-jd.mjs marked this text as cut short. Scoring and archiving it
  1335. # would treat a partial JD as the whole posting, so leave the JD file
  1336. # empty and let the existing thin-JD handling decide what happens next.
  1337. echo " โ„น๏ธ JD prefetch: fetch-jd.mjs returned a truncated JD โ€” not used as the full posting"
  1338. elif [[ "$jd_api_words" -ge "$prefetch_min_words" ]]; then
  1339. cat "$jd_api_tmp" > "$jd_file"
  1340. echo " โ„น๏ธ JD prefetch: ${jd_api_words} words from the ATS API (fetch-jd.mjs)"
  1341. fi
  1342. fi
  1343. rm -f "$jd_api_tmp"
  1344. fi
  1345. # --thin-action: no usable JD text after the static fetch and the ATS API
  1346. # fallback (a JS-rendered page, an empty shell, a failed fetch). The default,
  1347. # 'webfetch', changes nothing: the worker is told to WebFetch. Detection only
  1348. # here; the skip is recorded after the "Processing offer" line below.
  1349. local jd_thin_skip=0 jd_thin_note="" jd_thin_retryable=0
  1350. if [[ -n "$jd_closed_note" ]]; then
  1351. # A closed posting is terminal ('skipped', not retryable) whatever --thin-action says.
  1352. jd_thin_skip=1
  1353. jd_thin_note="$jd_closed_note"
  1354. elif [[ ! -s "$jd_file" && "$THIN_ACTION" != "webfetch" ]]; then
  1355. if [[ "$THIN_ACTION" == "browser-extract" && -f "$PROJECT_DIR/browser-extract.mjs" ]]; then
  1356. # Real-browser read: zero model tokens, ~5-15s of Chromium. Success is
  1357. # compact JSON on stdout, a failure is exit 1 with a JSON error on stderr.
  1358. # A near-empty or taken-down page is a failure there, so the word floor
  1359. # below only has to stop a login wall or an error page full of navigation.
  1360. local be_out be_err be_words=0 be_code=""
  1361. be_out="$(mktemp "${TMPDIR:-/tmp}/batch-jd-browser.XXXXXX")"
  1362. be_err="$(mktemp "${TMPDIR:-/tmp}/batch-jd-browser-err.XXXXXX")"
  1363. if node "$PROJECT_DIR/browser-extract.mjs" "$url" --mode jd > "$be_out" 2> "$be_err"; then
  1364. node -e '
  1365. let d = "";
  1366. process.stdin.on("data", (c) => d += c);
  1367. process.stdin.on("end", () => {
  1368. try { const o = JSON.parse(d); process.stdout.write(typeof o.text === "string" ? o.text : ""); } catch { /* empty */ }
  1369. });
  1370. ' < "$be_out" > "$jd_file" 2>/dev/null || : > "$jd_file"
  1371. be_words=$(wc -w < "$jd_file" 2>/dev/null) || be_words=0
  1372. be_words="${be_words//[^0-9]/}"; be_words="${be_words:-0}"
  1373. if [[ "$be_words" -ge 200 ]]; then
  1374. echo " ๐ŸŽญ Browser extract: recovered ${be_words} words"
  1375. jd_prefetch_words="$be_words"
  1376. else
  1377. : > "$jd_file"
  1378. echo " ๐ŸŽญ Browser extract: no readable JD (${be_words} words)"
  1379. fi
  1380. else
  1381. : > "$jd_file"
  1382. be_code=$(node -e '
  1383. let d = "";
  1384. process.stdin.on("data", (c) => d += c);
  1385. process.stdin.on("end", () => { try { process.stdout.write(String(JSON.parse(d).code || "")); } catch { /* none */ } });
  1386. ' < "$be_err" 2>/dev/null) || be_code=""
  1387. echo " ๐ŸŽญ Browser extract failed${be_code:+ ($be_code)}"
  1388. # Allowlist of transient failures worth another try. Everything else
  1389. # (unavailable, URL or validation errors, an unknown code) is a verdict
  1390. # about the page or the call and stays terminal.
  1391. case "$be_code" in
  1392. no_playwright)
  1393. jd_thin_retryable=1
  1394. jd_thin_note="browser-extract: Playwright is not installed โ€” run 'npx playwright install chromium' and retry"
  1395. ;;
  1396. empty_text|navigation_error)
  1397. jd_thin_retryable=1
  1398. jd_thin_note="browser-extract failed ($be_code) โ€” retryable"
  1399. ;;
  1400. esac
  1401. fi
  1402. rm -f "$be_out" "$be_err"
  1403. fi
  1404. if [[ ! -s "$jd_file" ]]; then
  1405. jd_thin_skip=1
  1406. # curl never got an HTTP response (DNS, refused, timeout, TLS): a transport
  1407. # problem, not a page that cannot be read. 'skipped' is terminal and
  1408. # --retry-failed would never pick it up again, so it stays retryable.
  1409. case "$curl_status" in
  1410. 6|7|28|35|52|55|56)
  1411. jd_thin_retryable=1
  1412. [[ -n "$jd_thin_note" ]] || jd_thin_note="JD prefetch failed to connect (curl exit $curl_status) โ€” retryable"
  1413. ;;
  1414. esac
  1415. [[ -n "$jd_thin_note" ]] || jd_thin_note="no usable JD text (--thin-action $THIN_ACTION) โ€” needs-playwright"
  1416. fi
  1417. fi
  1418. echo "--- Processing offer #$id${JOB_POS:+ ($JOB_POS)}: $url (report $report_num, attempt $((retries + 1)))"
  1419. if [[ "$jd_thin_skip" == "1" ]]; then
  1420. local thin_state="skipped" thin_now
  1421. local thin_retries="$retries"
  1422. if [[ "$jd_thin_retryable" == "1" ]]; then
  1423. thin_state="failed"
  1424. # Spend one retry so --retry-failed stops at MAX_RETRIES instead of looping.
  1425. (( thin_retries < MAX_RETRIES )) && thin_retries=$((retries + 1))
  1426. fi
  1427. thin_now=$(date -u +%Y-%m-%dT%H:%M:%SZ)
  1428. update_state_retrying "$id" "$url" "$thin_state" "$started_at" "$thin_now" "-" "-" "$jd_thin_note" "$thin_retries" || true
  1429. release_report_num "$report_num"
  1430. rm -f "$jd_file"
  1431. echo " ๐Ÿšซ $jd_thin_note โ€” no worker spawned"
  1432. return 0
  1433. fi
  1434. # Build the prompt with placeholders replaced
  1435. local prompt
  1436. if [[ "${MODE:-full}" == "match" ]]; then
  1437. prompt="Process this job offer in single-pass CV match mode. Extract CV facts, score the match against the JD requirements in a single pass, output the report and tracker addition, and output final JSON. Do not generate PDF; write โŒ in the tracker PDF column and set \"pdf\": null in the final JSON."
  1438. elif [[ "$SKIP_PDF" == "true" ]]; then
  1439. prompt="Process this job offer. Run the pipeline: A-G evaluation + report .md + tracker line. Do not generate PDF; write โŒ in the tracker PDF column and set \"pdf\": null in the final JSON."
  1440. echo " โญ๏ธ --skip-pdf set โ€” skipping PDF generation for #$id ($url)"
  1441. else
  1442. prompt="Process this job offer. Run the full pipeline: A-G evaluation + report .md + optional PDF + tracker line."
  1443. fi
  1444. prompt="$prompt URL: $url"
  1445. prompt="$prompt JD file: $jd_file"
  1446. prompt="$prompt Report number: $report_num"
  1447. prompt="$prompt Date: $date"
  1448. prompt="$prompt Batch ID: $id"
  1449. local log_file="$LOGS_DIR/${report_num}-${id}.log"
  1450. # Prepare system prompt with placeholders resolved
  1451. local resolved_prompt="$BATCH_DIR/.resolved-prompt-${id}.md"
  1452. # Resolve placeholders to STABLE labels, not per-offer values: the concrete
  1453. # URL / JD file / report number / date / batch ID travel in the per-job user
  1454. # prompt built above. Keeping the system prompt byte-identical across offers
  1455. # lets prompt caching reuse the whole prefix instead of breaking at the first
  1456. # substituted value (~4 KB into a ~41 KB prompt). This also means the URL
  1457. # never enters a sed replacement, so it needs no delimiter/metacharacter
  1458. # escaping (no more esc_url/esc_jd_file/&-in-query-string concerns).
  1459. sed \
  1460. -e "s|{{URL}}|<URL from the job message>|g" \
  1461. -e "s|{{JD_FILE}}|<JD file from the job message>|g" \
  1462. -e "s|{{REPORT_NUM}}|<report number from the job message>|g" \
  1463. -e "s|{{DATE}}|<date from the job message>|g" \
  1464. -e "s|{{ID}}|<batch ID from the job message>|g" \
  1465. "$PROMPT_FILE" > "$resolved_prompt"
  1466. # The prompt names user-layer files by repo-relative path, and the worker
  1467. # resolves those against its working directory, the checkout. With the data
  1468. # root elsewhere that is the wrong tree for everything it reads about the
  1469. # candidate and everything it writes, so say where the user layer is. The
  1470. # text depends only on the data root, so it is the same for every offer in a
  1471. # run, and the default layout adds nothing.
  1472. if [[ "$DATA_ROOT" != "$PROJECT_DIR" ]]; then
  1473. {
  1474. printf '\n\n---\n\n'
  1475. printf '## Runtime paths: data root outside the checkout\n\n'
  1476. if [[ "${MODE:-full}" == "match" ]]; then
  1477. # Match mode names no profile file (#5297), so do not point at one.
  1478. printf 'This install keeps the user layer in a separate data root. Wherever this prompt names one of the paths below, resolve it under the data root, not the working directory:\n\n'
  1479. else
  1480. printf 'This install keeps the user layer in a separate data root. The profile for this run is `%s`. Where this prompt names `config/profile.yml`, read that file instead. Wherever this prompt names one of the paths below, resolve it under the data root, not the working directory:\n\n' "$PROFILE_FILE"
  1481. fi
  1482. printf 'Data root: %s\n\n' "$DATA_ROOT"
  1483. printf '`cv.md`, `article-digest.md`, `modes/_profile.md`, `modes/_custom.md`, `reports/`, `output/`, `data/`, `interview-prep/`, `batch/tracker-additions/`\n\n'
  1484. printf 'Write the report under `reports/` in the data root and the tracker TSV under `batch/tracker-additions/` in the data root, and give scripts absolute data-root paths for these files. Markdown links written inside the report and the tracker TSV keep the relative form this prompt shows (`reports/...`). Every other path (`templates/`, the `.mjs` scripts, the other `modes/` files) stays relative to the working directory.\n'
  1485. } >> "$resolved_prompt"
  1486. fi
  1487. # The .md prompts name config/profile.yml. When CAREER_OPS_PROFILE names a
  1488. # different file, the worker would read the wrong profile for everything
  1489. # except the personalization injected below, so name the file this run uses.
  1490. # With the data root outside the checkout the runtime block above already
  1491. # does that, so this only fills the default layout.
  1492. # The override is checked first, the way hasProfileOverride() does it, and
  1493. # the path comparison only after it. With no override the two paths are the
  1494. # same file, but not always the same string: through a junction or symlink
  1495. # getProfilePath() returns the real path while DATA_ROOT keeps the one pwd
  1496. # gave, and that alone used to add this block.
  1497. local profile_override="${CAREER_OPS_PROFILE:-}"
  1498. if [[ "${MODE:-full}" != "match" && -n "${profile_override//[[:space:]]/}" && "$PROFILE_RESOLVED" == "true" && "$PROFILE_FILE" != "$DATA_ROOT/config/profile.yml" && "$DATA_ROOT" == "$PROJECT_DIR" ]]; then
  1499. {
  1500. printf '\n\n---\n\n'
  1501. printf '## Runtime profile\n\n'
  1502. printf 'The profile for this run is `%s`. Where this prompt names `config/profile.yml`, read that file instead.\n' "$PROFILE_FILE"
  1503. } >> "$resolved_prompt"
  1504. fi
  1505. # Inject user-layer personalization into the temporary worker prompt.
  1506. # The resolved prompt is gitignored runtime state, so user profile data stays
  1507. # out of the system layer while batch scoring matches interactive scoring.
  1508. # Match mode gets none of it, only the output language (#5297).
  1509. if [[ "${MODE:-full}" == "match" ]]; then
  1510. {
  1511. printf '\n\n---\n\n'
  1512. printf '## Runtime output language\n\n'
  1513. printf 'language.output for this run: `%s`\n' "${MATCH_OUTPUT_LANGUAGE:-en}"
  1514. } >> "$resolved_prompt"
  1515. else
  1516. for context_file in "$DATA_ROOT/modes/_profile.md" "$PROFILE_FILE" "$DATA_ROOT/modes/_custom.md"; do
  1517. if [[ -f "$context_file" ]]; then
  1518. {
  1519. printf '\n\n---\n\n'
  1520. printf '## Runtime personalization: %s\n\n' "${context_file#"$DATA_ROOT/"}"
  1521. sed 's/^/ /' "$context_file"
  1522. printf '\n'
  1523. } >> "$resolved_prompt"
  1524. fi
  1525. done
  1526. fi
  1527. # The run-wide evidence freshness result (Step 1.7). The same text for every
  1528. # offer, so it keeps the resolved prompt byte-identical across the run.
  1529. if [[ -n "$EVIDENCE_FRESHNESS_JSON" ]]; then
  1530. {
  1531. printf '\n\n---\n\n'
  1532. printf '## Runtime: GitHub evidence freshness\n\n'
  1533. printf 'The runner ran `node evidence-freshness.mjs` once for this batch run. Its JSON output:\n\n'
  1534. printf '%s\n' "$EVIDENCE_FRESHNESS_JSON" | sed 's/^/ /'
  1535. } >> "$resolved_prompt"
  1536. fi
  1537. # Build the launch command for the configured CLI.
  1538. # For claude the model is resolved once per run from spend_tier unless --model
  1539. # was passed; other CLIs take --model verbatim. Building claude's command in an
  1540. # array keeps quoting safe regardless.
  1541. # --strict-mcp-config (with no --mcp-config) starts workers with no MCP
  1542. # servers: they only evaluate offers and need none. Without it each parallel
  1543. # worker inherits the parent session's MCP (e.g. Playwright) and they deadlock
  1544. # fighting over the single shared browser when --parallel > 1 (issue #506).
  1545. local -a claude_args=(-p --dangerously-skip-permissions --strict-mcp-config)
  1546. if [[ -n "$RESOLVED_MODEL" ]]; then
  1547. claude_args+=(--model "$RESOLVED_MODEL")
  1548. fi
  1549. claude_args+=(--append-system-prompt-file "$resolved_prompt" "$prompt")
  1550. # Non-claude CLIs lack --append-system-prompt-file, so concatenate the
  1551. # resolved system prompt and the per-job prompt into a single argument.
  1552. # model_args is expanded with the ${arr[@]+"${arr[@]}"} idiom below: under
  1553. # `set -u`, bash 3.2 (macOS /bin/bash) treats an empty array expansion as an
  1554. # unbound variable and would abort every worker launched without --model.
  1555. local full_prompt=""
  1556. local -a model_args=()
  1557. if [[ "$CLI" != "claude" ]]; then
  1558. full_prompt="$(cat "$resolved_prompt")"$'\n\n'"$prompt"
  1559. [[ -n "$MODEL" ]] && model_args=(--model "$MODEL")
  1560. fi
  1561. # Non-claude CLIs run a single attempt (explicit break after dispatch); the
  1562. # rate-limit/session retry loop only applies to claude.
  1563. local exit_code=0
  1564. local terminal_failure_recorded=false
  1565. local shim_retries=0
  1566. local max_shim_retries=4
  1567. local match_output_retries=0
  1568. local max_match_output_retries=1
  1569. while true; do
  1570. exit_code=0
  1571. case "$CLI" in
  1572. claude)
  1573. run_worker claude "${claude_args[@]}" > "$log_file" 2>&1 || exit_code=$?
  1574. ;;
  1575. opencode)
  1576. if command -v opencode &>/dev/null; then
  1577. run_worker opencode run ${model_args[@]+"${model_args[@]}"} "$full_prompt" > "$log_file" 2>&1 || exit_code=$?
  1578. else
  1579. run_worker ollama launch opencode ${model_args[@]+"${model_args[@]}"} -y -- run "$full_prompt" > "$log_file" 2>&1 || exit_code=$?
  1580. fi
  1581. ;;
  1582. gemini)
  1583. run_worker gemini ${model_args[@]+"${model_args[@]}"} -p "$full_prompt" > "$log_file" 2>&1 || exit_code=$?
  1584. ;;
  1585. qwen)
  1586. run_worker qwen ${model_args[@]+"${model_args[@]}"} -p "$full_prompt" > "$log_file" 2>&1 || exit_code=$?
  1587. ;;
  1588. grok)
  1589. run_worker grok --always-approve --output-format plain ${model_args[@]+"${model_args[@]}"} -p "$full_prompt" > "$log_file" 2>&1 || exit_code=$?
  1590. ;;
  1591. esac
  1592. # In match mode, check if worker result JSON is malformed or missing required
  1593. # fields (parsed_status or parsed_score). If so, retry once before finalizing
  1594. # so a transient formatting glitch does not permanently fail or bypass gates.
  1595. if [[ "${MODE:-full}" == "match" && $exit_code -eq 0 ]]; then
  1596. local worker_check_json
  1597. worker_check_json=$(awk '
  1598. /^```json[[:space:]]*$/ { in_block=1; block=""; next }
  1599. in_block && /^```[[:space:]]*$/ { in_block=0; last=block; next }
  1600. in_block { block = block $0 "\n" }
  1601. END { printf "%s", last }
  1602. ' "$log_file" 2>/dev/null || true)
  1603. local check_status="" check_score=""
  1604. if [[ -n "$worker_check_json" ]]; then
  1605. local check_parsed
  1606. check_parsed=$(printf '%s' "$worker_check_json" | node -e '
  1607. let data = "";
  1608. process.stdin.on("data", d => data += d);
  1609. process.stdin.on("end", () => {
  1610. try {
  1611. const obj = JSON.parse(data);
  1612. const status = typeof obj.status === "string" ? obj.status : "";
  1613. const score = typeof obj.score === "number" ? String(obj.score) : "";
  1614. process.stdout.write(status + "\x1f" + score);
  1615. } catch {
  1616. process.stdout.write("");
  1617. }
  1618. });
  1619. ' 2>/dev/null || true)
  1620. if [[ -n "$check_parsed" ]]; then
  1621. IFS=$'\x1f' read -r check_status check_score <<< "$check_parsed"
  1622. fi
  1623. fi
  1624. if [[ -z "$check_status" || ( "$check_status" != "needs_confirmation" && "$check_status" != "skipped" && "$check_status" != "failed" && -z "$check_score" ) ]]; then
  1625. if (( match_output_retries < max_match_output_retries && retries < MAX_RETRIES )); then
  1626. match_output_retries=$((match_output_retries + 1))
  1627. retries=$((retries + 1))
  1628. # Move the malformed attempt's artifacts aside first. Left in place,
  1629. # a retry with another slug leaves two reports on one number, and a
  1630. # retry that writes no report would pass the report-file check on the
  1631. # stale one. Same quarantine as the confirmation-hold path below.
  1632. local retry_quarantine="$LOGS_DIR/quarantine" retry_artifact
  1633. shopt -s nullglob
  1634. for retry_artifact in "$REPORTS_DIR/$report_num-"*.md "$TRACKER_DIR/$id.tsv"; do
  1635. [[ "$retry_artifact" == *-RESERVED.md ]] && continue
  1636. mkdir -p "$retry_quarantine"
  1637. mv "$retry_artifact" "$retry_quarantine/$id-match-retry-${retry_artifact##*/}"
  1638. done
  1639. shopt -u nullglob
  1640. echo " โš ๏ธ Malformed or missing match mode JSON summary (attempt $retries/$MAX_RETRIES); retrying once..."
  1641. sleep 2
  1642. continue
  1643. fi
  1644. fi
  1645. fi
  1646. # Non-claude CLIs run a single attempt: the session-limit and rate-limit
  1647. # detection below greps generic phrases (429, quota, session limit) that
  1648. # another CLI's log could match by coincidence and pause or retry the batch.
  1649. if [[ "$CLI" != "claude" ]]; then
  1650. break
  1651. fi
  1652. if [[ $exit_code -eq 0 ]]; then
  1653. break
  1654. fi
  1655. # Check for Claude Code npm shim swap (exit code 127 + command not found)
  1656. if [[ $exit_code -eq 127 ]] && grep -qE "(claude: command not found|claude:.*not found|cannot find.*claude)" "$log_file" && (( shim_retries < max_shim_retries )); then
  1657. shim_retries=$((shim_retries + 1))
  1658. echo " โณ Claude command not found (shim swap detected). Retrying in 30s (attempt $shim_retries/$max_shim_retries)..."
  1659. sleep 30
  1660. continue
  1661. fi
  1662. if is_session_limit_log "$log_file"; then
  1663. mark_paused_rate_limit "$id" "$url" "$started_at" "$report_num" "$retries" "$log_file"
  1664. echo " โธ๏ธ Session/rate limit reached; pausing batch without consuming retry budget."
  1665. terminal_failure_recorded=true
  1666. break
  1667. fi
  1668. if is_rate_limit_log "$log_file" && (( retries < MAX_RETRIES )); then
  1669. if (( RATE_LIMIT_SLEEP <= 0 )); then
  1670. mark_paused_rate_limit "$id" "$url" "$started_at" "$report_num" "$retries" "$log_file"
  1671. echo " โธ๏ธ Rate limited and --rate-limit-sleep is 0; pausing batch without consuming retry budget."
  1672. terminal_failure_recorded=true
  1673. break
  1674. fi
  1675. local retry_delay
  1676. retry_delay=$(rate_limit_delay "$log_file" "$retries")
  1677. if [[ "$retry_delay" == pause ]]; then
  1678. mark_paused_rate_limit "$id" "$url" "$started_at" "$report_num" "$retries" "$log_file"
  1679. echo " โธ๏ธ Retry-After exceeds --rate-limit-sleep; pausing batch without consuming retry budget."
  1680. terminal_failure_recorded=true
  1681. break
  1682. fi
  1683. : > "$THROTTLE_FILE"
  1684. retries=$((retries + 1))
  1685. local retry_completed_at
  1686. retry_completed_at=$(date -u +%Y-%m-%dT%H:%M:%SZ)
  1687. update_state_retrying "$id" "$url" "rate_limited" "$started_at" "$retry_completed_at" "$report_num" "-" "rate-limit; retrying after ${retry_delay}s" "$retries" || true
  1688. echo " โณ Rate limited (attempt $retries/$MAX_RETRIES). Waiting ${retry_delay}s before retry..."
  1689. sleep "$retry_delay"
  1690. continue
  1691. fi
  1692. break
  1693. done
  1694. # Cleanup resolved prompt and pre-fetched JD file
  1695. rm -f "$resolved_prompt" "$jd_file"
  1696. local completed_at
  1697. completed_at=$(date -u +%Y-%m-%dT%H:%M:%SZ)
  1698. if [[ $exit_code -eq 0 ]]; then
  1699. # A worker can exit 0 (no crash) but still self-report failure inside its
  1700. # own JSON summary โ€” e.g. it correctly declines to fabricate an evaluation
  1701. # when the JD couldn't be extracted (Data Contract: never fabricate).
  1702. # Without this check such offers were silently marked "completed" with no
  1703. # report file on disk and score "-" (found 2026-07-29, offer id 6 / report
  1704. # 019 โ€” Deepgram JD unextractable in headless mode). Only the downstream
  1705. # reconcile-pipeline.mjs safety net (which leaves an entry in Pending when
  1706. # its report file is missing) prevented the offer from being lost.
  1707. # Extract only the LAST ```json fenced block in the log -- that's the
  1708. # worker's one authoritative final result (batch-prompt.md Step 6), not
  1709. # arbitrary text anywhere else in stdout/stderr -- and parse it as real
  1710. # JSON so an unrelated line merely containing the substring
  1711. # `"status": "failed"` can never falsely flip a successful run.
  1712. local worker_result_json
  1713. worker_result_json=$(awk '
  1714. /^```json[[:space:]]*$/ { in_block=1; block=""; next }
  1715. in_block && /^```[[:space:]]*$/ { in_block=0; last=block; next }
  1716. in_block { block = block $0 "\n" }
  1717. END { printf "%s", last }
  1718. ' "$log_file" 2>/dev/null || true)
  1719. # Parse status, error, AND score from the same authoritative JSON object
  1720. # in one pass -- score extraction used to be a separate sed regex over
  1721. # the whole log (`.*"score":...`), which grabbed the first match
  1722. # anywhere in the log rather than the one from this final result object,
  1723. # producing a spurious score "-" whenever an earlier line in the log
  1724. # (reasoning text, an intermediate example, Block D's "Comp score: 4/5"
  1725. # mention, etc.) matched first. Reading it from the same parsed object
  1726. # as status/error fixes both by construction -- there's only one place
  1727. # left to look.
  1728. local worker_failed_match="" worker_error_match="" score="-"
  1729. local parsed_status="" parsed_error="" parsed_score=""
  1730. if [[ -n "$worker_result_json" ]]; then
  1731. local parsed
  1732. parsed=$(printf '%s' "$worker_result_json" | node -e '
  1733. let data = "";
  1734. process.stdin.on("data", d => data += d);
  1735. process.stdin.on("end", () => {
  1736. try {
  1737. const obj = JSON.parse(data);
  1738. const status = typeof obj.status === "string" ? obj.status : "";
  1739. const message = status === "needs_confirmation" ? obj.question : obj.error;
  1740. const error = typeof message === "string" ? message.replace(/[\x00-\x1f\x7f]/g, " ") : "";
  1741. const score = typeof obj.score === "number" ? String(obj.score) : "";
  1742. process.stdout.write(status + "\x1f" + error + "\x1f" + score);
  1743. } catch {
  1744. process.stdout.write("");
  1745. }
  1746. });
  1747. ' 2>/dev/null || true)
  1748. if [[ -n "$parsed" ]]; then
  1749. # \x1f (US), not \t: tab is IFS *whitespace*, so bash collapses runs of it
  1750. # and strips leading/trailing ones. On the common path -- a worker that
  1751. # succeeded, so `error` is empty -- the two tabs around that empty field
  1752. # collapse into one, `score` slides into parsed_error, and parsed_score
  1753. # comes back empty. The `elif` below then never fires and every
  1754. # successful offer records score "-". A non-whitespace delimiter gets
  1755. # one-field-per-unit splitting with empty fields preserved.
  1756. IFS=$'\x1f' read -r parsed_status parsed_error parsed_score <<< "$parsed"
  1757. if [[ "$parsed_status" == "failed" ]]; then
  1758. worker_failed_match="failed"
  1759. worker_error_match="$parsed_error"
  1760. elif [[ -n "$parsed_score" ]]; then
  1761. score="$parsed_score"
  1762. fi
  1763. fi
  1764. fi
  1765. # In match mode, enforce valid JSON output and required fields. If malformed
  1766. # or missing parsed_status or parsed_score (and not a human-confirmation hold
  1767. # or dealbreaker skip), treat as a failed worker attempt so it cannot be
  1768. # recorded as completed with score "-" or bypass --min-score.
  1769. if [[ "${MODE:-full}" == "match" && -z "$worker_failed_match" ]]; then
  1770. if [[ -z "$worker_result_json" || -z "$parsed" || -z "$parsed_status" ]]; then
  1771. worker_failed_match="failed"
  1772. worker_error_match="malformed or missing match mode JSON summary (parsed_status missing)"
  1773. elif [[ "$parsed_status" != "needs_confirmation" && "$parsed_status" != "skipped" ]] && { [[ -z "$parsed_score" ]] || ! is_decimal_number "$score"; }; then
  1774. worker_failed_match="failed"
  1775. worker_error_match="missing or invalid match mode score: $score"
  1776. fi
  1777. fi
  1778. # A human gate is not a failure or a successful evaluation (#4359).
  1779. # Keep it out of retries and release the unused report reservation.
  1780. if [[ "$parsed_status" == "needs_confirmation" ]]; then
  1781. local question="${parsed_error:-Which agency did this posting come through?}"
  1782. # A malformed/misattributed handoff stays held, never becomes retryable
  1783. # and never supplies an agency answer for a different posting.
  1784. if ! node -e '
  1785. const result = JSON.parse(process.argv[1]);
  1786. process.exit(result.reason === "agency_confirmation" &&
  1787. result.url === process.argv[2] && String(result.id) === process.argv[3] ? 0 : 1);
  1788. ' "$worker_result_json" "$url" "$id"; then
  1789. question="Invalid confirmation handoff identity/reason; parent must inspect the job log and ask for this URL"
  1790. fi
  1791. local tracker_artifact="$TRACKER_DIR/$id.tsv"
  1792. local -a report_artifacts=()
  1793. if [[ -n "$report_num" && "$report_num" != "-" ]]; then
  1794. shopt -s nullglob
  1795. local candidate
  1796. # The reservation sentinel ({num}-RESERVED.md) is the parent's own file,
  1797. # not a worker artifact (same exclusion as the report-file check below).
  1798. for candidate in "$REPORTS_DIR/$report_num-"*.md; do
  1799. [[ "$candidate" == *-RESERVED.md ]] || report_artifacts+=("$candidate")
  1800. done
  1801. shopt -u nullglob
  1802. fi
  1803. if [[ -f "$tracker_artifact" || ${#report_artifacts[@]} -gt 0 ]]; then
  1804. local quarantine_dir="$LOGS_DIR/quarantine"
  1805. mkdir -p "$quarantine_dir"
  1806. if [[ -f "$tracker_artifact" ]]; then
  1807. mv "$tracker_artifact" "$quarantine_dir/$id-tracker.tsv"
  1808. fi
  1809. local report_artifact
  1810. for report_artifact in ${report_artifacts[@]+"${report_artifacts[@]}"}; do
  1811. mv "$report_artifact" "$quarantine_dir/$id-${report_artifact##*/}"
  1812. done
  1813. question="Worker violated the confirmation hold by writing artifacts; inspect the quarantined tracker/report before answering"
  1814. update_state_retrying "$id" "$url" "needs_confirmation" "$started_at" "$completed_at" "-" "-" "$question" "$retries" || true
  1815. echo " ERROR: confirmation hold produced artifacts; reservation kept and tracker merge skipped." >&2
  1816. return 2
  1817. fi
  1818. local hold_rc=0
  1819. update_state_retrying "$id" "$url" "needs_confirmation" "$started_at" "$completed_at" "-" "-" "$question" "$retries" || hold_rc=$?
  1820. release_report_num "$report_num"
  1821. echo " Needs confirmation: $url โ€” $question (resume in the parent session after an explicit answer)"
  1822. if (( hold_rc == 2 )); then
  1823. echo " ERROR: confirmation hold could not be persisted; stop and repair state before rerunning this posting." >&2
  1824. return 2
  1825. fi
  1826. return 0
  1827. fi
  1828. # A dealbreaker hit is a finished skip, not a hold (batch-prompt.md Step
  1829. # 1.6). The worker writes one SKIP row quoting the rule and the JD line and
  1830. # no report. Accept it only when the handoff names this posting and the row
  1831. # proves it; anything else is quarantined and retried like a failure.
  1832. if [[ "$parsed_status" == "skipped" ]]; then
  1833. local skip_tracker="$TRACKER_DIR/$id.tsv" skip_rule=""
  1834. local -a skip_reports=()
  1835. if [[ -n "$report_num" && "$report_num" != "-" ]]; then
  1836. shopt -s nullglob
  1837. local skip_candidate
  1838. for skip_candidate in "$REPORTS_DIR/$report_num-"*.md; do
  1839. [[ "$skip_candidate" == *-RESERVED.md ]] || skip_reports+=("$skip_candidate")
  1840. done
  1841. shopt -u nullglob
  1842. fi
  1843. if (( ${#skip_reports[@]} == 0 )); then
  1844. skip_rule=$(node -e '
  1845. const fs = require("fs");
  1846. const [json, url, id, tsv, reserved] = process.argv.slice(1);
  1847. const flat = (s) => String(s).replace(/[\x00-\x1f\x7f]+/g, " ").replace(/ +/g, " ").trim();
  1848. const result = JSON.parse(json);
  1849. if (result.reason !== "dealbreaker" || result.url !== url || String(result.id) !== id) process.exit(1);
  1850. if (typeof result.rule !== "string" || !flat(result.rule)) process.exit(1);
  1851. const lines = fs.readFileSync(tsv, "utf8").replace(/\r\n/g, "\n").split("\n").filter(Boolean);
  1852. if (lines.length !== 2) process.exit(1);
  1853. const head = lines[0].split("\t");
  1854. const row = lines[1].split("\t");
  1855. const field = (name) => row[head.indexOf(name)] ?? "";
  1856. if (field("status") !== "SKIP" || field("url") !== url || !field("notes").startsWith("Dealbreaker: \"")) process.exit(1);
  1857. // merge-tracker keeps the TSV num as the row id, so it must be the
  1858. // number this runner reserved (zero padding aside).
  1859. if (!/^\d+$/.test(field("num")) || Number(field("num")) !== Number(reserved)) process.exit(1);
  1860. process.stdout.write(flat(result.rule));
  1861. ' "$worker_result_json" "$url" "$id" "$skip_tracker" "$report_num" 2>/dev/null) || skip_rule=""
  1862. fi
  1863. if [[ -n "$skip_rule" ]]; then
  1864. # Keep the reservation. The SKIP row uses this number and no report
  1865. # file holds it, so releasing now would hand it to the next job before
  1866. # the merge. merge_tracker releases it once the row is in the tracker.
  1867. # rc 2: nothing was recorded and the job runs again as pending, so it
  1868. # is not a discard yet.
  1869. local skip_state_rc=0
  1870. update_state_retrying "$id" "$url" "skipped" "$started_at" "$completed_at" "$report_num" "-" "dealbreaker: \"$skip_rule\"" "$retries" || skip_state_rc=$?
  1871. if (( skip_state_rc != 2 )); then
  1872. log_discard "$id" "$url" "dealbreaker: \"$skip_rule\""
  1873. fi
  1874. echo " โญ๏ธ Skipped (dealbreaker): $url โ€” \"$skip_rule\""
  1875. return 0
  1876. fi
  1877. local skip_quarantine="$LOGS_DIR/quarantine"
  1878. mkdir -p "$skip_quarantine"
  1879. [[ -f "$skip_tracker" ]] && mv "$skip_tracker" "$skip_quarantine/$id-tracker.tsv"
  1880. local skip_report
  1881. for skip_report in ${skip_reports[@]+"${skip_reports[@]}"}; do
  1882. mv "$skip_report" "$skip_quarantine/$id-${skip_report##*/}"
  1883. done
  1884. if (( retries < MAX_RETRIES )); then
  1885. retries=$((retries + 1))
  1886. fi
  1887. update_state_retrying "$id" "$url" "failed" "$started_at" "$completed_at" "$report_num" "-" "dealbreaker skip without a valid SKIP tracker row for this posting, or with a report; artifacts quarantined" "$retries" || true
  1888. release_report_num "$report_num"
  1889. echo " โŒ Failed (invalid dealbreaker skip, attempt $retries): artifacts moved to $skip_quarantine"
  1890. return 0
  1891. fi
  1892. if [[ -n "$worker_failed_match" ]]; then
  1893. [[ -z "$worker_error_match" ]] && worker_error_match="worker reported status:failed (exit code 0)"
  1894. if (( retries < MAX_RETRIES )); then
  1895. retries=$((retries + 1))
  1896. fi
  1897. update_state_retrying "$id" "$url" "failed" "$started_at" "$completed_at" "$report_num" "-" "$worker_error_match" "$retries" || true
  1898. release_report_num "$report_num"
  1899. echo " โŒ Failed (worker-reported, attempt $retries): $worker_error_match"
  1900. return 0
  1901. fi
  1902. # A worker can exit 0, self-report a non-"failed" status (or no parseable
  1903. # JSON at all), and STILL never actually write the report file it claims
  1904. # -- exit code and JSON status alone are not proof a report exists. Found
  1905. # 2026-07-30: offer id 6/report 049 was marked "completed" this way with
  1906. # no file on disk, silently freeing that number for a second, unrelated
  1907. # offer to claim (a real collision). Verify the file before trusting
  1908. # "completed" -- fail closed, not open.
  1909. # The reservation sentinel ({num}-RESERVED.md) must NOT count as the
  1910. # report: it is created before the worker runs, so an unfiltered glob
  1911. # matches it and the guard is defeated every single time (found 2026-09-22:
  1912. # 16 no-op workers recorded as "completed" with score "-").
  1913. if ! has_report_file "$report_num"; then
  1914. if (( retries < MAX_RETRIES )); then
  1915. retries=$((retries + 1))
  1916. fi
  1917. update_state_retrying "$id" "$url" "failed" "$started_at" "$completed_at" "$report_num" "-" "worker exited cleanly but wrote no report file for this report number" "$retries" || true
  1918. release_report_num "$report_num"
  1919. echo " โŒ Failed (no report file on disk, attempt $retries)"
  1920. return 0
  1921. fi
  1922. # Check min-score gate
  1923. if is_decimal_number "$score" && LC_ALL=C awk -v min="$MIN_SCORE" 'BEGIN{exit !(min > 0)}'; then
  1924. # LC_ALL=C: under a non-English locale awk parses "4.5" as 4, so the
  1925. # MIN_SCORE comparison would silently run on truncated integers.
  1926. if LC_ALL=C awk -v score="$score" -v min="$MIN_SCORE" 'BEGIN{exit !(score < min)}'; then
  1927. local score_state_rc=0
  1928. update_state_retrying "$id" "$url" "skipped" "$started_at" "$completed_at" "$report_num" "$score" "below-min-score" "$retries" || score_state_rc=$?
  1929. if (( score_state_rc != 2 )); then
  1930. log_discard "$id" "$url" "below-min-score"
  1931. fi
  1932. release_report_num "$report_num"
  1933. echo " โญ๏ธ Skipped (score: $score < min-score: $MIN_SCORE)"
  1934. return 0
  1935. fi
  1936. fi
  1937. # Defensive guard: in match mode, an offer must never complete without a valid numeric score.
  1938. if [[ "${MODE:-full}" == "match" ]] && ! is_decimal_number "$score"; then
  1939. if (( retries < MAX_RETRIES )); then
  1940. retries=$((retries + 1))
  1941. fi
  1942. update_state_retrying "$id" "$url" "failed" "$started_at" "$completed_at" "$report_num" "-" "missing or invalid numeric score in match mode: $score" "$retries" || true
  1943. release_report_num "$report_num"
  1944. echo " โŒ Failed (invalid match mode score: $score, attempt $retries)"
  1945. return 0
  1946. fi
  1947. update_state_retrying "$id" "$url" "completed" "$started_at" "$completed_at" "$report_num" "$score" "-" "$retries" || true
  1948. release_report_num "$report_num"
  1949. echo " โœ… Completed (score: $score, report: $report_num)"
  1950. elif [[ "$terminal_failure_recorded" == "false" ]]; then
  1951. if (( retries < MAX_RETRIES )); then
  1952. retries=$((retries + 1))
  1953. fi
  1954. local error_msg
  1955. error_msg=$(tail -5 "$log_file" 2>/dev/null | tr '\n' ' ' | cut -c1-200 || echo "Unknown error (exit code $exit_code)")
  1956. update_state_retrying "$id" "$url" "failed" "$started_at" "$completed_at" "$report_num" "-" "$error_msg" "$retries" || true
  1957. release_report_num "$report_num"
  1958. echo " โŒ Failed (attempt $retries, exit code $exit_code)"
  1959. fi
  1960. }
  1961. # GitHub evidence freshness (#5674): one check for the whole run, not one per
  1962. # worker. The check makes a `gh search prs` call and its answer is the same for
  1963. # every offer, so the runner makes it once and every worker reads the result
  1964. # from its prompt. It is opt-in: with evidence_freshness.enabled not true it
  1965. # reports status "disabled" without any network call, and workers get nothing.
  1966. check_evidence_freshness() {
  1967. [[ -f "$PROJECT_DIR/evidence-freshness.mjs" ]] || return 0
  1968. local out status
  1969. out=$(node "$PROJECT_DIR/evidence-freshness.mjs" 2>/dev/null) || return 0
  1970. status=$(printf '%s\n' "$out" | sed -n 's/^ "status": "\(.*\)",$/\1/p')
  1971. [[ -z "$status" || "$status" == "disabled" ]] && return 0
  1972. EVIDENCE_FRESHNESS_JSON="$out"
  1973. if printf '%s\n' "$out" | grep -q '^ "stale": true,$'; then
  1974. echo "GitHub evidence: stale (checked once for this run; see each report header)"
  1975. else
  1976. echo "GitHub evidence: $status (checked once for this run)"
  1977. fi
  1978. echo ""
  1979. }
  1980. # Merge tracker additions into applications.md
  1981. merge_tracker() {
  1982. echo ""
  1983. echo "=== Merging tracker additions ==="
  1984. node "$PROJECT_DIR/merge-tracker.mjs"
  1985. # A dealbreaker SKIP row holds its report number until it is merged. The
  1986. # tracker row now holds it for reserve-report-num.mjs, so drop the sentinel.
  1987. if [[ -f "$STATE_FILE" ]]; then
  1988. local mid mstatus mnum merror
  1989. while IFS=$'\t' read -r mid _ mstatus _ _ mnum _ merror _; do
  1990. if [[ "$mid" != "id" && "$mstatus" == "skipped" && "$merror" == dealbreaker:* ]]; then
  1991. release_report_num "$mnum"
  1992. fi
  1993. done < "$STATE_FILE"
  1994. fi
  1995. echo ""
  1996. echo "=== Reconciling pipeline.md ==="
  1997. node "$PROJECT_DIR/reconcile-pipeline.mjs" || echo "โš ๏ธ Pipeline reconcile had issues (see above)"
  1998. echo ""
  1999. echo "=== Verifying pipeline integrity ==="
  2000. node "$PROJECT_DIR/verify-pipeline.mjs" || echo "โš ๏ธ Verification found issues (see above)"
  2001. }
  2002. report_progress() {
  2003. echo "=== Progress: $1/$2 offers finished, $((SECONDS - BATCH_START))s elapsed ==="
  2004. }
  2005. # Print summary
  2006. print_summary() {
  2007. echo ""
  2008. echo "=== Batch Summary ==="
  2009. if [[ ! -f "$STATE_FILE" ]]; then
  2010. echo "No state file found."
  2011. return
  2012. fi
  2013. local total=0 completed=0 skipped=0 failed=0 pending=0 needs_confirmation=0
  2014. local score_sum=0 score_count=0
  2015. local -a dealbreaker_lines=()
  2016. while IFS=$'\t' read -r sid surl sstatus _ _ _ sscore serror _; do
  2017. [[ "$sid" == "id" ]] && continue
  2018. total=$((total + 1))
  2019. case "$sstatus" in
  2020. completed) completed=$((completed + 1))
  2021. if is_decimal_number "$sscore"; then
  2022. score_sum=$(LC_ALL=C awk -v sum="$score_sum" -v score="$sscore" 'BEGIN{print sum + score}' 2>/dev/null || echo "$score_sum")
  2023. score_count=$((score_count + 1))
  2024. fi
  2025. ;;
  2026. skipped) skipped=$((skipped + 1))
  2027. [[ "$serror" == dealbreaker:* ]] && dealbreaker_lines+=("#$sid $surl โ€” $serror")
  2028. ;;
  2029. failed) failed=$((failed + 1)) ;;
  2030. needs_confirmation) needs_confirmation=$((needs_confirmation + 1)) ;;
  2031. *) pending=$((pending + 1)) ;;
  2032. esac
  2033. done < "$STATE_FILE"
  2034. echo "Total: $total | Completed: $completed | Skipped: $skipped | Failed: $failed | Pending: $pending | Needs confirmation: $needs_confirmation"
  2035. if (( ${#dealbreaker_lines[@]} > 0 )); then
  2036. echo "Dealbreaker skips: ${#dealbreaker_lines[@]} (SKIP rows merged; to evaluate one anyway, rerun it in a session and say \"evaluate anyway\")"
  2037. local dealbreaker_line
  2038. for dealbreaker_line in "${dealbreaker_lines[@]}"; do
  2039. echo " $dealbreaker_line"
  2040. done
  2041. fi
  2042. if (( score_count > 0 )); then
  2043. local avg
  2044. # LC_ALL=C: under e.g. a German locale awk formats "%.1f" as "4,5"
  2045. # instead of "4.5", and a decimal comma breaks every downstream parser.
  2046. avg=$(LC_ALL=C awk -v sum="$score_sum" -v count="$score_count" 'BEGIN{printf "%.1f", sum / count}' 2>/dev/null || echo "N/A")
  2047. echo "Average score: $avg/5 ($score_count scored)"
  2048. fi
  2049. if [[ -f "$BATCH_DIR/aggregate-tokens.mjs" ]]; then
  2050. if ! node "$BATCH_DIR/aggregate-tokens.mjs"; then
  2051. echo "Warning: token aggregation failed." >&2
  2052. fi
  2053. fi
  2054. }
  2055. print_status_table() {
  2056. # $1 = "full" forces the per-job table whatever STATUS_FULL says (--watch).
  2057. local force_full="${1:-}"
  2058. if [[ ! -f "$STATE_FILE" ]]; then
  2059. echo "No state file found at $STATE_FILE"
  2060. return
  2061. fi
  2062. local total=0 completed=0 processing=0 failed=0 pending=0 skipped=0 rate_limited=0 paused_rate_limit=0 needs_confirmation=0
  2063. local score_sum=0 score_count=0
  2064. # Read first line to skip header
  2065. local header=true
  2066. while IFS=$'\t' read -r sid surl sstatus sstarted scompleted sreport sscore serror sretries || [[ -n "$sid" ]]; do
  2067. if [[ "$header" == "true" ]]; then
  2068. header=false
  2069. continue
  2070. fi
  2071. [[ -z "$sid" ]] && continue
  2072. sstatus="${sstatus%$'\r'}"
  2073. sscore="${sscore%$'\r'}"
  2074. serror="${serror%$'\r'}"
  2075. sreport="${sreport%$'\r'}"
  2076. total=$((total + 1))
  2077. case "$sstatus" in
  2078. completed)
  2079. completed=$((completed + 1))
  2080. if is_decimal_number "$sscore"; then
  2081. score_sum=$(LC_ALL=C awk -v sum="$score_sum" -v score="$sscore" 'BEGIN{print sum + score}' 2>/dev/null || echo "$score_sum")
  2082. score_count=$((score_count + 1))
  2083. fi
  2084. ;;
  2085. processing) processing=$((processing + 1)) ;;
  2086. needs_confirmation) needs_confirmation=$((needs_confirmation + 1)) ;;
  2087. failed) failed=$((failed + 1)) ;;
  2088. skipped) skipped=$((skipped + 1)) ;;
  2089. rate_limited) rate_limited=$((rate_limited + 1)) ;;
  2090. paused_rate_limit) paused_rate_limit=$((paused_rate_limit + 1)) ;;
  2091. *) pending=$((pending + 1)) ;;
  2092. esac
  2093. done < "$STATE_FILE"
  2094. echo "=== Batch Progress ==="
  2095. echo "Total: $total | Completed: $completed | Processing: $processing | Failed: $failed | Pending: $pending | Skipped: $skipped | Rate Limited: $rate_limited | Paused: $paused_rate_limit | Needs confirmation: $needs_confirmation"
  2096. if (( score_count > 0 )); then
  2097. local avg
  2098. # LC_ALL=C: under e.g. a German locale awk formats "%.1f" as "4,5"
  2099. # instead of "4.5", and a decimal comma breaks every downstream parser.
  2100. avg=$(LC_ALL=C awk -v sum="$score_sum" -v count="$score_count" 'BEGIN{printf "%.1f", sum / count}' 2>/dev/null || echo "N/A")
  2101. echo "Average score: $avg/5 ($score_count scored)"
  2102. fi
  2103. if [[ "$force_full" != "full" && "$STATUS_FULL" != "true" ]]; then
  2104. return
  2105. fi
  2106. echo ""
  2107. # Format the per-job table:
  2108. # Columns: ID, Status, Report, Score, Target (URL or explanation)
  2109. printf "%-4s | %-17s | %-6s | %-5s | %-40s\n" "ID" "Status" "Report" "Score" "URL / Detail"
  2110. printf "%-4s+%-19s+%-8s+%-7s+%-42s\n" "----" "-------------------" "--------" "-------" "------------------------------------------"
  2111. header=true
  2112. while IFS=$'\t' read -r sid surl sstatus sstarted scompleted sreport sscore serror sretries || [[ -n "$sid" ]]; do
  2113. if [[ "$header" == "true" ]]; then
  2114. header=false
  2115. continue
  2116. fi
  2117. [[ -z "$sid" ]] && continue
  2118. sstatus="${sstatus%$'\r'}"
  2119. sscore="${sscore%$'\r'}"
  2120. serror="${serror%$'\r'}"
  2121. sreport="${sreport%$'\r'}"
  2122. local target="$surl"
  2123. if [[ "$sstatus" == "needs_confirmation" ]]; then
  2124. target="$surl โ€” Needs confirmation: $serror"
  2125. elif [[ "$sstatus" == "failed" && -n "$serror" && "$serror" != "-" ]]; then
  2126. target="Error: $serror"
  2127. fi
  2128. # Trim target to fit nicely (e.g. 50 chars)
  2129. if [[ "$sstatus" != "needs_confirmation" ]] && (( ${#target} > 50 )); then
  2130. target="${target:0:47}..."
  2131. fi
  2132. printf "%-4s | %-17s | %-6s | %-5s | %-50s\n" "$sid" "$sstatus" "$sreport" "$sscore" "$target"
  2133. done < "$STATE_FILE"
  2134. }
  2135. watch_status() {
  2136. local active_pid=""
  2137. if [[ -f "$LOCK_FILE" ]]; then
  2138. active_pid=$(cat "$LOCK_FILE" 2>/dev/null || true)
  2139. fi
  2140. if [[ -n "$active_pid" ]] && kill -0 "$active_pid" 2>/dev/null; then
  2141. echo "Watching batch-runner (PID $active_pid)... Press Ctrl+C to stop."
  2142. while kill -0 "$active_pid" 2>/dev/null; do
  2143. clear || printf "\033[c"
  2144. echo "=== Watching Batch Progress (PID $active_pid) ==="
  2145. print_status_table full
  2146. sleep 2
  2147. done
  2148. echo ""
  2149. echo "=== Batch runner process (PID $active_pid) has finished ==="
  2150. else
  2151. echo "No active batch-runner detected."
  2152. fi
  2153. echo "Showing final status:"
  2154. print_status_table full
  2155. # Chain verify-pipeline.mjs
  2156. if [[ -f "$PROJECT_DIR/verify-pipeline.mjs" ]]; then
  2157. echo ""
  2158. echo "=== Running pipeline verification ==="
  2159. node "$PROJECT_DIR/verify-pipeline.mjs" || echo "โš ๏ธ Verification found issues"
  2160. fi
  2161. }
  2162. # Main
  2163. main() {
  2164. if [[ "$STATUS_ONLY" == "true" ]]; then
  2165. check_status_prerequisites
  2166. print_status_table
  2167. exit 0
  2168. fi
  2169. if [[ "$WATCH_MODE" == "true" ]]; then
  2170. check_status_prerequisites
  2171. watch_status
  2172. exit 0
  2173. fi
  2174. check_prerequisites
  2175. resolve_worker_model
  2176. if [[ "$DRY_RUN" == "false" ]]; then
  2177. acquire_lock
  2178. rm -f "$PAUSE_FILE" "$THROTTLE_FILE"
  2179. select_state_lock
  2180. fi
  2181. init_state
  2182. if [[ "$DRY_RUN" == "false" ]]; then
  2183. reconcile_recovery_records
  2184. fi
  2185. # Count input offers (skip header, ignore blank lines)
  2186. local total_input
  2187. total_input=$(tail -n +2 "$INPUT_FILE" | grep -c '[^[:space:]]' 2>/dev/null || true)
  2188. total_input="${total_input:-0}"
  2189. if (( total_input == 0 )); then
  2190. echo "No offers in $INPUT_FILE. Add offers first."
  2191. exit 0
  2192. fi
  2193. echo "=== career-ops batch runner ==="
  2194. if [[ "${MODE:-full}" == "match" ]]; then
  2195. echo "Mode: match (single-pass CV evaluation)"
  2196. fi
  2197. if (( LIMIT > 0 )); then
  2198. echo "Parallel: $PARALLEL | Max retries: $MAX_RETRIES | Limit: $LIMIT"
  2199. else
  2200. echo "Parallel: $PARALLEL | Max retries: $MAX_RETRIES"
  2201. fi
  2202. if [[ "$RESOLVED_SPEND_TIER" == "override" ]]; then
  2203. echo "CLI: $CLI | Model: $RESOLVED_MODEL (explicit --model override)"
  2204. elif [[ "$RESOLVED_SPEND_TIER" == "cli-default" ]]; then
  2205. echo "CLI: $CLI | Model: $CLI default"
  2206. else
  2207. echo "CLI: $CLI | Model: $RESOLVED_MODEL (spend_tier=${RESOLVED_SPEND_TIER})"
  2208. fi
  2209. echo "Input: $total_input offers"
  2210. echo ""
  2211. # Build list of offers to process
  2212. local -a pending_ids=()
  2213. local -a pending_urls=()
  2214. local -a pending_sources=()
  2215. local -a pending_notes=()
  2216. # --company: resolve the matching ids ONCE (batch/company-scope.mjs), then skip
  2217. # every other row before --limit is counted.
  2218. local scope_ids=""
  2219. if [[ -n "$SCOPE_COMPANY" ]]; then
  2220. if ! scope_ids=$(node "$BATCH_DIR/company-scope.mjs" --input "$INPUT_FILE" --company "$SCOPE_COMPANY"); then
  2221. echo "ERROR: could not resolve --company scope (company-scope.mjs failed)." >&2
  2222. exit 1
  2223. fi
  2224. scope_ids=" $(echo "$scope_ids" | tr '\n' ' ') "
  2225. echo "Scope: company=$SCOPE_COMPANY ($(echo "$scope_ids" | wc -w | tr -d ' ') matching row(s))"
  2226. fi
  2227. while IFS=$'\t' read -r id url source notes; do
  2228. [[ "$id" == "id" ]] && continue # skip header
  2229. [[ -z "$id" || -z "$url" ]] && continue
  2230. # Guard against non-numeric id values
  2231. [[ "$id" =~ ^[0-9]+$ ]] || continue
  2232. # Skip if before start-from
  2233. if (( id < START_FROM )); then
  2234. continue
  2235. fi
  2236. if [[ -n "$SCOPE_COMPANY" && "$scope_ids" != *" $id "* ]]; then
  2237. continue
  2238. fi
  2239. local status
  2240. status=$(get_status "$id")
  2241. # No command-line retry flag constitutes a user's agency answer.
  2242. if [[ "$status" == "needs_confirmation" ]]; then
  2243. echo "HOLD #$id: needs confirmation in the parent session ($url)"
  2244. continue
  2245. fi
  2246. if [[ "$RESUME_PAUSED" == "true" ]]; then
  2247. if [[ "$status" != "paused_rate_limit" ]]; then
  2248. continue
  2249. fi
  2250. elif [[ "$RETRY_FAILED" == "true" ]]; then
  2251. # Only process failed offers
  2252. if [[ "$status" != "failed" ]]; then
  2253. continue
  2254. fi
  2255. # Check retry limit
  2256. local retries
  2257. retries=$(get_retries "$id")
  2258. if (( retries >= MAX_RETRIES )); then
  2259. echo "SKIP #$id: max retries ($MAX_RETRIES) reached"
  2260. continue
  2261. fi
  2262. else
  2263. # Skip terminal offers
  2264. if [[ "$status" == "completed" || "$status" == "skipped" ]]; then
  2265. continue
  2266. fi
  2267. # Paused rate-limit offers resume explicitly with --resume-paused.
  2268. if [[ "$status" == "paused_rate_limit" ]]; then
  2269. continue
  2270. fi
  2271. # Skip failed offers that hit retry limit (unless --retry-failed)
  2272. if [[ "$status" == "failed" ]]; then
  2273. local retries
  2274. retries=$(get_retries "$id")
  2275. if (( retries >= MAX_RETRIES )); then
  2276. echo "SKIP #$id: failed and max retries reached (use --retry-failed to force)"
  2277. continue
  2278. fi
  2279. fi
  2280. fi
  2281. if (( LIMIT > 0 )) && (( ${#pending_ids[@]} >= LIMIT )); then
  2282. break
  2283. fi
  2284. pending_ids+=("$id")
  2285. pending_urls+=("$url")
  2286. pending_sources+=("$source")
  2287. pending_notes+=("$notes")
  2288. done < "$INPUT_FILE"
  2289. local pending_count=${#pending_ids[@]}
  2290. if (( pending_count == 0 )); then
  2291. echo "No offers to process."
  2292. print_summary
  2293. exit 0
  2294. fi
  2295. echo "Pending: $pending_count offers"
  2296. echo ""
  2297. # Dry run: just list
  2298. if [[ "$DRY_RUN" == "true" ]]; then
  2299. echo "=== DRY RUN (no processing) ==="
  2300. for i in "${!pending_ids[@]}"; do
  2301. local status
  2302. status=$(get_status "${pending_ids[$i]}")
  2303. echo " #${pending_ids[$i]}: ${pending_urls[$i]} [${pending_sources[$i]}] (status: $status)"
  2304. done
  2305. echo ""
  2306. echo "Would process $pending_count offers"
  2307. exit 0
  2308. fi
  2309. check_evidence_freshness
  2310. # --skip-pdf must not start a painter, even when the parent shell already opted in.
  2311. if [[ "$SKIP_PDF" == "true" ]]; then
  2312. unset CAREER_OPS_PDF_SHARED
  2313. else
  2314. export CAREER_OPS_PDF_SHARED=1
  2315. PDF_SERVER_OWNER=1
  2316. fi
  2317. # Process offers
  2318. BATCH_START=$SECONDS
  2319. local done_jobs=0
  2320. if (( PARALLEL <= 1 )); then
  2321. # Sequential processing
  2322. for i in "${!pending_ids[@]}"; do
  2323. JOB_POS="$((i + 1))/$pending_count"
  2324. process_offer "${pending_ids[$i]}" "${pending_urls[$i]}" "${pending_sources[$i]}" "${pending_notes[$i]}"
  2325. done_jobs=$((done_jobs + 1))
  2326. report_progress "$done_jobs" "$pending_count"
  2327. if [[ "$BATCH_PAUSED" == "true" || -f "$PAUSE_FILE" ]]; then
  2328. echo "=== Batch paused: session/rate limit reached. Resume later with --resume-paused. ==="
  2329. break
  2330. fi
  2331. done
  2332. else
  2333. # Parallel processing with job control
  2334. local running=0
  2335. local parallel_rc=0 worker_rc=0
  2336. local -a pids=()
  2337. local -a pid_ids=()
  2338. for i in "${!pending_ids[@]}"; do
  2339. if [[ "$BATCH_PAUSED" == "true" || -f "$PAUSE_FILE" ]]; then
  2340. echo "=== Batch paused: session/rate limit reached. Waiting for running workers, not scheduling new offers. ==="
  2341. break
  2342. fi
  2343. # Wait if we're at parallel limit. The throttle check opens every pass,
  2344. # so it also runs after the sleep below: a rate limit recorded while this
  2345. # loop slept drops the run to one worker before the next offer starts,
  2346. # not after it.
  2347. while :; do
  2348. if (( PARALLEL > 1 )) && [[ -f "${THROTTLE_FILE:-}" ]]; then
  2349. PARALLEL=1
  2350. echo "=== Rate limited: running the remaining offers on 1 worker. ==="
  2351. fi
  2352. if (( running < PARALLEL )); then
  2353. break
  2354. fi
  2355. # Wait for any child to finish
  2356. for j in "${!pids[@]}"; do
  2357. if ! kill -0 "${pids[$j]}" 2>/dev/null; then
  2358. worker_rc=0
  2359. wait "${pids[$j]}" 2>/dev/null || worker_rc=$?
  2360. if (( worker_rc != 0 )); then
  2361. parallel_rc=$worker_rc
  2362. fi
  2363. unset 'pids[j]'
  2364. unset 'pid_ids[j]'
  2365. running=$((running - 1))
  2366. done_jobs=$((done_jobs + 1))
  2367. report_progress "$done_jobs" "$pending_count"
  2368. fi
  2369. done
  2370. # Compact arrays
  2371. pids=(${pids[@]+"${pids[@]}"})
  2372. pid_ids=(${pid_ids[@]+"${pid_ids[@]}"})
  2373. if (( parallel_rc != 0 )); then
  2374. break
  2375. fi
  2376. if [[ "$BATCH_PAUSED" == "true" || -f "$PAUSE_FILE" ]]; then
  2377. echo "=== Batch paused: session/rate limit reached. Waiting for running workers, not scheduling new offers. ==="
  2378. break
  2379. fi
  2380. sleep 1
  2381. done
  2382. if (( parallel_rc != 0 )) || [[ "$BATCH_PAUSED" == "true" || -f "$PAUSE_FILE" ]]; then
  2383. break
  2384. fi
  2385. # Launch worker in background
  2386. JOB_POS="$((i + 1))/$pending_count"
  2387. process_offer "${pending_ids[$i]}" "${pending_urls[$i]}" "${pending_sources[$i]}" "${pending_notes[$i]}" &
  2388. pids+=($!)
  2389. pid_ids+=("${pending_ids[$i]}")
  2390. running=$((running + 1))
  2391. done
  2392. # Wait for remaining workers
  2393. for pid in ${pids[@]+"${pids[@]}"}; do
  2394. worker_rc=0
  2395. wait "$pid" 2>/dev/null || worker_rc=$?
  2396. if (( worker_rc != 0 )); then
  2397. parallel_rc=$worker_rc
  2398. fi
  2399. done_jobs=$((done_jobs + 1))
  2400. report_progress "$done_jobs" "$pending_count"
  2401. done
  2402. if (( parallel_rc != 0 )); then
  2403. echo "ERROR: a parallel worker failed fatally; tracker merge skipped. Inspect the worker log and repair state before rerunning." >&2
  2404. return "$parallel_rc"
  2405. fi
  2406. fi
  2407. # Merge tracker additions
  2408. merge_tracker
  2409. # Print summary
  2410. print_summary
  2411. exit 0
  2412. }
  2413. main "$@"
lib/ats-board-url.mjs ยท Bundled script ยท 232 lines ยท 2 findings marked
  1. // @ts-check
  2. /**
  3. * lib/ats-board-url.mjs: the canonical careers URL of an ATS board, in the
  4. * exact shape that ATS's provider detect() reads.
  5. *
  6. * Two callers depend on these being the same functions:
  7. * - discover-ats.mjs builds every probe candidate with BOARD_URL_BUILDERS,
  8. * so a board it confirms is one the scanner can later read;
  9. * - the employer registry (employers/) keys every row on the URL
  10. * buildBoardUrl() returns, and seeds/employers-validate.mjs rejects a row
  11. * whose careers_url is anything else.
  12. * One definition means a registry row and a discovered board cannot drift
  13. * apart. discover-ats.mjs re-exports buildBoardUrl and SLUG_RE.
  14. *
  15. * Dependency-free on purpose (no npm packages, no provider modules): the
  16. * registry's CI gate runs the validator without `npm install`, the same way
  17. * the plugin registry gate runs its validator.
  18. */
  19. // Safe charset for a slug that will be interpolated into an ATS URL. Mixed
  20. // case is intentional: Ashby boards are case-sensitive (AlephAlpha, DeepL).
  21. export const SLUG_RE = /^[A-Za-z0-9._-]+$/;
  22. const lower = (/** @type {string} */ s) => String(s).toLowerCase();
  23. /**
  24. * slug โ†’ careers_url on each vendor's default host. Vendors that put the
  25. * company in a subdomain are matched lowercase-only by their providers, so
  26. * their builder lowercases the slug rather than emit a URL the provider would
  27. * then reject.
  28. * @type {Readonly<Record<string, (slug: string) => string>>}
  29. */
  30. export const BOARD_URL_BUILDERS = Object.freeze({
  31. greenhouse: (s) => `https://job-boards.greenhouse.io/${s}`,
  32. ashby: (s) => `https://jobs.ashbyhq.com/${s}`,
  33. lever: (s) => `https://jobs.lever.co/${s}`,
  34. workable: (s) => `https://apply.workable.com/${s}`,
  35. smartrecruiters: (s) => `https://careers.smartrecruiters.com/${s}`,
  36. rippling: (s) => `https://ats.rippling.com/${s}/jobs`,
  37. join: (s) => `https://join.com/companies/${s}`,
  38. recruitee: (s) => `https://${lower(s)}.recruitee.com`,
  39. breezy: (s) => `https://${lower(s)}.breezy.hr`,
  40. bamboohr: (s) => `https://${lower(s)}.bamboohr.com`,
  41. pinpoint: (s) => `https://${lower(s)}.pinpointhq.com`,
  42. applicantpro: (s) => `https://${lower(s)}.applicantpro.com/jobs/`,
  43. hibob: (s) => `https://${lower(s)}.careers.hibob.com/jobs`,
  44. hirehive: (s) => `https://${lower(s)}.hirehive.com`,
  45. homerun: (s) => `https://${lower(s)}.homerun.co`,
  46. jazzhr: (s) => `https://${lower(s)}.applytojob.com/apply`,
  47. keka: (s) => `https://${lower(s)}.keka.com/careers`,
  48. personio: (s) => `https://${lower(s)}.jobs.personio.de`,
  49. prevueaps: (s) => `https://${lower(s)}.prevueaps.ca/jobs/`,
  50. schoolspring: (s) => `https://${lower(s)}.schoolspring.com`,
  51. taleez: (s) => `https://${lower(s)}.taleez.com`,
  52. teamtailor: (s) => `https://${lower(s)}.teamtailor.com`,
  53. trakstar: (s) => `https://${lower(s)}.hire.trakstar.com`,
  54. turbohire: (s) => `https://${lower(s)}.turbohire.co`,
  55. 'zoho-recruit': (s) => `https://${lower(s)}.zohorecruit.com/jobs/Careers`,
  56. });
  57. /**
  58. * Vendors that host boards in more than one region. A board in the EU region
  59. * is a different board from a US one with the same slug, so the region is part
  60. * of the URL and of the registry key. `us` is the default host above.
  61. * @type {Readonly<Record<string, Readonly<Record<string, (slug: string) => string>>>>}
  62. */
  63. export const REGIONAL_BOARD_URL_BUILDERS = Object.freeze({
  64. greenhouse: Object.freeze({
  65. us: BOARD_URL_BUILDERS.greenhouse,
  66. eu: (s) => `https://job-boards.eu.greenhouse.io/${s}`,
  67. }),
  68. lever: Object.freeze({
  69. us: BOARD_URL_BUILDERS.lever,
  70. eu: (s) => `https://jobs.eu.lever.co/${s}`,
  71. }),
  72. });
  73. export const REGIONS = Object.freeze(['us', 'eu']);
  74. /**
  75. * Vendors whose boards live on one default host (the builder above) or on one
  76. * of a few other hosts. The region names the other host; it is part of the URL
  77. * and of the registry key, and a board on the default host has no region.
  78. * - Zoho Recruit: <tenant>.zohorecruit.com by default; the region names its
  79. * other data-centre domains, au for .com.au and cn for .com.cn.
  80. * - Personio: <tenant>.jobs.personio.de by default (the host its provider
  81. * reads a bare tenant on); com for <tenant>.jobs.personio.com.
  82. * @type {Readonly<Record<string, Readonly<Record<string, (slug: string) => string>>>>}
  83. */
  84. export const OPTIONAL_REGION_BOARD_URL_BUILDERS = Object.freeze({
  85. personio: Object.freeze({
  86. com: (s) => `https://${lower(s)}.jobs.personio.com`,
  87. }),
  88. 'zoho-recruit': Object.freeze({
  89. au: (s) => `https://${lower(s)}.zohorecruit.com.au/jobs/Careers`,
  90. ca: (s) => `https://${lower(s)}.zohorecruit.ca/jobs/Careers`,
  91. cn: (s) => `https://${lower(s)}.zohorecruit.com.cn/jobs/Careers`,
  92. eu: (s) => `https://${lower(s)}.zohorecruit.eu/jobs/Careers`,
  93. in: (s) => `https://${lower(s)}.zohorecruit.in/jobs/Careers`,
  94. jp: (s) => `https://${lower(s)}.zohorecruit.jp/jobs/Careers`,
  95. sa: (s) => `https://${lower(s)}.zohorecruit.sa/jobs/Careers`,
  96. }),
  97. });
  98. /**
  99. * The regions a row of `ats` may name: required ones for greenhouse and lever,
  100. * optional ones for the vendors above, none elsewhere.
  101. * @param {string} ats
  102. * @returns {readonly string[]}
  103. */
  104. export function regionsOf(ats) {
  105. if (Object.hasOwn(REGIONAL_BOARD_URL_BUILDERS, ats)) return REGIONS;
  106. if (Object.hasOwn(OPTIONAL_REGION_BOARD_URL_BUILDERS, ats)) return Object.keys(OPTIONAL_REGION_BOARD_URL_BUILDERS[ats]);
  107. return [];
  108. }
  109. /** Vendors whose board id is the company's subdomain on the vendor's host. */
  110. export const SUBDOMAIN_ATS = Object.freeze([
  111. 'recruitee', 'breezy', 'bamboohr', 'pinpoint', 'applicantpro', 'hibob', 'hirehive', 'homerun', 'jazzhr', 'keka',
  112. 'personio', 'prevueaps', 'schoolspring', 'taleez', 'teamtailor', 'trakstar', 'turbohire', 'zoho-recruit',
  113. ]);
  114. /**
  115. * The subdomain vendors whose board id the builder and the validator check as
  116. * one DNS label of at most 63 characters, not one of the vendor's own
  117. * subdomains. recruitee, breezy, bamboohr and pinpoint, admitted first, keep
  118. * the rule they were admitted with.
  119. */
  120. export const DNS_LABEL_ATS = Object.freeze(
  121. SUBDOMAIN_ATS.filter((ats) => !['recruitee', 'breezy', 'bamboohr', 'pinpoint'].includes(ats)),
  122. );
  123. // One DNS label: letters, digits and inner hyphens, at most 63 characters.
  124. export const SUBDOMAIN_LABEL_RE = /^[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?$/;
  125. /**
  126. * Subdomains a vendor runs for itself, which its provider never reads as a
  127. * board: each list is the one in providers/<ats>.mjs. The validator and the
  128. * importer refuse them as board ids, and no URL is built for them.
  129. * @type {Readonly<Record<string, readonly string[]>>}
  130. */
  131. export const SUBDOMAIN_RESERVED_LABELS = Object.freeze({
  132. applicantpro: Object.freeze(['www', 'feeds']),
  133. hirehive: Object.freeze(['www', 'app', 'accounts', 'apply']),
  134. homerun: Object.freeze(['www', 'app', 'feed', 'help', 'static', 'cdn', 'api', 'developers', 'blog', '404']),
  135. keka: Object.freeze(['www', 'app', 'api', 'help', 'support', 'blog', 'docs', 'status', 'mail', 'developers']),
  136. schoolspring: Object.freeze(['www', 'api']),
  137. taleez: Object.freeze(['www', 'app', 'api', 'blog', 'files', 'static', 'help', 'aide', 'status']),
  138. trakstar: Object.freeze(['www', 'jsapi', 'api', 'app', 'mail', 'support', 'help', 'status', 'blog', 'docs']),
  139. turbohire: Object.freeze(['app', 'api', 'www']),
  140. });
  141. /**
  142. * Whether `board` is a board id a subdomain vendor can serve: one DNS label
  143. * (any letter case; the builders lowercase it) that is not one of the vendor's
  144. * own subdomains.
  145. * @param {string} ats
  146. * @param {string} board
  147. */
  148. export function isSubdomainBoard(ats, board) {
  149. const label = lower(board);
  150. if (!SUBDOMAIN_LABEL_RE.test(label)) return false;
  151. return !(Object.hasOwn(SUBDOMAIN_RESERVED_LABELS, ats) && SUBDOMAIN_RESERVED_LABELS[ats].includes(label));
  152. }
  153. // Workday needs three coordinates, not a slug: tenant, instance (wdN) and site.
  154. // The same board can be served from two host forms, so the form is recorded too:
  155. // jobs: https://<tenant>.<wdN>.myworkdayjobs.com/<site>
  156. // site: https://<wdN>.myworkdaysite.com/recruiting/<tenant>/<site>
  157. export const WORKDAY_HOSTS = Object.freeze(['jobs', 'site']);
  158. export const WORKDAY_SEGMENT_RE = /^[A-Za-z0-9_-]+$/;
  159. export const WORKDAY_INSTANCE_RE = /^wd[0-9]{1,3}$/;
  160. /** The closed list of ATS ids a registry row may name. */
  161. export const BOARD_ATS = Object.freeze([...Object.keys(BOARD_URL_BUILDERS), 'workday']);
  162. /**
  163. * Build a Workday careers URL from its coordinates.
  164. * @param {{tenant: string, instance: string, site: string, host?: string}} coords
  165. * @returns {string|null} null when a coordinate is malformed or the host form is unknown.
  166. */
  167. export function workdayBoardUrl({ tenant, instance, site, host = 'jobs' }) {
  168. if (typeof tenant !== 'string' || !WORKDAY_SEGMENT_RE.test(tenant)) return null;
  169. if (typeof site !== 'string' || !WORKDAY_SEGMENT_RE.test(site)) return null;
  170. if (typeof instance !== 'string' || !WORKDAY_INSTANCE_RE.test(instance)) return null;
  171. if (host === 'jobs') return `https://${tenant}.${instance}.myworkdayjobs.com/${site}`;
    • Dismissed after review: Looks up a DNS name built from data, which carries it out past web filters
    • Line 186 on GitHub
  172. if (host === 'site') return `https://${instance}.myworkdaysite.com/recruiting/${tenant}/${site}`;
    • Dismissed after review: Looks up a DNS name built from data, which carries it out past web filters
    • Line 187 on GitHub
  173. return null;
  174. }
  175. /**
  176. * The canonical careers URL for a registry row.
  177. *
  178. * `region` is required on the regional vendors (greenhouse, lever), optional on
  179. * personio and zoho-recruit (absent on the default host) and must be absent
  180. * elsewhere; `workday_host` is required on workday and must be absent
  181. * elsewhere; a Workday `board` is `tenant/wdN/site`; a subdomain board id
  182. * must be a host label its vendor serves. Anything that does not fit returns
  183. * null rather than a URL, so a caller never probes or stores a guess. The
  184. * builder checks the shape of the address only: the registry's admission rules
  185. * (length, numeric-only ids, placeholders, letter case) are the validator's.
  186. *
  187. * @param {{ats?: unknown, region?: unknown, board?: unknown, workday_host?: unknown}} row
  188. * @returns {string|null}
  189. */
  190. export function buildBoardUrl({ ats, region, board, workday_host } = {}) {
  191. if (typeof ats !== 'string' || typeof board !== 'string' || !board) return null;
  192. if (ats === 'workday') {
  193. if (region !== undefined || typeof workday_host !== 'string') return null;
  194. const parts = board.split('/');
  195. if (parts.length !== 3) return null;
  196. const [tenant, instance, site] = parts;
  197. return workdayBoardUrl({ tenant, instance, site, host: workday_host });
  198. }
  199. // Own-property lookups only: a plain index would find Object.prototype keys
  200. // ("constructor", "toString") and call them as builders.
  201. if (!Object.hasOwn(BOARD_URL_BUILDERS, ats)) return null;
  202. if (workday_host !== undefined || !SLUG_RE.test(board)) return null;
  203. if (DNS_LABEL_ATS.includes(ats) && !isSubdomainBoard(ats, board)) return null;
  204. if (Object.hasOwn(REGIONAL_BOARD_URL_BUILDERS, ats)) {
  205. const byRegion = REGIONAL_BOARD_URL_BUILDERS[ats];
  206. if (typeof region !== 'string' || !Object.hasOwn(byRegion, region)) return null;
  207. return byRegion[region](board);
  208. }
  209. if (region !== undefined && Object.hasOwn(OPTIONAL_REGION_BOARD_URL_BUILDERS, ats)) {
  210. const byRegion = OPTIONAL_REGION_BOARD_URL_BUILDERS[ats];
  211. if (typeof region !== 'string' || !Object.hasOwn(byRegion, region)) return null;
  212. return byRegion[region](board);
  213. }
  214. if (region !== undefined) return null;
  215. return BOARD_URL_BUILDERS[ats](board);
  216. }

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If itโ€™s helped, pass it on.