career-ops
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Check it deeper on your machine
It has changed since its scan on 11 Nov 2026. Its scripts were only partly scanned.
As a Claude Code plugin, install it and then ask Claude Code:
/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/career-ops-hq/career-ops with scanskillsafety
Or from a terminal, clone it and run it:
git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/career-ops-hq/career-ops'
It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.
Badge for your README
Markdown
[](/listings/career-ops-hq-career-ops)HTML
<a href="/listings/career-ops-hq-career-ops"><img src="/api/listings/career-ops-hq-career-ops/badge.svg" alt="findsafeskills safety rating" /></a>Install
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
**Fastest way, one command:**
npx @santifer/career-ops init
> ๐ก `npx` ships with [Node.js](https://nodejs.org): it runs the installer once, > without installing anything globally. No Node yet? Install it first. > (Already using a Claude Code / Codex CLI? Then you already have it.)
This clones the latest release into `./career-ops` and installs dependencies. Then:
cd career-ops claude # or codex / qwen / opencode / agy / grok โ open your AI CLI here
**On first launch, career-ops walks you through setup (your CV, profile and target roles) just by chatting. Nothing to edit by hand.**
<details> <summary><b>Prefer to set it up manually? (git clone)</b></summary>
git clone https://github.com/career-ops-hq/career-ops.git cd career-ops && npm install npx playwright install chromium # only needed for PDF generation # On a non-Debian/Ubuntu Linux distro (Fedora, Arch, ...), Chromium's system # libraries aren't installed by the line above โ install them yourself with # your distro's package manager if PDF generation fails to launch the browser # (Playwright's own docs list the required libraries per platform). # 2. Open your AI CLI in this directory. On first launch it walks you # through setup (your CV, profile and target roles) by chatting claude # or codex / opencode / qwen / agy / grok # Then ask your CLI to adapt the system to you: # "Change the archetypes to backend engineering roles" # "Translate the modes to English" # "Add these 5 companies to portals.yml" # "Update my profile with this CV I'm pasting" # 3. Start using # Paste a job URL or JD text to trigger auto-pipeline. # Every other mode, and how to ask for it in your CLI: see Usage below.
LinkedIn job alerts: paste the email. No Google account. See [docs/LINKEDIN_JOBS.md](docs/LINKEDIN_JOBS.md).
</details>
Global install
npm i -g @santifer/career-ops
This installs the `career-ops` binary globally, so you can run `career-ops init [folder]` without `npx`. `init` is its only command: it sets up a workspace, and everything after that happens inside your AI coding CLI (see Usage below), not through this binary.
**Which one should you use?** - `npx @santifer/career-ops init`: best for first use; creates a dedicated project folder. - `npm i -g @santifer/career-ops`: if you'd rather type `career-ops init` than the `npx` form. It adds no other commands.
Other ways to run it
- **From any folder, as a Claude Code plugin:** [docs/CLAUDE_PLUGIN.md](docs/CLAUDE_PLUGIN.md). - **Without an AI CLI:** standalone Gemini, Ollama and OpenAI-compatible scripts evaluate a JD on their own. See [docs/SCRIPTS.md](docs/SCRIPTS.md#geminieval--ollamaeval--openaieval).
> **The system is designed to be customized by your AI coding CLI itself.** Modes, archetypes, scoring weights, negotiation scripts -- just ask it to change them. It reads the same files it uses, so it knows exactly what to edit.
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
.claude-plugin/marketplace.json ยท Manifest ยท 20 lines
- {
- "name": "career-ops",
- "metadata": {
- "description": "Open-source AI job search that runs locally in your AI coding CLI: scan job portals, evaluate listings into a structured A-H report with a 1-5 score, tailor your CV, track applications",
- "version": "1.40.0"
- },
- "owner": {
- "name": "santifer",
- "url": "https://santifer.io"
- },
- "plugins": [
- {
- "name": "career-ops",
- "source": "./",
- "description": "Open-source AI job search that runs locally in your AI coding CLI: scan job portals, evaluate listings into a structured A-H report with a 1-5 score, tailor your CV, track applications",
- "category": "productivity",
- "version": "1.40.0"
- }
- ]
- }
README.md ยท README ยท 634 lines
- <p align="center"><picture><source media="(prefers-color-scheme: dark)" srcset="docs/wordmark-dark.svg"><img src="docs/wordmark-light.svg" alt="career-ops" width="250" height="56"></picture></p>
- <p align="center">The open-source AI job search agent.</p>
- <p align="center">
- <a href="README.md">English</a> |
- <a href="README.es.md">Espaรฑol</a> |
- <a href="README.de.md">Deutsch</a> |
- <a href="README.fr.md">Franรงais</a> |
- <a href="README.pt-BR.md">Portuguรชs (Brasil)</a> |
- <a href="README.ko-KR.md">ํ๊ตญ์ด</a> |
- <a href="README.ja.md">ๆฅๆฌ่ช</a> |
- <a href="README.cn.md">็ฎไฝไธญๆ</a> |
- <a href="README.zh-TW.md">็น้ซไธญๆ</a> |
- <a href="README.ua.md">ะฃะบัะฐัะฝััะบะฐ</a> |
- <a href="README.ru.md">ะ ัััะบะธะน</a> |
- <a href="README.pl.md">Polski</a> |
- <a href="README.da.md">Dansk</a> |
- <a href="README.ta.md">เฎคเฎฎเฎฟเฎดเฏ</a> |
- <a href="README.ar.md">ุงูุนุฑุจูุฉ</a> |
- <a href="README.hi.md">เคนเคฟเคจเฅเคฆเฅ</a> |
- <a href="README.tr.md">Tรผrkรงe</a> |
- <a href="README.vi.md">Tiแบฟng Viแปt</a> |
- <a href="README.id.md">Bahasa Indonesia</a>
- </p>
- <!-- The non-breaking spaces ( ) and word joiners (⁠) below decide where each line wraps on a phone. Keep them when editing. -->
- <p align="center"><a href="https://github.com/santifer"><img src="docs/avatar-santifer.png" width="48" height="48" align="middle" alt="Santiago Fernรกndez de Valderrama Aparicio"></a> Created by <a href="https://github.com/santifer"><strong>@santifer</strong></a>, with <a href="https://github.com/career-ops-hq/career-ops#builders"><picture><source media="(prefers-color-scheme: dark)" srcset="docs/builders-stack-dark.svg"><img src="docs/builders-stack-light.svg" alt="" height="32" align="middle"></picture></a> <a href="https://github.com/career-ops-hq/career-ops#builders">700+ contributors</a></p>
- <p align="center">
- <strong>Months of sending CVs into silence.</strong><br>
- So I built the filter I needed.
- </p>
- <p align="center">
- <strong>740 listings evaluated. 68 applied. 12 interviews. 1 offer.</strong><br>
- I was its first user. I got the job. Then I open-⁠sourced it.
- </p>
- <p align="center">
- Paste a job. On your machine, it tells you if it's still open and if it fits.<br>
- It tailors your CV and drafts your answers. <strong>You press Submit.</strong>
- </p>
- <p align="center">
- <img src="docs/demo.gif" alt="The author's own search in the career-ops dashboard: scored listings, a red count of the ones marked do not apply, then a full evaluation report" width="800">
- </p>
- <p align="center"><sub>My own search, partway through, with the UI in Spanish.</sub></p>
- <p align="center">
- Companies use AI to filter candidates.<br>
- <strong>I just gave candidates AI to <em>choose</em> companies.</strong>
- </p>
- <p align="center">
- Six months later, <strong>I left that job.</strong><br>
- Now we build career-ops, so you can land yours.
- </p>
- <p align="center"><a href="#your-turn">Try it on one job โ</a> ยท <a href="https://santifer.io/career-ops-system">Read the full story โ</a></p>
- <p align="center"><sub>Prefer not to use a terminal? <a href="https://career-ops.org/waitlist?ref=readme-en">Join the waitlist for a web version โ</a></sub></p>
- <br>
- <p align="center"><a href="HIRED.md"><img src="docs/hired-badge.svg" alt="Hired with career-ops: public stories on the record"></a></p>
- <p align="center">The people who came after me wrote down how they got hired.</p>
- <p align="center">
- <a href="HIRED.md"><img src="docs/hired-wall.svg" alt="Hired stories you can open and read" width="800"></a>
- </p>
- <p align="center"><sub>Every card is a public issue you can open. Landed yours? <a href="https://github.com/career-ops-hq/career-ops/issues/new?template=i-got-hired.yml">Leave your card โ</a></sub></p>
- <p align="center">โญ <strong>If career-ops helped you, a star helps the next person find it.</strong></p>
- <br>
- <p align="center">
- <a href="https://wired.com.gr/article/to-ai-ergaleio-pou-fernei-epanastasi-ston-tropo-pou-psachnoume-douleia/" rel="noopener noreferrer nofollow"><picture><source media="(prefers-color-scheme: dark)" srcset="docs/press/wired-dark.svg"><img src="docs/press/wired.svg" alt="WIRED" height="32"></picture></a>
-
- <a href="https://www.businessinsider.com/how-i-built-tool-filter-job-listings-landed-head-ai-2026-4" rel="noopener noreferrer nofollow"><picture><source media="(prefers-color-scheme: dark)" srcset="docs/press/business-insider-dark.svg"><img src="docs/press/business-insider.svg" alt="Business Insider" height="32"></picture></a>
- </p>
- <p align="center">
- <a href="https://vercel.com/open-source-program"><img alt="Vercel OSS Program" src="https://vercel.com/oss/program-badge-2026.svg" height="32"></a>
- </p>
- <p align="center">
- <a href="https://trendshift.io/repositories/25195" rel="noopener noreferrer"><img src="https://trendshift.io/api/badge/repositories/25195" alt="GitHub Trending: #1 repository of the day (Trendshift)" width="250" height="55"></a>
-
- <a href="https://www.producthunt.com/products/santifer-io?utm_source=badge-featured&utm_medium=badge" rel="noopener noreferrer"><img src="docs/press/producthunt.svg" alt="Featured on Product Hunt" width="206" height="54"></a>
- </p>
- ## Your turn
- Paste one job you were about to apply to tonight. If it comes back <em>do not apply</em>, you just got the night back. If it comes back with a plan, you know what to do next.
- ```bash
- npx @santifer/career-ops init
- ```
- <p align="center"><sub>Open source. Local. In the AI CLI you already use. <a href="docs/RUNNING_ON_A_BUDGET.md">Free and local models included.</a></sub></p>
- ## You don't have to search alone
- The silence after you hit send is not about you. Enough people saw the same thing to write the practice down, in six lines:
- > Apply better to fewer. Signal over volume. Evidence over keywords. A human decides. Local-first. Dignity on both sides of the table.
- <p align="center"><a href="https://career-ops.org/manifesto?utm_source=readme"><img src="https://img.shields.io/badge/dynamic/regex?url=https%3A%2F%2Fraw.githubusercontent.com%2Fcareer-ops-hq%2Fcareer-ops%2Fmain%2FSIGNATURES.md&search=n%3A(%5Cd%2B)%5Cs*%24&replace=%241%20signatories&label=THE%20CAREEROPS%20MANIFESTO&color=DD7627&style=for-the-badge&labelColor=2b3137" alt="The CareerOps manifesto: live count of signatories"></a></p>
- <p align="center">A few of them, in their own words.</p>
- <p align="center">
- <a href="https://career-ops.org/manifesto?utm_source=readme"><img src="docs/manifesto-wall.svg" alt="Three signatures of the CareerOps manifesto, in their own words" width="800"></a>
- </p>
- <p align="center"><sub>Every signature is a commit you can audit. <a href="https://career-ops.org/manifesto?utm_source=readme">Read them all, or add yours โ</a> ยท <a href="https://career-ops.org/manifesto?utm_source=readme#what-is-ai-job-search">What is AI job search?</a></sub></p>
- <br>
- Hiring will not fix itself. The people going through it can, and they are already in the room, comparing notes and fixing each other's setups.
- <a id="builders"></a>
- <p align="center"><a href="CONTRIBUTING.md"><img src="https://warpchart.dev/api/builders?lang=en" alt="Animated wall of contributors' GitHub avatars, filling in order of arrival since April 2026, with the next square left open." width="100%"></a></p>
- [This is where the fix gets written](CONTRIBUTING.md). A room, not a club. **Let's build together.**
- <p align="center"><a href="https://discord.gg/8pRpHETxa4"><img src="docs/discord-badge.svg" alt="Discord: live member count of the career-ops server"></a></p>
- ## Sponsors
- career-ops is free for candidates, forever. These companies sponsor the project:
- <p align="center">
- <a href="https://serpapi.com/career-ops-org" title="SerpApi"><img src="docs/sponsors/serpapi.svg" alt="SerpApi" height="48"></a>
- </p>
- <p align="center"><strong>SerpApi</strong> ยท Build a portfolio project with live search data. SerpApi gives developers structured JSON/Markdown from Google Search, Maps, Shopping, and other engines through a simple API call.</p>
- > Sponsorship buys clearly labeled visibility, never influence: no amount of money changes the roadmap or places anything in the product. Sponsors never appear in evaluations, rankings or recommendations.
- ## What career-ops does for you
- Paste a job. It tells you whether that night is worth it.
- - **Still open?** It checks that the posting is still live before you write a word.
- - **Not you?** It scores the role against your real CV and tells you to skip a weak fit. You can override it.
- - **Worth it?** It drafts the CV, the cover letter and the answers. You read them. You send them.
- - **Who do I talk to?** It finds the person and drafts the note. It never sends it.
- - **Where does it all go?** Every application stays on your machine. Nothing is uploaded to us.
- - **What should I learn?** After a run of noes, it names the gap.
- The first runs are rough. It does not know you yet. Talk to it: your CV, what you want, what you refuse. Think of it as a recruiter's first week.
- On first launch it asks for all that in chat. Nothing to configure by hand.
- ## What career-ops will not do
- - **Auto-submit an application.** It drafts the answer to every field; you review and click Submit. The script never POSTs (`prepare-application.mjs`).
- - **Send an email.** Drafts only. There is no mail transport anywhere in this codebase.
- - **Phone home.** No telemetry, no backend of ours. Your CV goes from your machine to the AI provider you chose, and nowhere else. The only public ledger is this repo: `HIRED.md` and its issues.
- - **Push you to apply below 4.0/5.** It will tell you not to. You can override it, and it will say so.
- It reformulates your CV; it must never fabricate it. A check in code stops a PDF whose numbers or facts appear in neither your CV nor your article digest, but it cannot judge every rewording yet. Read every CV before you send it. Details in the [FAQ](#faq).
- ## Features
- | Feature | Description |
- | ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------- |
- | **A-H Evaluation** | Role summary, CV match (with how much each requirement matters for this posting, and whether that weight came from the JD's own wording, its structure, or an estimate, labelled per requirement; an estimate can never be top-band), level strategy, comp research, personalization, interview prep (STAR+R) -- plus a Block G posting-legitimacy check (is it still open, is it a repost: observations, not verdicts), and a Work-Auth signal that flags an explicit no-sponsorship JD as a hard blocker |
- | **Human-in-the-Loop** | AI evaluates and recommends, you decide and act. The system never submits an application -- you always have the final call <!-- hitl: absolute guarantee. Do not add "automatically", "by itself", "without your permission" or any other hedge when translating this row. --> |
- | **ATS PDF Generation** | ATS-readable CVs tailored to each JD from your own experience, in Space Grotesk + DM Sans design |
- | **Cover Letter Generator** | Research-backed cover letters with keyword mirroring, four interactive angle prompts (why/problems/approach/tone), draft-in-chat approval gate, and A4 PDF via the same HTML + Playwright pipeline as CVs. Auto-drafts on every evaluation; complete and generate on demand via `/career-ops cover` |
- | **Beyond the CV** | Company research ([`deep`](modes/deep.md)) surfaces AI strategy, recent moves, engineering culture, and the angle your profile should take. Contact discovery ([`contacto`](modes/contacto.md)) identifies the hiring manager, recruiter, or team peer worth reaching out to and drafts a โค300-character LinkedIn message tuned to each contact type. Formal application email drafts ([`email`](modes/email.md)) turn an evaluated report or pasted JD into a subject line, body, and attachment checklist without sending, submitting, or clicking anything. Applications get you in the queue; research gets you a conversation. |
- | **Pattern Analysis** | Rejection patterns and per-ATS-channel advance rates (`analyze-patterns.mjs`), lifetime funnel stats (`stats.mjs`), repost detection, a possible sign of a ghost job (`detect-reposts.mjs`) |
- <details>
- <summary><b>Everything else it does</b></summary>
- | Feature | Description |
- | ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------- |
- | **Auto-Pipeline** | Paste a URL, get a full evaluation + PDF + tracker entry |
- | **Interview Story Bank** | Accumulates STAR+Reflection stories across evaluations -- 5-10 master stories you can adapt to the behavioral questions you get |
- | **Negotiation Scripts** | Salary negotiation frameworks, geographic discount pushback, competing offer leverage |
- | **Application Email Drafts** | Formal recruiter/referral/cold application emails from a report or pasted JD, with subject line, attachment checklist, source-backed fit points, and a profile-driven contact block. Draft-only -- career-ops never sends, submits, or clicks anything. |
- | **Portal Scanner** | 100+ companies pre-configured (Anthropic, OpenAI, ElevenLabs, Retool, n8n...) + custom web-search queries across Ashby, Greenhouse, Lever, Wellfound |
- | **Funded Company Discovery** | Review-first `npm run company:funded` script surfaces recently funded companies and source diagnostics from structured public feeds without editing your data |
- | **Batch Processing** | Parallel evaluation with headless CLI workers (`claude -p` / `opencode run`) |
- | **Dashboard TUI** | Terminal UI to browse, filter, and sort your pipeline |
- | **Pipeline Integrity** | Automated merge, dedup, status normalization, health checks |
- | **Interview Suite** | Time-blocked prep plans, practice sessions with feedback, post-interview debriefs ([`interview/`](modes/interview/README.md)), and a company red-flag detector ([`interview-redflag`](modes/interview-redflag.md)) |
- | **Offer Stage** | Contract reading companion -- clause walk plus a lawyer question list ([`offer-prep`](modes/offer-prep.md)) -- and a desired/advertised/actual salary-gap analyzer (`salary-gap.mjs`) |
- | **Follow-ups & Replies** | Follow-up cadence calculator and seeded reminders (`followup-cadence.mjs`, `followup-seed.mjs`); turn them off with `follow_ups.enabled: false` in `config/profile.yml`; employer reply classification into tracker updates ([`reply-watch`](modes/reply-watch.md)) |
- | **Plugin System** | Opt-in integrations (Gmail, Notion, Apify + a community registry), disabled by default -- see [docs/PLUGINS.md](docs/PLUGINS.md) |
- | **LinkedIn job alerts** | Paste a LinkedIn alert email on the dashboard or with `node linkedin-import.mjs`. No Google account. See [docs/LINKEDIN_JOBS.md](docs/LINKEDIN_JOBS.md) |
- </details>
- ## How It Works
- <p align="center"><picture><source media="(prefers-color-scheme: dark)" srcset="docs/how-it-works-dark.svg"><img src="docs/how-it-works-light.svg" alt="How it works: you paste a job URL or description. Archetype Detection classifies it as LLMOps, Agentic, PM, SA, FDE or Transformation. The A-H Evaluation reads cv.md and covers match, gaps, comp research, STAR stories and legitimacy. It produces a report (reports/*.md), the tailored CV in the format set by cv.output_format โ a PDF by default โ and a tracker entry (data/applications.md)." width="760"></picture></p>
- ## Quick Start
- **Fastest way, one command:**
- ```bash
- npx @santifer/career-ops init
- ```
- > ๐ก `npx` ships with [Node.js](https://nodejs.org): it runs the installer once,
- > without installing anything globally. No Node yet? Install it first.
- > (Already using a Claude Code / Codex CLI? Then you already have it.)
- This clones the latest release into `./career-ops` and installs dependencies. Then:
- ```bash
- cd career-ops
- claude # or codex / qwen / opencode / agy / grok โ open your AI CLI here
- ```
- **On first launch, career-ops walks you through setup (your CV, profile and target roles) just by chatting. Nothing to edit by hand.**
- <details>
- <summary><b>Prefer to set it up manually? (git clone)</b></summary>
- ```bash
- git clone https://github.com/career-ops-hq/career-ops.git
- cd career-ops && npm install
- npx playwright install chromium # only needed for PDF generation
- # On a non-Debian/Ubuntu Linux distro (Fedora, Arch, ...), Chromium's system
- # libraries aren't installed by the line above โ install them yourself with
- # your distro's package manager if PDF generation fails to launch the browser
- # (Playwright's own docs list the required libraries per platform).
- # 2. Open your AI CLI in this directory. On first launch it walks you
- # through setup (your CV, profile and target roles) by chatting
- claude # or codex / opencode / qwen / agy / grok
- # Then ask your CLI to adapt the system to you:
- # "Change the archetypes to backend engineering roles"
- # "Translate the modes to English"
- # "Add these 5 companies to portals.yml"
- # "Update my profile with this CV I'm pasting"
- # 3. Start using
- # Paste a job URL or JD text to trigger auto-pipeline.
- # Every other mode, and how to ask for it in your CLI: see Usage below.
- ```
- LinkedIn job alerts: paste the email. No Google account. See [docs/LINKEDIN_JOBS.md](docs/LINKEDIN_JOBS.md).
- </details>
- ### Global install
- ```bash
- npm i -g @santifer/career-ops
- ```
- This installs the `career-ops` binary globally, so you can run `career-ops init [folder]` without `npx`. `init` is its only command: it sets up a workspace, and everything after that happens inside your AI coding CLI (see Usage below), not through this binary.
- **Which one should you use?**
- - `npx @santifer/career-ops init`: best for first use; creates a dedicated project folder.
- - `npm i -g @santifer/career-ops`: if you'd rather type `career-ops init` than the `npx` form. It adds no other commands.
- ### Other ways to run it
- - **From any folder, as a Claude Code plugin:** [docs/CLAUDE_PLUGIN.md](docs/CLAUDE_PLUGIN.md).
- - **Without an AI CLI:** standalone Gemini, Ollama and OpenAI-compatible scripts evaluate a JD on their own. See [docs/SCRIPTS.md](docs/SCRIPTS.md#geminieval--ollamaeval--openaieval).
- > **The system is designed to be customized by your AI coding CLI itself.** Modes, archetypes, scoring weights, negotiation scripts -- just ask it to change them. It reads the same files it uses, so it knows exactly what to edit.
- See [docs/SETUP.md](docs/SETUP.md) for the full setup guide (including [scanning through an outbound proxy](docs/SETUP.md#scanning-through-an-outbound-proxy)), [docs/SUPPORTED_CLIS.md](docs/SUPPORTED_CLIS.md) for how to run career-ops in each supported CLI, and its headless command where it has one, [docs/RUNNING_ON_A_BUDGET.md](docs/RUNNING_ON_A_BUDGET.md) for instructions on running career-ops cheaply using custom or local models (and [docs/FREE_TIER.md](docs/FREE_TIER.md) for running it at zero cost on Antigravity CLI's free tier), [docs/AUTOMATION.md](docs/AUTOMATION.md) for scheduling recurring scans and a zero-token triage-to-shortlist recipe, [docs/KEEPING_PORTALS_LIVE.md](docs/KEEPING_PORTALS_LIVE.md) for keeping your company list current, [docs/BACKUP_STRATEGIES.md](docs/BACKUP_STRATEGIES.md) for backing up your CV, tracker and reports, [docs/APPLY_AUTOFILL.md](docs/APPLY_AUTOFILL.md) for details on the ATS auto-fill flow, [docs/LINKEDIN_JOIN.md](docs/LINKEDIN_JOIN.md) for cross-referencing a LinkedIn connections export against the companies in your funnel, [docs/LINKEDIN_ALERTS.md](docs/LINKEDIN_ALERTS.md) for adding LinkedIn job-alert emails on the scan schedule, and [docs/FAQ.md](docs/FAQ.md) for answers to common setup questions, including [how story provenance prevents invented numbers](docs/FAQ.md#why-does-career-ops-refuse-to-use-a-number-from-my-story-bank). Design principles live in [ARCHITECTURE.md](ARCHITECTURE.md); runtime flows in [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md).
- ## Usage
- Paste a job URL or description into your AI CLI. career-ops checks that a linked posting is still live (pasted text has no link to check), scores it against your CV, writes a report, tailors your CV (a PDF by default) and adds a row to your tracker. That covers most days. Everything else is a mode you ask for by name. If a posting can't be read, the pipeline stops and asks you to paste the text or share a screenshot.
- ### Asking for a mode
- | CLI | How you ask for `scan` |
- | --- | --- |
- | Claude Code, OpenCode, Antigravity CLI (`agy`), Grok Build CLI | `/career-ops scan` |
- | Pi | `/skill:career-ops scan` |
- | Codex, Qwen, Cursor, or any other supported CLI | Plain language: "Run the career-ops scan mode" |
- `/career-ops` with nothing after it shows the command menu in the CLIs that use it. Plain language works in every supported CLI. As a Claude Code plugin, the command is `/career-ops:career-ops`.
- All of them share one router, `.agents/skills/career-ops/SKILL.md`, and the same `modes/*.md` files, so every CLI routes a request the same way (browser-heavy modes also need Playwright tools). Slash commands are not guaranteed in Codex: ask in plain language, or run one prompt headless with `codex exec "Evaluate this JD with career-ops auto-pipeline: https://company.com/jobs/123"` ([docs/CODEX.md](docs/CODEX.md)). How to run it in each supported CLI, headless commands where a CLI has one, and per-CLI setup notes are in [docs/SUPPORTED_CLIS.md](docs/SUPPORTED_CLIS.md).
- ### The everyday commands
- | Command | What it does |
- | --- | --- |
- | *paste a URL or JD* | Full pipeline: liveness check (URLs only), A-G evaluation plus H at high scores, report, tailored CV (PDF by default), tracker row |
- | `scan` | Scan your portals for new postings and add them to the inbox (`data/pipeline.md`) |
- | `pipeline` | Work through the URLs waiting in the inbox (the standard and premium tiers may skip obvious mismatches) |
- | `apply` | Fill an application form with you, field by field. It stops before Submit |
- | `tracker` | Where every application stands |
- | `followup` | Which applications are due a follow-up, with drafts |
- <details>
- <summary><b>All commands, by stage of the search</b></summary>
- `oferta`, `ofertas` and `contacto` keep their original Spanish names: offer, offers, contact.
- **Find roles**
- | Command | What it does |
- | --- | --- |
- | [`scan`](modes/scan.md) | Scan portals and discover new offers |
- | [`discover`](modes/discover.md) | Resolve a list of company names to scannable ATS boards. Previews by default; adds them to `portals.yml` only with `--write` (zero-token) |
- | [`pipeline`](modes/pipeline.md) | Process pending URLs from the inbox (`data/pipeline.md`) |
- | [`triage`](modes/triage.md) | Fast first-pass go/no-go score from `modes/_brief.md`, before a full evaluation (writes no files) |
- | [`batch`](modes/batch.md) | Evaluate many postings with headless workers (parallel with Claude, one at a time with other CLIs) |
- | [`titles`](modes/titles.md) | Suggest adjacent job titles from your CV to broaden the search |
- **Evaluate a role**
- | Command | What it does |
- | --- | --- |
- | *paste a URL or JD* | [Auto-pipeline](modes/auto-pipeline.md): evaluation + report + tailored CV (PDF by default) + tracker entry |
- | [`oferta`](modes/oferta.md) | Evaluation only, Blocks A to G plus H at high scores (no PDF) |
- | [`ofertas`](modes/ofertas.md) | Compare and rank several offers |
- | [`deep`](modes/deep.md) | Deep research prompt about the company |
- | [`eu-swe`](modes/regional/eu-swe.md) | Calibrate a European software-engineering application before CV, apply or interview |
- **Apply**
- | Command | What it does |
- | --- | --- |
- | [`pdf`](modes/pdf.md) | ATS-optimized CV as a PDF |
- | [`ats`](modes/ats.md) | ATS-friendliness check of a generated CV: score and fixable issues |
- | [`text`](modes/text.md) | Tailored CV as markdown (mirrors `cv.md`, no PDF) |
- | [`latex`](modes/latex.md) | Export the CV as LaTeX/Overleaf `.tex` |
- | [`latex-tex`](modes/latex-tex.md) | Tailor your own `resume.tex` in place (opt-in; `cv.md` stays the default) |
- | [`cover`](modes/cover.md) | Cover letter, from a pasted JD or an evaluated report (`cover {slug}`) |
- | [`email`](modes/email.md) | Formal application email draft. Draft-only: never sends, submits or clicks |
- | [`contacto`](modes/contacto.md) | Find the hiring manager, recruiter or a peer, and draft a LinkedIn message |
- | [`apply`](modes/apply.md) | Live application assistant (Playwright from this checkout). Does not submit |
- **Interview**
- | Command | What it does |
- | --- | --- |
- | [`interview-prep`](modes/interview-prep.md) | Company-specific interview prep document |
- | [`interview/plan`](modes/interview/plan.md) | Time-blocked prep plan for an upcoming interview |
- | [`interview/practice`](modes/interview/practice.md) | Practice interview, one question at a time, with feedback |
- | [`interview/debrief`](modes/interview/debrief.md) | Post-interview debrief: close gaps, predict the next round |
- | [`interview-redflag`](modes/interview-redflag.md) | Analyze an employer's warning signs before you join |
- **After you apply**
- | Command | What it does |
- | --- | --- |
- | [`tracker`](modes/tracker.md) | Application status overview |
- | [`followup`](modes/followup.md) | Follow-up cadence: flag overdue applications, generate drafts |
- | [`reply-watch`](modes/reply-watch.md) | Classify employer replies and suggest tracker updates |
- | [`outcome`](modes/outcome.md) | Record an application's outcome and archive its artifacts |
- | [`offer-prep`](modes/offer-prep.md) | Read a received offer or contract with you: clause walk and questions for a lawyer (not legal advice) |
- **Your profile and CV**
- | Command | What it does |
- | --- | --- |
- | [`interview`](modes/interview.md) | Profile setup: an interactive interview about your experience that builds your CV and profile (not interview practice) |
- | [`intake`](modes/intake.md) | Build or enrich your profile from files in `documents/` (master CV, LinkedIn export). Nothing is written without your confirmation |
- | [`master-profile`](modes/master-profile.md) | Import, review and validate your Master Career Profile, and prepare selected facts as CV bullets |
- | [`add`](modes/add.md) | Add a project, paper or role to your CV (fetch, preview, confirm) |
- | [`expand`](modes/expand.md) | Discover competencies missing from your CV, from your profile links |
- | [`retro`](modes/retro.md) | Review this session and keep what you taught it as rules (writes only the ones you accept) |
- **Improve your targeting**
- | Command | What it does |
- | --- | --- |
- | [`patterns`](modes/patterns.md) | Analyze rejection patterns and improve targeting |
- | [`calibrate`](modes/calibrate.md) | Check whether your evaluation scores predict your real outcomes. Advisory: never changes scoring |
- | [`upskill`](modes/upskill.md) | Skill-gap analysis across your evaluated reports |
- | [`training`](modes/training.md) | Evaluate a course or certification against your North Star |
- | [`project`](modes/project.md) | Evaluate a portfolio project idea |
- **System**
- | Command | What it does |
- | --- | --- |
- | [`agent-inbox`](modes/agent-inbox.md) | Queue requests for the next session, and work through them then (`data/agent-inbox.md`). `inbox` works too |
- | [`update`](modes/update.md) | Update career-ops system files, with a diff preview and compatibility check |
- </details>
- ## Pre-configured Portals
- The scanner comes with **100+ companies** ready to scan and **35+ search queries** across major job boards. Copy `templates/portals.example.yml` to `portals.yml` and add your own:
- **AI Labs:** Anthropic, OpenAI, Mistral, Cohere, LangChain, Pinecone
- **Voice AI:** ElevenLabs, PolyAI, Parloa, Hume AI, Deepgram, Vapi, Bland AI
- **AI Platforms:** Retool, Airtable, Vercel, Temporal, Glean, Arize AI
- **Contact Center:** Ada, LivePerson, Sierra, Decagon, Talkdesk, Genesys
- **Enterprise:** Salesforce, Twilio, Gong, Dialpad
- **LLMOps:** Langfuse, Weights & Biases, Lindy, Cognigy, Speechmatics
- **Automation:** n8n, Zapier, Make.com
- **European:** Factorial, Attio, Tinybird, Clarity AI, Travelperk
- **Job boards searched:** 55+ provider modules cover ATS APIs, board-wide feeds, XML/RSS feeds, markdown feeds, and local parsers. See [Supported job boards](docs/SUPPORTED_JOB_BOARDS.md) for the full table.
- Keep the list current: `portals.yml` is designed to stay live, not be a one-time setup file. See [Keeping portals.yml live](docs/KEEPING_PORTALS_LIVE.md) for the house rule that adds new companies automatically and the commands to validate the file after every change.
- By default `node scan.mjs` (a.k.a. `npm run scan`) trusts what each ATS feed returns. Some companies leave stale postings in their public API even after the role is closed, so those expired entries can leak into `pipeline.md`. Pass `--verify` to launch Playwright after the API pass and drop expired postings before they hit the pipeline:
- ```bash
- node scan.mjs --verify # zero-token discovery + Playwright liveness check
- ```
- The verification is sequential and only runs against new offers (after dedup), so the cost stays bounded.
- ## Dashboard TUI
- The built-in terminal dashboard lets you browse your pipeline visually:
- ```bash
- npm run serve:dashboard # launch the TUI
- npm run build:dashboard # optional: build the standalone binary
- ```
- Features: 10 filter tabs, 8 sort modes, grouped/flat view, lazy-loaded previews, inline status changes.
- There is also an **experimental web UI** (alpha and opt-in: nothing runs unless you start it): see [`web/README.md`](web/README.md).
- ## Project Structure
- ```
- career-ops/
- โโโ AGENTS.md # Canonical agent instructions (all CLIs)
- โโโ CLAUDE.md # Claude Code wrapper (imports AGENTS.md)
- โโโ CODEX.md # Codex wrapper (imports AGENTS.md)
- โโโ OPENCODE.md # OpenCode wrapper (imports AGENTS.md)
- โโโ GEMINI.md # Legacy no-op guard to avoid Antigravity duplicate context
- โโโ cv.md # Your CV (create this)
- โโโ article-digest.md # Your proof points (optional)
- โโโ config/
- โ โโโ profile.example.yml # Template for your profile
- โโโ modes/ # Skill modes
- โ โโโ _shared.md # Shared context (system file; local edit kept unless apply --force)
- โ โโโ _profile.md # Your archetypes and framing (customize this)
- โ โโโ oferta.md # Single evaluation
- โ โโโ pdf.md # PDF generation
- โ โโโ cover.md # Cover letter generation
- โ โโโ email.md # Formal application email drafts
- โ โโโ scan.md # Portal scanner
- โ โโโ batch.md # Batch processing
- โ โโโ ...
- โโโ templates/
- โ โโโ cv-template.html # ATS-optimized CV template
- โ โโโ portals.example.yml # Scanner config template
- โ โโโ states.yml # Canonical statuses
- โโโ batch/
- โ โโโ batch-prompt.md # Self-contained worker prompt
- โ โโโ batch-runner.sh # Orchestrator script
- โโโ dashboard/ # Go TUI pipeline viewer
- โโโ data/ # Your tracking data (gitignored)
- โโโ reports/ # Evaluation reports (gitignored)
- โโโ output/ # Generated PDFs (gitignored)
- โโโ fonts/ # Space Grotesk + DM Sans
- โโโ docs/ # Setup, customization, budget guide, architecture
- โโโ examples/ # Sample CV, report, proof points
- ```
- ## External Data Directory (Optional)
- By default, user-layer data (such as `cv.md`, `portals.yml`, and `data/` / `reports/` / `output/` folders) lives inside the project root folder.
- To separate your personal data from the code (making it easier to switch branches, pull updates, or test multiple profiles), you can configure an external data directory using the following precedence:
- 1. **Environment Variables:** Set the `CAREER_OPS_ROOT` or `CAREER_OPS_DATA_DIR` environment variable:
- ```bash
- export CAREER_OPS_ROOT=~/my-career-data
- ```
- 2. **Marker File:** Create a `.career-ops-data` file in the repository root containing the path to your data directory.
- 3. **Default:** Defaults to the repository root.
- Once resolved, all user files are resolved and written relative to that folder, while prompt files and scripts continue to resolve relative to the repository.
- - **Tracker Override:** You can also set `CAREER_OPS_TRACKER` to override the applications tracker file path directly.
- - **Writes:** All write operations (such as merges) canonically target `{DATA_ROOT}/data/applications.md`.
- - **Scanner config:** `portals.yml` is read and validated at `{DATA_ROOT}/portals.yml`, so `node validate-portals.mjs` checks the same file `scan.mjs` reads.
- - **Generated documents:** tailored CVs and cover letters are written under `{DATA_ROOT}/output/`, and the PDF manifest that links them to a report lives at `{DATA_ROOT}/data/pdf-index.tsv`. While `CAREER_OPS_TRACKER` is unset, the tracker workspace that bounds those writes is the data root, not the checkout.
- - **PDFs under a tracker override:** `generate-pdf.mjs` resolves `CAREER_OPS_TRACKER` before it derives the workspace, so with the override set the workspace is the folder that holds that tracker (or the folder above it, when the tracker sits in a `data/` folder). The CV's HTML and every PDF must then sit inside that workspace, and the manifest moves to its `data/pdf-index.tsv`. Cover letters still target `{DATA_ROOT}/output/`, so they are refused when that folder falls outside the tracker's workspace.
- - **Code layer stays put:** `node_modules/`, `providers/`, `modes/` and the scripts themselves always resolve against the repository, never the data root.
- The Go dashboard TUI, Node.js scripts, and AI agent modes all automatically respect this resolution hierarchy.
- ## Tech Stack
- 
- 
- 
- 
- 
- - **Agent**: AI coding CLI with shared skills and modes (`AGENTS.md` + CLI wrapper)
- - **PDF**: Playwright + HTML template
- - **Cover letters**: HTML template + Playwright (A4 PDF, same pipeline as CVs)
- - **Scanner**: Playwright + Greenhouse API + WebSearch
- - **Dashboard**: Go + Bubble Tea + Lipgloss (Catppuccin Mocha theme)
- - **Data**: Markdown tables + YAML config + TSV batch files
- <p align="center">
- <a href="https://github.com/career-ops-hq/career-ops/releases/latest"><img src="docs/npm-badge.svg" alt="Latest release"></a>
- </p>
- <p align="center">
- <a href="https://claude.com/claude-code"><img src="https://img.shields.io/badge/Built_with-Claude_Code-000?style=for-the-badge&logo=anthropic&logoColor=white" alt="Built with Claude Code"></a>
- </p>
- <p align="center">
- <sub>Also runs on any agent-skill-standard CLI. See <a href="docs/SUPPORTED_CLIS.md">Supported CLIs</a>.</sub><br>
- <img src="https://img.shields.io/badge/Claude_Code-000?style=flat&logo=anthropic&logoColor=white" alt="Claude Code">
- <img src="https://img.shields.io/badge/OpenCode-111827?style=flat&logo=terminal&logoColor=white" alt="OpenCode">
- <img src="https://img.shields.io/badge/Antigravity_CLI-4285F4?style=flat&logo=google&logoColor=white" alt="Antigravity CLI">
- <img src="https://img.shields.io/badge/Codex-412991?style=flat&logo=openai&logoColor=white" alt="Codex">
- <img src="https://img.shields.io/badge/Qwen-615CED?style=flat" alt="Qwen">
- <img src="https://img.shields.io/badge/Kimi-FF4B4B?style=flat" alt="Kimi">
- <img src="https://img.shields.io/badge/GitHub_Copilot-000?style=flat&logo=githubcopilot&logoColor=white" alt="GitHub Copilot">
- <img src="https://img.shields.io/badge/Grok_Build_CLI-000?style=flat&logo=x&logoColor=white" alt="Grok Build CLI">
- <img src="https://img.shields.io/badge/Pi-4B3F72?style=flat" alt="Pi">
- <img src="https://img.shields.io/badge/Oh_My_Pi-4B3F72?style=flat" alt="Oh My Pi">
- <br>
- <img src="https://img.shields.io/badge/Node.js-339933?style=flat&logo=node.js&logoColor=white" alt="Node.js">
- <img src="https://img.shields.io/badge/Go-00ADD8?style=flat&logo=go&logoColor=white" alt="Go">
- <img src="https://img.shields.io/badge/Playwright-2EAD33?style=flat&logo=playwright&logoColor=white" alt="Playwright">
- <img src="https://img.shields.io/badge/Bubble_Tea-FF75B5?style=flat&logo=go&logoColor=white" alt="Bubble Tea">
- <img src="https://img.shields.io/badge/License-MIT-blue.svg" alt="MIT">
- <a href="TRADEMARK.md"><img src="https://img.shields.io/badge/Trademark-Policy-blue.svg" alt="Trademark Policy"></a>
- </p>
- ## Also Open Source
- - **[cv-santiago](https://github.com/santifer/cv-santiago)** -- The portfolio website (santifer.io) with AI chatbot, LLMOps dashboard, and case studies. If you need a portfolio to showcase alongside your job search, fork it and make it yours.
- ## FAQ
- **What is career-ops?**
- career-ops ([career-ops.org](https://career-ops.org), also known as **careerops**) is an open-source AI job search that runs locally in your AI coding CLI (Claude Code, Codex, OpenCode and others) and leaves every decision to you. It evaluates job offers against your CV, generates ATS-tailored PDFs, finds the right person to contact, and tracks everything in one place: you always have the final call. It is the first reference implementation of the [CareerOps Manifesto](https://career-ops.org/manifesto).
- **Can the tailored CV invent things?**
- It must not, and the prompts say so: reformulate, never fabricate. `generate-pdf` blocks a CV whose numbers or facts are missing from your sources, unless you pass `--skip-fact-check`. The `pdf` mode also runs `cv-title-check.mjs`, which compares each job title in the tailored CV with the title cv.md gives the same job (matched by company and dates) and asks you to keep or revert any that differ. It warns rather than blocks, because reframing a title is sometimes your call. Nothing judges rewording yet: [#1411](https://github.com/career-ops-hq/career-ops/issues/1411) (fail-closed faithfulness check) tracks that. Until then, read every CV before you send it. The [Legal Disclaimer](LEGAL_DISCLAIMER.md) says the same in longer words.
- **Can I run career-ops for free, or on a cheaper / local model?**
- Yes. career-ops is CLI-agnostic and runs on free and local models (OpenRouter free models, Ollama or any OpenAI-compatible endpoint), so you are not tied to a paid subscription. See [docs/RUNNING_ON_A_BUDGET.md](docs/RUNNING_ON_A_BUDGET.md) for the full setup.
- **I pay for Claude Pro/Max but career-ops is burning API credits. Why?**
- Because an `ANTHROPIC_API_KEY` in your environment takes precedence over your logged-in subscription: the CLI uses the key and bills per token. Run `echo $ANTHROPIC_API_KEY`, and if it prints anything, remove it from your shell profile, restart the terminal and run `/login`. Batch mode is the exception, since `claude -p` workers do not use the interactive login: run `claude setup-token` once and export the result as `CLAUDE_CODE_OAUTH_TOKEN`. Full walkthrough in [docs/RUNNING_ON_A_BUDGET.md](docs/RUNNING_ON_A_BUDGET.md#2b-already-paying-for-a-subscription-make-sure-you-are-using-it).
- **Which AI CLIs does career-ops work with?**
- career-ops runs on any major AI coding CLI (Claude Code, Codex, Gemini / Antigravity, OpenCode, Grok, Qwen and more) through the open Agent Skill Standard, so it is never locked to a single vendor. Use the CLI you already have.
- **How do I install career-ops on Windows?**
- career-ops runs on Windows. Platform-specific setup and the known sharp edges (Git Bash discovery, line endings, Task Scheduler) are in [docs/WINDOWS.md](docs/WINDOWS.md). If skills fail to load with a symlink error during install, the fix is in [docs/FAQ.md](docs/FAQ.md). Full steps are in [docs/SETUP.md](docs/SETUP.md). On NixOS, use the flake dev shell instead of `npx playwright install`; see [docs/NIXOS.md](docs/NIXOS.md).
- **Does career-ops auto-apply to jobs for me?**
- No. career-ops is a filter, not a spray-and-pray auto-applier. The AI evaluates, ranks and drafts; you review and decide. It drafts and fills; it never submits. You press Submit. That human-in-the-loop design is the whole point.
- **Is career-ops free and open source?**
- Yes. career-ops is free and open source, and for the candidate it always will be. It is the first reference implementation of the [CareerOps Manifesto](https://career-ops.org/manifesto). Read it, and if it says what you believe, sign it.
- ## About the Author
- I'm [Santiago Fernรกndez de Valderrama Aparicio](https://santifer.io/about) (santifer), a former founder: I built and sold a business that still runs with my name on it. I built career-ops to manage my own job search, and it worked: it landed me a Head of Applied AI role. Six months later I left that role to focus on building career-ops full time.
- Curious how a repo this size is maintained with a fleet of AI agents and a human deciding every merge? Read [Agentic maintenance: how career-ops is run by a fleet of AI agents](https://santifer.io/ai-agent-fleet).
- My portfolio and other open source projects โ [santifer.io](https://santifer.io)
- Wikidata: [Santiago Fernรกndez de Valderrama Aparicio](https://www.wikidata.org/wiki/Q138710224) ยท [career-ops](https://www.wikidata.org/wiki/Q139007988).
- ## Disclaimer
- **career-ops is a local, open-source tool, NOT a hosted service.** By using this software, you acknowledge:
- 1. **You control your data.** Your CV, contact info, and personal data stay on your machine and are sent directly to the AI provider you choose (Anthropic, OpenAI, etc.). We do not collect, store, or have access to any of your data.
- 2. **You control the AI.** The default prompts instruct the AI not to auto-submit applications, but AI models can behave unpredictably. If you modify the prompts or use different models, you do so at your own risk. **Always review AI-generated content for accuracy before submitting.**
- 3. **You comply with third-party ToS.** You must use this tool in accordance with the Terms of Service of the career portals you interact with (Greenhouse, Lever, Workday, LinkedIn, etc.). Do not use this tool to spam employers or overwhelm ATS systems.
- 4. **No guarantees.** Evaluations are recommendations, not truth. AI models may hallucinate skills or experience. The authors are not liable for employment outcomes, rejected applications, account restrictions, or any other consequences.
- See [LEGAL_DISCLAIMER.md](LEGAL_DISCLAIMER.md) for full details. This software is provided under the [MIT License](LICENSE) "as is", without warranty of any kind.
- ## Contributors
- <a href="https://github.com/career-ops-hq/career-ops/graphs/contributors">
- <img src="https://contrib.rocks/image?repo=career-ops-hq/career-ops&v=2" />
- </a>
- Every person who has shipped code, docs, translations or tests is listed in
- [CONTRIBUTORS.md](CONTRIBUTORS.md), including non-code contributions, which
- the graph above cannot show.
- Got hired using career-ops? [Share your story!](https://github.com/career-ops-hq/career-ops/issues/new?template=i-got-hired.yml)
- ## License & Trademark
- The code is licensed under [MIT](LICENSE). The "career-ops" name and
- brand are governed by the [Trademark Policy](TRADEMARK.md), permissive
- for community use, reserved for commercial product naming and
- endorsement.
- ## Star history
- <p align="center">
- <a href="https://warpchart.dev/hq">
- <picture>
- <source media="(prefers-color-scheme: dark)" srcset="https://warpchart.dev/api/chart?theme=dark&v=3">
- <img alt="Live star telemetry of career-ops-hq/career-ops" src="https://warpchart.dev/api/chart?theme=light&v=3" loading="lazy">
- </picture>
- </a>
- </p>
- <p align="center"><sub>Created and maintained by <a href="https://santifer.io">Santiago Fernรกndez de Valderrama Aparicio</a> (<a href="https://github.com/santifer">@santifer</a>)</sub></p>
- ## Let's Connect
- [](https://santifer.io)
- [](https://linkedin.com/in/santifer)
- [](https://x.com/santifer)
- [](https://discord.gg/8pRpHETxa4)
- [](mailto:hi@santifer.io)
batch/batch-runner.sh ยท Bundled script ยท 2,601 lines ยท 1 finding marked
- #!/usr/bin/env bash
- set -euo pipefail
- # career-ops batch runner โ standalone orchestrator for headless agent workers
- # Reads batch-input.tsv, delegates each offer to a worker, tracks state in
- # batch-state.tsv for resumability.
- #
- # Supported CLIs (--cli flag):
- # claude โ claude -p with --dangerously-skip-permissions (default)
- Note: Starts a coding agent with its permission prompts off
- Line 9 on GitHub
- # opencode โ opencode run (falls back to ollama launch opencode if not in PATH)
- # gemini โ gemini -p
- # qwen โ qwen -p
- # grok โ grok -p with --always-approve
- #
- # Only claude supports --strict-mcp-config, the rate-limit/session retry loop,
- # and --parallel > 1; other CLIs run sequentially with a single attempt.
- SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
- PROJECT_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
- # Under Git Bash / MSYS, pwd gives a POSIX path (/c/...). Node is a native
- # Windows binary and gets it unconverted when MSYS path conversion is off
- # (MSYS_NO_PATHCONV=1), so every `node "$PROJECT_DIR/..."` call failed with
- # MODULE_NOT_FOUND (#4910). cygpath -m yields C:/... which both bash and node
- # resolve, as for jd_file below.
- case "$(uname -s)" in
- MINGW*|MSYS*|CYGWIN*) command -v cygpath >/dev/null 2>&1 && PROJECT_DIR="$(cygpath -m "$PROJECT_DIR")" ;;
- esac
- # The user layer (profile, reports, tracker additions) lives under the data
- # root, which CAREER_OPS_ROOT / CAREER_OPS_DATA_DIR / a .career-ops-data marker
- # can place outside the checkout. Every path here used to be built from
- # PROJECT_DIR, so under an external data root workers were scored against the
- # checkout's template profile, and their reports and tracker lines were looked
- # for in directories reserve-report-num.mjs and merge-tracker.mjs never read
- # (#5115). path-resolver.mjs owns the precedence, so ask it instead of
- # re-deriving it in bash. It prints nothing when the data root is the checkout,
- # which keeps DATA_ROOT spelled exactly like PROJECT_DIR in the default layout.
- DATA_ROOT="$PROJECT_DIR"
- if [[ -f "$PROJECT_DIR/path-resolver.mjs" ]]; then
- if ! resolved_data_root="$(node --input-type=module -e '
- import { realpathSync } from "node:fs";
- import { dirname } from "node:path";
- import { pathToFileURL } from "node:url";
- const resolver = process.argv[1];
- const { getCareerOpsRoot } = await import(pathToFileURL(resolver).href);
- const real = (p) => { try { return realpathSync(p); } catch { return p; } };
- const root = getCareerOpsRoot();
- if (real(root) !== real(dirname(resolver))) {
- process.stdout.write(process.platform === "win32" ? root.replace(/\\/g, "/") : root);
- }
- ' "$PROJECT_DIR/path-resolver.mjs")"; then
- echo "ERROR: could not resolve the career-ops data root through path-resolver.mjs" >&2
- exit 1
- fi
- if [[ -n "$resolved_data_root" ]]; then
- DATA_ROOT="$resolved_data_root"
- fi
- fi
- BATCH_DIR="$SCRIPT_DIR"
- INPUT_FILE="$BATCH_DIR/batch-input.tsv"
- STATE_FILE="$BATCH_DIR/batch-state.tsv"
- PROMPT_FILE="$BATCH_DIR/batch-prompt.md"
- # The profile follows CAREER_OPS_PROFILE as well as the data root, like every
- # other reader does through path-resolver.mjs (#5144). Building it from the
- # data root here left a run reading the data root's profile for spend_tier and
- # for the prompt's personalization while the .mjs readers used the override.
- # The .md prompt files still name config/profile.yml; the script-generated
- # runtime block and the injected personalization use $PROFILE_FILE.
- PROFILE_FILE="$DATA_ROOT/config/profile.yml"
- PROFILE_RESOLVED=false
- if [[ -f "$PROJECT_DIR/path-resolver.mjs" ]]; then
- if ! resolved_profile_file="$(node --input-type=module -e '
- import { pathToFileURL } from "node:url";
- const resolver = process.argv[1];
- const { getProfilePath } = await import(pathToFileURL(resolver).href);
- if (typeof getProfilePath !== "function") {
- process.stderr.write("WARN: path-resolver.mjs has no getProfilePath(), so CAREER_OPS_PROFILE is ignored in this run. Update the install to use the override.\n");
- } else {
- const profile = getProfilePath();
- process.stdout.write(process.platform === "win32" ? profile.replace(/\\/g, "/") : profile);
- }
- ' "$PROJECT_DIR/path-resolver.mjs")"; then
- echo "ERROR: could not resolve the career-ops profile path through path-resolver.mjs" >&2
- exit 1
- fi
- if [[ -n "$resolved_profile_file" ]]; then
- PROFILE_FILE="$resolved_profile_file"
- PROFILE_RESOLVED=true
- fi
- elif [[ -n "${CAREER_OPS_PROFILE:-}" ]]; then
- echo "WARN: no path-resolver.mjs next to this script, so CAREER_OPS_PROFILE is ignored in this run." >&2
- fi
- if [[ -n "${CAREER_OPS_PROFILE:-}" && "$PROFILE_RESOLVED" == "true" && ! -f "$PROFILE_FILE" ]]; then
- # Before the override was honored this read the data root's profile anyway.
- # Say it, instead of silently running with no profile at all.
- echo "WARN: no profile at $PROFILE_FILE (CAREER_OPS_PROFILE); the prompt gets no profile and spend_tier is not taken from it." >&2
- fi
- LOGS_DIR="$BATCH_DIR/logs"
- DISCARD_LOG="$LOGS_DIR/discard.log"
- TRACKER_DIR="$DATA_ROOT/batch/tracker-additions"
- REPORTS_DIR="$DATA_ROOT/reports"
- # The runner's own bookkeeping (input, state, logs, locks) stays beside the
- # script. reserve-report-num.mjs and merge-tracker.mjs read batch-state.tsv
- # there too, and a data root's own copy beside it (resolveBatchStatePaths), and
- # both skip report numbers recorded as failed. The variable still names the file for every worker, and
- # learn.mjs reads it to tell a batch worker. Built from PROJECT_DIR, which is
- # the form node can open under Git Bash (see above).
- export CAREER_OPS_BATCH_STATE="$PROJECT_DIR/batch/batch-state.tsv"
- # Workers run unattended, so nobody can confirm a change to the user's standing
- # rules. modes/_shared.md tells an agent that sees this flag not to write them.
- export CAREER_OPS_HEADLESS=1
- LOCK_FILE="$BATCH_DIR/batch-runner.pid"
- PAUSE_FILE="$BATCH_DIR/batch-runner.paused"
- THROTTLE_FILE="$BATCH_DIR/batch-runner.throttled"
- STATE_LOCK_DIR="$BATCH_DIR/.batch-state.lock"
- STATE_LOCK_PID_FILE="$STATE_LOCK_DIR/pid"
- STATE_LOCK_FLOCK_FILE="$BATCH_DIR/.batch-state.flock"
- STATE_LOCK_FLOCK=0
- STATE_LOCK_TIMEOUT_SECONDS=30
- MAIN_PID="${BASHPID:-$$}"
- # Defaults
- PARALLEL=""
- DEFAULT_CLAUDE_PARALLEL=3
- DRY_RUN=false
- RETRY_FAILED=false
- RESUME_PAUSED=false
- START_FROM=0
- SCOPE_COMPANY=""
- MAX_RETRIES=2
- MIN_SCORE=0
- SKIP_PDF=false
- MODE="full"
- MODEL="" # explicit override; otherwise resolved from the profile's spend_tier
- RESOLVED_MODEL=""
- RESOLVED_SPEND_TIER=""
- CLI=claude
- RATE_LIMIT_SLEEP=300
- WORKER_TIMEOUT=1800
- THIN_ACTION=webfetch # webfetch | skip | browser-extract: what to do when the JD is still empty after the prefetch and the ATS API fallback
- BATCH_PAUSED=false
- EVIDENCE_FRESHNESS_JSON="" # set once per run by check_evidence_freshness
- STATUS_ONLY=false
- STATUS_FULL=true
- WATCH_MODE=false
- LIMIT=0 # internal sentinel only; explicit --limit must be positive
- # Return success for non-negative integer or decimal strings.
- is_decimal_number() {
- [[ "$1" =~ ^[0-9]+([.][0-9]+)?$ ]]
- }
- usage() {
- cat <<'USAGE'
- career-ops batch runner โ process job offers in batch via headless agent workers
- Defaults to claude; other CLIs via --cli. For claude the model is resolved from
- spend_tier in config/profile.yml (or in the file CAREER_OPS_PROFILE names)
- unless --model overrides it.
- Usage: batch-runner.sh [OPTIONS]
- Options:
- --cli NAME Agent CLI to use: claude (default), opencode, gemini, qwen, grok
- --model NAME Model for the CLI (e.g. qwen2.5:32b for opencode/ollama).
- For claude, overrides the tier-resolved model (otherwise
- the spend_tier in config/profile.yml, or in the file
- CAREER_OPS_PROFILE names: economy/standard/premium;
- default standard).
- --parallel N Number of parallel workers (default: 3 for claude, 1 otherwise).
- On a rate limit, the rest of the batch runs on 1 worker.
- --dry-run Show what would be processed, don't execute
- --retry-failed Only retry offers marked as "failed" in state
- --resume-paused Resume offers paused by a Claude session/rate limit
- --start-from N Start from offer ID N (skip earlier IDs)
- --company A,B Only process offers whose company matches (case- and
- punctuation-insensitive substring of the notes' company
- half or of the URL, leaving out a shared ATS domain;
- comma-separated). Applies before --limit
- --limit N Max offers: positive integer; omit for no limit
- --max-retries N Max retry attempts per offer (default: 2)
- --min-score N Skip PDF/tracker for offers scoring below N (default: 0 = off)
- --skip-pdf Skip PDF generation entirely (write โ in tracker PDF column)
- --worker-timeout N Stop a worker after N seconds and record it as failed
- (default: 1800; 0 = no limit)
- --mode MODE Evaluation mode: full (default) or match (condensed single-pass CV match)
- --single-pass Alias for --mode match
- --thin-action MODE What to do when no usable JD text could be fetched (JS-rendered
- page, empty shell): 'webfetch' (default) spawns the worker and
- lets it try WebFetch; 'skip' records the offer as skipped with
- a needs-playwright marker and spawns no worker; 'browser-extract'
- renders the page with browser-extract.mjs (headless Chromium,
- zero model tokens, needs 'npx playwright install chromium') and
- hands the recovered text to the worker
- --rate-limit-sleep N Maximum adaptive retry delay in seconds; 0 pauses immediately
- (default: 300; range: 0โ2147483647; claude only)
- --status Show batch progress and a per-job table, then exit
- --summary With --status, print the totals only (the table is one row per offer)
- --full With --status, print the per-job table (the default)
- --watch Live-refresh progress (always with the per-job table) until the run completes
- -h, --help Show this help
- Files:
- batch-input.tsv Input offers (id, url, source, notes)
- batch-state.tsv Processing state (auto-managed)
- batch-prompt.md Prompt template for workers
- logs/ Per-offer logs
- tracker-additions/ Tracker lines for post-batch merge
- With a data root outside the checkout (CAREER_OPS_ROOT, CAREER_OPS_DATA_DIR
- or a .career-ops-data marker), config/profile.yml, reports/ and
- batch/tracker-additions/ are read and written there. The other files stay
- beside this script. CAREER_OPS_PROFILE, when set, names the profile read for
- spend_tier and injected into the prompt, as it does for the other scripts.
- Examples:
- # Dry run to see pending offers
- ./batch-runner.sh --dry-run
- # Process all pending
- ./batch-runner.sh
- # Retry only failed offers
- ./batch-runner.sh --retry-failed
- # Process 2 at a time starting from ID 10 (claude only)
- ./batch-runner.sh --parallel 2 --start-from 10
- # Local LLM via OpenCode (free, runs sequentially)
- ./batch-runner.sh --cli opencode --model qwen2.5:32b
- USAGE
- }
- # Parse arguments
- while [[ $# -gt 0 ]]; do
- case "$1" in
- --cli) CLI="$2"; shift 2 ;;
- --parallel) PARALLEL="$2"; shift 2 ;;
- --dry-run) DRY_RUN=true; shift ;;
- --retry-failed) RETRY_FAILED=true; shift ;;
- --resume-paused) RESUME_PAUSED=true; shift ;;
- --start-from) START_FROM="$2"; shift 2 ;;
- --company)
- [[ $# -ge 2 && -n "$2" && "$2" != -* ]] || { echo "ERROR: --company requires an argument"; exit 1; }
- SCOPE_COMPANY="$2"; shift 2 ;;
- --limit)
- if [[ $# -lt 2 ]] || ! [[ "$2" =~ ^[0-9]+$ ]] || [[ "$2" =~ ^0+$ ]]; then
- echo "ERROR: --limit must be a positive integer; omit --limit for no limit."
- exit 1
- fi
- # Remove leading zeroes before Bash arithmetic (which otherwise reads octal).
- LIMIT="${2#"${2%%[!0]*}"}"
- # Compare decimal strings before arithmetic can wrap an oversized integer.
- if [[ ${#LIMIT} -gt 19 ]] || { [[ ${#LIMIT} -eq 19 ]] && [[ "$LIMIT" > 9223372036854775807 ]]; }; then
- echo "ERROR: --limit must be at most 9223372036854775807; omit --limit for no limit."
- exit 1
- fi
- shift 2
- ;;
- --max-retries) MAX_RETRIES="$2"; shift 2 ;;
- --min-score) MIN_SCORE="$2"; shift 2 ;;
- --skip-pdf) SKIP_PDF=true; shift ;;
- --mode)
- [[ $# -ge 2 ]] || { echo "ERROR: --mode requires an argument (full or match)"; exit 1; }
- MODE="$2"
- if [[ "$MODE" == "match" || "$MODE" == "single-pass" ]]; then
- MODE="match"
- fi
- shift 2
- ;;
- --single-pass)
- MODE="match"
- shift
- ;;
- --rate-limit-sleep)
- [[ $# -ge 2 ]] || { echo "ERROR: --rate-limit-sleep requires an argument"; exit 1; }
- RATE_LIMIT_SLEEP="$2"
- shift 2
- ;;
- --model) MODEL="$2"; shift 2 ;;
- --thin-action)
- [[ $# -ge 2 ]] || { echo "ERROR: --thin-action requires an argument (skip, webfetch or browser-extract)"; exit 1; }
- case "$2" in
- skip|webfetch|browser-extract) THIN_ACTION="$2" ;;
- *) echo "ERROR: --thin-action must be 'skip', 'webfetch' or 'browser-extract' (got '$2')."; exit 1 ;;
- esac
- shift 2
- ;;
- --worker-timeout)
- if [[ $# -lt 2 ]] || ! [[ "$2" =~ ^[0-9]+$ ]]; then
- echo "ERROR: --worker-timeout must be a non-negative integer (seconds)."
- exit 1
- fi
- # Strip leading zeroes, then compare as a decimal string: Bash arithmetic
- # wraps past 2^63-1, and a wrapped value at or below 0 disables the watchdog.
- WORKER_TIMEOUT="${2#"${2%%[!0]*}"}"
- WORKER_TIMEOUT="${WORKER_TIMEOUT:-0}"
- if [[ ${#WORKER_TIMEOUT} -gt 19 ]] || { [[ ${#WORKER_TIMEOUT} -eq 19 ]] && [[ "$WORKER_TIMEOUT" > 9223372036854775807 ]]; }; then
- echo "ERROR: --worker-timeout must be at most 9223372036854775807 seconds."
- exit 1
- fi
- shift 2
- ;;
- --status) STATUS_ONLY=true; shift ;;
- --summary) STATUS_FULL=false; shift ;;
- --full) STATUS_FULL=true; shift ;;
- --watch) WATCH_MODE=true; shift ;;
- -h|--help) usage; exit 0 ;;
- *) echo "Unknown option: $1"; usage; exit 1 ;;
- esac
- done
- if [[ "$MODE" != "full" && "$MODE" != "match" ]]; then
- echo "ERROR: Unknown --mode '$MODE'. Supported: full, match."
- exit 1
- fi
- MATCH_OUTPUT_LANGUAGE=""
- if [[ "$MODE" == "match" ]]; then
- PROMPT_FILE="$BATCH_DIR/batch-prompt-match.md"
- SKIP_PDF=true
- # #5297: a match worker never sees the user profile, so targeting and
- # compensation cannot move the score. language.output is the one setting it
- # gets, resolved here once per run with the same reader the evaluators use.
- if [[ -f "$PROFILE_FILE" ]]; then
- MATCH_OUTPUT_LANGUAGE="$(node --input-type=module -e '
- import { readFileSync } from "node:fs";
- import { pathToFileURL } from "node:url";
- const { parseOutputLanguage } = await import(pathToFileURL(process.argv[1]).href);
- process.stdout.write(parseOutputLanguage(readFileSync(process.argv[2], "utf-8")));
- ' "$PROJECT_DIR/profile-language.mjs" "$PROFILE_FILE" 2>/dev/null)" || {
- MATCH_OUTPUT_LANGUAGE=""
- echo "WARN: could not read language.output from $PROFILE_FILE; match workers will write in en." >&2
- }
- fi
- [[ -n "$MATCH_OUTPUT_LANGUAGE" ]] || MATCH_OUTPUT_LANGUAGE="en"
- fi
- if [[ -z "$PARALLEL" ]]; then
- if [[ "$CLI" == "claude" ]]; then PARALLEL=$DEFAULT_CLAUDE_PARALLEL; else PARALLEL=1; fi
- fi
- if ! [[ "$RATE_LIMIT_SLEEP" =~ ^[0-9]+$ ]]; then
- echo "ERROR: --rate-limit-sleep must be a non-negative integer (seconds)."
- exit 1
- fi
- # Normalize decimal input before shell arithmetic (08 must not mean octal),
- # and bound it before conversion so oversized input cannot wrap around.
- RATE_LIMIT_SLEEP=$(printf '%s' "$RATE_LIMIT_SLEEP" | sed 's/^0*//')
- RATE_LIMIT_SLEEP=${RATE_LIMIT_SLEEP:-0}
- if [[ ${#RATE_LIMIT_SLEEP} -gt 10 ]] || { [[ ${#RATE_LIMIT_SLEEP} -eq 10 ]] && [[ "$RATE_LIMIT_SLEEP" > 2147483647 ]]; }; then
- echo "ERROR: --rate-limit-sleep must not exceed 2147483647 seconds."
- exit 1
- fi
- if ! is_decimal_number "$MIN_SCORE"; then
- echo "ERROR: --min-score must be a non-negative number."
- exit 1
- fi
- # Lock file to prevent double execution
- # noclobber makes the create atomic, so two runners started together cannot
- # both own the lock (and both stop the shared PDF painter on exit).
- acquire_lock() {
- if ! (set -C; printf '%s\n' "$MAIN_PID" > "$LOCK_FILE") 2>/dev/null; then
- local old_pid
- old_pid=$(cat "$LOCK_FILE" 2>/dev/null || true)
- if [[ -n "$old_pid" ]] && kill -0 "$old_pid" 2>/dev/null; then
- echo "ERROR: Another batch-runner is already running (PID $old_pid)"
- echo "If this is stale, remove $LOCK_FILE"
- exit 1
- fi
- echo "WARN: Stale lock file found (PID $old_pid not running). Removing."
- rm -f "$LOCK_FILE"
- if ! (set -C; printf '%s\n' "$MAIN_PID" > "$LOCK_FILE") 2>/dev/null; then
- echo "ERROR: Another batch-runner took the lock first"
- exit 1
- fi
- fi
- }
- release_lock() {
- if [[ "${BASHPID:-$$}" != "$MAIN_PID" ]]; then
- return
- fi
- # Stop the painter this run started. A copied runner in a fixture has no
- # lib/pdf-server.mjs, and a run that never painted has no pid file. Either
- # miss must not fail the batch.
- if [[ "${PDF_SERVER_OWNER:-}" == "1" ]]; then
- PDF_SERVER_MODULE="$PROJECT_DIR/lib/pdf-server.mjs" PDF_SERVER_ROOT="$PROJECT_DIR" \
- node --input-type=module -e 'import { pathToFileURL } from "node:url"; const m = await import(pathToFileURL(process.env.PDF_SERVER_MODULE).href); await m.stopSharedPdfServer(process.env.PDF_SERVER_ROOT);' \
- >/dev/null 2>&1 || true
- fi
- rm -f "$LOCK_FILE"
- }
- trap release_lock EXIT
- # Validate prerequisites
- check_prerequisites() {
- if [[ ! -f "$INPUT_FILE" ]]; then
- echo "ERROR: $INPUT_FILE not found. Add offers first."
- exit 1
- fi
- if [[ ! -f "$PROMPT_FILE" ]]; then
- echo "ERROR: $PROMPT_FILE not found."
- exit 1
- fi
- if [[ "$THIN_ACTION" == "browser-extract" && ! -f "$PROJECT_DIR/browser-extract.mjs" ]]; then
- echo "ERROR: --thin-action browser-extract needs browser-extract.mjs (and 'npx playwright install chromium')."
- exit 1
- fi
- # Resolve the binary the configured CLI actually needs. opencode can run
- # natively or fall back to ollama, so check for either.
- local cli_cmd
- case "$CLI" in
- claude) cli_cmd="claude" ;;
- opencode) command -v opencode &>/dev/null && cli_cmd="opencode" || cli_cmd="ollama" ;;
- gemini) cli_cmd="gemini" ;;
- qwen) cli_cmd="qwen" ;;
- grok) cli_cmd="grok" ;;
- *) echo "ERROR: Unknown --cli '$CLI'. Supported: claude, opencode, gemini, qwen, grok"; exit 1 ;;
- esac
- if ! command -v "$cli_cmd" &>/dev/null; then
- local level="ERROR"
- [[ "$DRY_RUN" == "true" ]] && level="WARN"
- echo "$level: '$cli_cmd' not found in PATH (required for --cli $CLI)."
- if [[ "$CLI" == "opencode" ]]; then
- echo " Install opencode (https://opencode.ai) or Ollama (https://ollama.ai) with an opencode model."
- fi
- [[ "$DRY_RUN" == "true" ]] || exit 1
- fi
- # Parallelism, the rate-limit retry loop, and --strict-mcp-config are
- # claude-only; local models run one at a time.
- if [[ "$CLI" != "claude" && "$PARALLEL" -gt 1 ]]; then
- echo "WARN: --parallel >1 is not supported for --cli $CLI (local models run sequentially). Resetting to 1."
- PARALLEL=1
- fi
- mkdir -p "$LOGS_DIR" "$TRACKER_DIR" "$REPORTS_DIR"
- }
- # Status/watch mode only needs prior batch state, not worker prerequisites.
- check_status_prerequisites() {
- if [[ ! -f "$STATE_FILE" ]]; then
- echo "No state file found at $STATE_FILE"
- exit 0
- fi
- }
- # Initialize state file if it doesn't exist
- init_state() {
- if [[ ! -f "$STATE_FILE" ]]; then
- printf 'id\turl\tstatus\tstarted_at\tcompleted_at\treport_num\tscore\terror\tretries\n' > "$STATE_FILE"
- fi
- }
- # A lock held longer than this is assumed abandoned, independent of the
- # PID-liveness check below. This exists because `kill -0 $pid` is unreliable
- # on Git Bash/Windows (MSYS PIDs from $!/$BASHPID don't reliably map to real
- # Windows process IDs), so a genuinely-dead lock holder can otherwise never
- # be recovered there and every other worker times out waiting for it.
- STATE_LOCK_STALE_AGE_SECONDS=15
- acquire_state_lock() {
- if [[ "${STATE_LOCK_DISABLED:-0}" -eq 1 ]]; then
- return 0
- fi
- local waited=0
- local owner_file=""
- local create_misses=0
- local max_waits=$((STATE_LOCK_TIMEOUT_SECONDS * 10))
- while true; do
- if mkdir "$STATE_LOCK_DIR" 2>/dev/null; then
- # A unique metadata path binds cleanup to this acquisition, not its successor.
- if owner_file=$(mktemp "$STATE_LOCK_DIR/owner.XXXXXXXXXXXX") &&
- printf '%s\t%s\n' "${BASHPID:-$$}" "$(date +%s)" > "$owner_file"; then
- # mkdir may have belonged to a generation reclaimed before mktemp.
- # Register first, then decline if another owner is still registered.
- local competing_owner=false registered
- for registered in "$STATE_LOCK_PID_FILE" "$STATE_LOCK_DIR"/owner.*; do
- if [[ -f "$registered" && "$registered" != "$owner_file" ]]; then
- competing_owner=true
- break
- fi
- done
- if [[ "$competing_owner" == "true" ]]; then
- rm -f "$owner_file" 2>/dev/null || true
- rmdir "$STATE_LOCK_DIR" 2>/dev/null || true
- continue
- fi
- STATE_LOCK_OWNED_FILE="$owner_file"
- STATE_LOCK_OWNED=1
- return 0
- fi
- if [[ -n "$owner_file" ]]; then rm -f "$owner_file" 2>/dev/null || true; fi
- rmdir "$STATE_LOCK_DIR" 2>/dev/null || true
- echo "ERROR: Failed to initialize state lock metadata at $STATE_LOCK_DIR" >&2
- return 1
- fi
- if [[ ! -d "$STATE_LOCK_DIR" ]]; then
- # mkdir can fail because the lock exists and the holder can release it
- # before this check (a wide window on Windows/Git Bash): retry mkdir a few
- # times before treating the directory as impossible to create.
- if (( create_misses < 3 )); then
- create_misses=$((create_misses + 1))
- continue
- fi
- if (( PARALLEL <= 1 )); then
- echo "WARN: State lock creation failed. Falling back to lock-free operation (single-worker mode)." >&2
- STATE_LOCK_DISABLED=1
- STATE_LOCK_OWNED=0
- return 0
- fi
- echo "ERROR: Failed to create state lock directory $STATE_LOCK_DIR" >&2
- return 1
- fi
- create_misses=0
- local lock_file="" candidate
- # Accept a dead lock left by the older fixed-name protocol as well.
- for candidate in "$STATE_LOCK_PID_FILE" "$STATE_LOCK_DIR"/owner.*; do
- if [[ -f "$candidate" ]]; then lock_file="$candidate"; break; fi
- done
- if [[ -n "$lock_file" ]]; then
- local lock_pid lock_epoch
- lock_pid=$(cut -f1 "$lock_file" 2>/dev/null || true)
- lock_epoch=$(cut -f2 "$lock_file" 2>/dev/null || true)
- local stale=false
- local stale_reason=""
- # SAFETY INVARIANT: never treat the lock as stale while kill -0
- # positively confirms the recorded PID is still running โ a
- # confirmed-alive owner may still write update_state_unlocked's
- # rewrite of STATE_FILE, and reclaiming under it would let two
- # processes rewrite $STATE_FILE.tmp concurrently (real data loss).
- # The age-based fallback below only ever fires when the PID check
- # could NOT confirm liveness (empty/missing PID, or kill -0 itself
- # reported not-running) โ it narrows, but does not replace, the PID
- # check. This intentionally leaves one Windows/Git-Bash edge case
- # unhandled: a `kill -0` FALSE POSITIVE (reports alive for a PID
- # that Windows has actually reused for an unrelated process). That
- # gap is accepted because the alternative โ reclaiming while any
- # chance remains the owner is genuinely alive โ risks silent
- # concurrent-write corruption, which is worse than this lock
- # occasionally timing out (recoverable via retry) in that rare case.
- local pid_confirmed_alive=false
- if [[ -n "$lock_pid" ]] && kill -0 "$lock_pid" 2>/dev/null; then
- pid_confirmed_alive=true
- fi
- if [[ "$pid_confirmed_alive" == "false" ]]; then
- if [[ -n "$lock_pid" ]]; then
- stale=true
- stale_reason="PID $lock_pid not running"
- elif [[ "$lock_epoch" =~ ^[0-9]+$ ]]; then
- local now age
- now=$(date +%s)
- age=$((now - lock_epoch))
- if (( age >= STATE_LOCK_STALE_AGE_SECONDS )); then
- stale=true
- stale_reason="lock age ${age}s >= ${STATE_LOCK_STALE_AGE_SECONDS}s (no PID recorded to check liveness against)"
- fi
- fi
- fi
- if [[ "$stale" == "true" ]]; then
- rm -f "$lock_file"
- if rmdir "$STATE_LOCK_DIR" 2>/dev/null; then
- echo "WARN: Recovered stale state lock ($stale_reason)." >&2
- continue
- fi
- fi
- fi
- if (( waited >= max_waits )); then
- echo "ERROR: Timed out waiting for state lock at $STATE_LOCK_DIR" >&2
- echo "If no batch-runner worker is active, remove the stale lock directory." >&2
- return 1
- fi
- sleep 0.1
- ((waited += 1))
- done
- }
- release_state_lock() {
- if [[ "${STATE_LOCK_OWNED:-0}" -ne 1 ]]; then
- return
- fi
- rm -f "$STATE_LOCK_OWNED_FILE" 2>/dev/null || true
- rmdir "$STATE_LOCK_DIR" 2>/dev/null || true
- STATE_LOCK_OWNED=0
- }
- run_with_state_lock() {
- if [[ "$STATE_LOCK_FLOCK" == "1" ]]; then
- run_with_state_flock "$@" || return $?
- return 0
- fi
- acquire_state_lock || return $?
- local status=0
- if "$@"; then
- status=0
- else
- status=$?
- fi
- release_state_lock
- return "$status"
- }
- # The mkdir lock is only as atomic as mkdir(1). On Ubuntu 26.04 mkdir is uutils
- # coreutils, and before uutils 0.11.0 it checks that the directory is absent,
- # then creates it, and exits 0 when a racing mkdir made it in between. Two
- # workers both took the lock, reserved the same report number and rewrote
- # batch-state.tsv together, with nothing printed (#6362). flock(1) leaves the
- # excluding to the kernel. main() calls this once, before any worker starts,
- # so every worker takes the same lock. Without flock (stock macOS, Git Bash)
- # mkdir(1) is atomic and the mkdir lock stays, as it does on a file system
- # that refuses flock.
- select_state_lock() {
- command -v flock >/dev/null 2>&1 || return 0
- if { flock -x -n 9; } 2>/dev/null 9>>"$STATE_LOCK_FLOCK_FILE"; then
- STATE_LOCK_FLOCK=1
- else
- echo "WARN: flock cannot lock $STATE_LOCK_FLOCK_FILE; using the mkdir state lock." >&2
- fi
- }
- # Polls every 0.1s up to STATE_LOCK_TIMEOUT_SECONDS, like the mkdir lock
- # (BusyBox flock has no -w). The lock lives on fd 9, open only inside this
- # function, so the workers, their CLI and the PDF painter never inherit it. A
- # program started inside the critical section does inherit it, and holds the
- # lock until it exits (9>&- would not stop that: bash saves fd 9 to another
- # descriptor that the program still gets), so start nothing long-lived there.
- # The lock file is never removed: a worker waiting on the old file and one
- # locking a new one would both get in.
- run_with_state_flock() {
- local status=1 waited=0
- local max_waits=$((STATE_LOCK_TIMEOUT_SECONDS * 10))
- {
- while true; do
- if flock -x -n 9; then
- status=0
- "$@" || status=$?
- break
- fi
- if (( waited >= max_waits )); then
- echo "ERROR: Timed out waiting for state lock at $STATE_LOCK_FLOCK_FILE" >&2
- break
- fi
- sleep 0.1
- waited=$((waited + 1))
- done
- } 9>>"$STATE_LOCK_FLOCK_FILE"
- return "$status"
- }
- # Get status of an offer from state file
- get_status() {
- local id="$1"
- if [[ ! -f "$STATE_FILE" ]]; then
- echo "none"
- return
- fi
- local status
- status=$(awk -F'\t' -v id="$id" '$1 == id { print $3 }' "$STATE_FILE")
- echo "${status:-none}"
- }
- # Get retry count for an offer
- get_retries() {
- local id="$1"
- if [[ ! -f "$STATE_FILE" ]]; then
- echo "0"
- return
- fi
- local retries
- retries=$(awk -F'\t' -v id="$id" '$1 == id { print $9 }' "$STATE_FILE")
- echo "${retries:-0}"
- }
- # Read spend_tier from $PROFILE_FILE (config/profile.yml, or the file
- # CAREER_OPS_PROFILE names). Defaults to "standard" if the key is absent or
- # invalid.
- read_spend_tier() {
- local raw=""
- if [[ -f "$PROFILE_FILE" ]]; then
- raw=$(
- awk -F: '
- /^[[:space:]]*spend_tier[[:space:]]*:/ {
- value = substr($0, index($0, ":") + 1)
- print value
- exit
- }
- ' "$PROFILE_FILE"
- )
- raw="${raw%%#*}"
- raw="${raw//$'\r'/}"
- raw="${raw#"${raw%%[![:space:]]*}"}"
- raw="${raw%"${raw##*[![:space:]]}"}"
- case "$raw" in
- \"*\") raw="${raw#\"}"; raw="${raw%\"}" ;;
- \'*\') raw="${raw#\'}"; raw="${raw%\'}" ;;
- esac
- raw="$(printf '%s' "$raw" | tr '[:upper:]' '[:lower:]')"
- fi
- case "$raw" in
- economy|standard|premium)
- printf '%s\n' "$raw"
- ;;
- "")
- printf '%s\n' "standard"
- ;;
- *)
- echo "WARN: Invalid spend_tier \"$raw\" in ${PROFILE_FILE#"$DATA_ROOT/"}; falling back to standard." >&2
- printf '%s\n' "standard"
- ;;
- esac
- }
- # Tier -> model mapping. Keep in sync with the table in modes/_shared.md.
- spend_tier_to_model() {
- case "$1" in
- economy) echo "claude-haiku-5-5" ;;
- premium) echo "claude-opus-5-5" ;;
- standard|*) echo "claude-sonnet-5-5" ;;
- esac
- }
- # Run one worker CLI with stdin closed, so a CLI that waits for input reads EOF
- # instead of hanging on the runner's terminal. With WORKER_TIMEOUT > 0 a
- # watchdog sends TERM at the limit and KILL 10 s later, and the worker exits
- # 124. Background jobs ignore SIGINT in a non-interactive shell, so Ctrl+C is
- # forwarded to the worker by hand.
- run_worker() {
- "$@" </dev/null &
- local pid=$!
- local watchdog=""
- if (( WORKER_TIMEOUT > 0 )); then
- (
- for (( s = 0; s < WORKER_TIMEOUT; s++ )); do
- sleep 1
- kill -0 "$pid" 2>/dev/null || exit 0
- done
- kill -TERM "$pid" 2>/dev/null
- for (( s = 0; s < 10; s++ )); do
- sleep 1
- kill -0 "$pid" 2>/dev/null || exit 124
- done
- kill -KILL "$pid" 2>/dev/null
- exit 124
- ) &
- watchdog=$!
- fi
- trap 'kill -TERM "$pid" 2>/dev/null; exit 130' INT TERM
- local rc=0
- wait "$pid" || rc=$?
- trap - INT TERM
- if [[ -n "$watchdog" ]]; then
- local watchdog_rc=0
- wait "$watchdog" || watchdog_rc=$?
- if (( watchdog_rc == 124 )); then
- echo "batch-runner: worker timed out after ${WORKER_TIMEOUT}s"
- rc=124
- fi
- fi
- return "$rc"
- }
- # Resolve the model to pass to the worker CLI. --model always wins.
- # spend_tier maps to Claude model names, so it only applies to --cli claude;
- # other CLIs use --model verbatim, or their own default when it is unset.
- resolve_worker_model() {
- if [[ -n "$MODEL" ]]; then
- RESOLVED_MODEL="$MODEL"
- RESOLVED_SPEND_TIER="override"
- return 0
- fi
- if [[ "$CLI" != "claude" ]]; then
- RESOLVED_MODEL=""
- RESOLVED_SPEND_TIER="cli-default"
- return 0
- fi
- RESOLVED_SPEND_TIER="$(read_spend_tier)"
- RESOLVED_MODEL="$(spend_tier_to_model "$RESOLVED_SPEND_TIER")"
- }
- # Append a one-line, auditable record of a pre-screen-gate discard to
- # batch/logs/discard.log (see modes/batch.md โ Pre-screen gate). Format:
- # {ISO8601 timestamp}\t{job id}\t{url}\t{reason}
- #
- # Callers write the "skipped" state first and call this only when that state
- # (or its recovery record) was written, so the same id, url and reason are on
- # disk. A failed append is therefore reported, not fatal: the record can be
- # rebuilt from state instead of aborting the run.
- log_discard() {
- local id="$1" url="$2" reason="$3"
- local ts
- ts="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
- if ! { mkdir -p "$LOGS_DIR" && printf '%s\t%s\t%s\t%s\n' "$ts" "$id" "$url" "$reason" >> "$DISCARD_LOG"; } 2>/dev/null; then
- echo " โ ๏ธ Could not append to $DISCARD_LOG for #$id; its skipped state record keeps the reason" >&2
- fi
- }
- # The mode a plain redirect gives a new file (0644 under umask 022). mktemp
- # creates 0600, and the state file keeps the mode it has always had.
- STATE_FILE_MODE=$(printf '%o' "$(( 0666 & ~0$(umask) ))")
- # Update or insert state for an offer.
- # Caller must hold the state lock (run_with_state_lock) while this runs.
- update_state_unlocked() {
- local id="$1" url="$2" status="$3" started="$4" completed="$5" report_num="$6" score="$7" error="$8" retries="$9"
- # batch-state.tsv is tab-separated with one row per line -- a literal tab,
- # newline, or carriage return inside $error (e.g. from a worker's raw error
- # text, or JSON.parse unescaping \n/\r/\t in a caller upstream) would split
- # into extra columns or extra rows and corrupt every row after it. Collapse
- # them to spaces centrally here so every caller is protected, not just the
- # one that happened to trigger this.
- error=${error//$'\r'/ }
- error=${error//$'\n'/ }
- error=${error//$'\t'/ }
- if [[ ! -f "$STATE_FILE" ]]; then
- init_state
- fi
- # A tmp file of its own per rewrite, beside the state file so mv stays a
- # rename. With one shared $STATE_FILE.tmp, two rewrites that overlapped
- # truncated and renamed each other's file: rows doubled and mv failed (#6362).
- local tmp
- tmp=$(mktemp "$STATE_FILE.tmp.XXXXXX") || return 1
- chmod "$STATE_FILE_MODE" "$tmp" 2>/dev/null || true
- local found=false
- # Write header
- head -1 "$STATE_FILE" > "$tmp"
- # Process existing lines. Don't split a row with `IFS=$'\t' read`: tab is IFS
- # whitespace, so the two tabs around an empty cell (an empty error from a
- # worker that died without output, or an empty completed_at/score written by
- # reconcile_recovery_records) collapse into one and every later cell shifts
- # left -- `retries` became the error text and the retry counter was lost.
- # Read the line raw and split on \x1f after translating the tabs, as
- # reconcile_recovery_records does.
- while IFS= read -r sline; do
- IFS=$'\x1f' read -r sid surl sstatus sstarted scompleted sreport sscore serror sretries <<< "${sline//$'\t'/$'\x1f'}"
- [[ "$sid" == "id" ]] && continue # skip header
- if [[ "$sid" == "$id" ]]; then
- printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n' \
- "$id" "$url" "$status" "$started" "$completed" "$report_num" "$score" "$error" "$retries" >> "$tmp"
- found=true
- else
- printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n' \
- "$sid" "$surl" "$sstatus" "$sstarted" "$scompleted" "$sreport" "$sscore" "$serror" "$sretries" >> "$tmp"
- fi
- done < "$STATE_FILE"
- if [[ "$found" == "false" ]]; then
- printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n' \
- "$id" "$url" "$status" "$started" "$completed" "$report_num" "$score" "$error" "$retries" >> "$tmp"
- fi
- if ! mv -f "$tmp" "$STATE_FILE"; then
- rm -f "$tmp"
- return 1
- fi
- }
- update_state() {
- run_with_state_lock update_state_unlocked "$@"
- }
- # Durable last-resort records of state transitions that could NOT be written
- # into $STATE_FILE (state-lock exhausted its retries). ONE FILE PER RECORD:
- # each failed transition gets its own uniquely-named file via mktemp
- # (O_CREAT|O_EXCL โ atomic creation, guaranteed-unique name), so no two
- # workers ever write to the same file and no shared-file truncate/append
- # race can exist on any platform. That matters here because recovery writes
- # are CORRELATED, not independent: they fire exactly when the state lock is
- # jammed, which makes all parallel workers fail (and try to record) at the
- # same moment โ a shared recovery file is racing precisely when it is
- # needed most (PR #2417 review). This mechanism must also never depend on
- # the state lock that just failed, and it doesn't: creation is the only
- # synchronization. reconcile_recovery_records() (start of the next run,
- # single-threaded, before any worker spawns) merges each record into
- # $STATE_FILE and deletes its file only on success โ there is no
- # rewrite-and-rename step, so nothing here needs cross-filesystem atomicity.
- RECOVERY_DIR="$BATCH_DIR/batch-state-recovery.d"
- append_recovery_record() {
- local id="$1" url="$2" status="$3" started="$4" completed="$5" report_num="$6" score="$7" error="$8" retries="$9"
- # Same rationale as update_state_unlocked: a literal tab/newline/CR in
- # $error would split this single-line record into extra fields or rows,
- # corrupting the record and the state row it later reconciles into.
- error=${error//$'\r'/ }
- error=${error//$'\n'/ }
- error=${error//$'\t'/ }
- # \x1f is used as the in-memory delimiter when this record is read back
- # (see reconcile_recovery_records); strip it here too so a stray unit
- # separator in $error can't inject a field boundary on the read side.
- error=${error//$'\x1f'/ }
- mkdir -p "$RECOVERY_DIR" || return 1
- local rec
- rec=$(mktemp "$RECOVERY_DIR/rec-XXXXXX") || return 1
- printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n' \
- "$id" "$url" "$status" "$started" "$completed" "$report_num" "$score" "$error" "$retries" > "$rec"
- }
- # A recovery record is ALWAYS older than whatever $STATE_FILE holds for the
- # same id: the record is only written when the lock was unreachable, so any
- # row present for that id now was written afterwards by a path that did reach
- # the lock. Merging it blindly therefore rolls a finished offer backwards --
- # the score and completed_at are overwritten, and since rate_limited and
- # failed are NOT terminal, the offer re-enters pending selection in main()
- # and gets evaluated a second time. Cost: one lost evaluation plus one
- # duplicate run, on a path that by construction only fires when something
- # already went wrong.
- #
- # Completed and human-held states must not be rolled back by stale records.
- # Keep these protected states in sync with the pending-selection guard.
- recovery_record_is_superseded() {
- local current="$1"
- [[ "$current" == "completed" || "$current" == "skipped" || "$current" == "needs_confirmation" ]]
- }
- # Merge one recovery record, but only into a row that has not already reached
- # a completed or human-held state. The status read happens INSIDE the lock: a
- # check-then-write gap would let a worker finish between the two and
- # reintroduce the same rollback. get_status is a lock-free reader (plain awk
- # over $STATE_FILE), so calling it here does not re-enter the non-reentrant
- # mkdir lock.
- #
- # Exit codes: 0 merged ยท 3 superseded (record is stale, caller should drop
- # it) ยท anything else is a real failure. 3 avoids colliding with the 1 that
- # acquire_state_lock returns when the lock itself is unreachable.
- reconcile_one_unlocked() {
- local id="$1"
- local current
- current=$(get_status "$id")
- if recovery_record_is_superseded "$current"; then
- echo " Superseded: offer id=$id already '$current' in state โ discarding stale recovery record (would have rolled it back to '$3')."
- return 3
- fi
- update_state_unlocked "$@"
- }
- reconcile_one() {
- run_with_state_lock reconcile_one_unlocked "$@"
- }
- # Merge any recovery records left by a prior run into $STATE_FILE. Runs
- # single-threaded at the very start of main(), before any worker is spawned,
- # so there is no lock contention here โ this is the one place these records
- # are guaranteed a clean shot at the lock. Each record file is deleted only
- # after its transition lands in $STATE_FILE (or is found to be superseded);
- # genuine failures leave the file in place for the run after that.
- reconcile_recovery_records() {
- [[ -d "$RECOVERY_DIR" ]] || return 0
- local -a rec_files=()
- local f
- for f in "$RECOVERY_DIR"/rec-*; do
- [[ -f "$f" ]] || continue
- rec_files+=("$f")
- done
- if (( ${#rec_files[@]} == 0 )); then
- rmdir "$RECOVERY_DIR" 2>/dev/null || true
- return 0
- fi
- echo "=== Reconciling ${#rec_files[@]} recovery record(s) from a prior interrupted run ==="
- local merged=0 superseded=0 still_failed=0
- local rid rurl rstatus rstarted rcompleted rreport rscore rerror rretries
- local rline
- local rc
- for f in "${rec_files[@]}"; do
- rline=""
- # Don't split with `IFS=$'\t' read`: tab is IFS *whitespace*, so a run of
- # tabs around an empty interior field (e.g. an empty completed_at on a
- # non-terminal record) collapses to one delimiter and every later field
- # shifts left, corrupting the merge. Read the line raw, then split on \x1f
- # (a non-whitespace unit separator that preserves empty fields) after
- # translating the on-disk tabs to it. Same rationale as process_offer.
- IFS= read -r rline < "$f" || true
- IFS=$'\x1f' read -r rid rurl rstatus rstarted rcompleted rreport rscore rerror rretries <<< "${rline//$'\t'/$'\x1f'}"
- if [[ -z "$rid" ]]; then
- echo " WARN: discarding unreadable recovery record $f" >&2
- rm -f "$f"
- continue
- fi
- rc=0
- reconcile_one "$rid" "$rurl" "$rstatus" "$rstarted" "$rcompleted" "$rreport" "$rscore" "$rerror" "$rretries" || rc=$?
- if (( rc == 0 )); then
- rm -f "$f"
- merged=$((merged + 1))
- elif (( rc == 3 )); then
- # Stale by definition, not a failure โ the row it targets already
- # finished. Dropping the file is correct; keeping it would retry the
- # same rollback on every subsequent run.
- rm -f "$f"
- superseded=$((superseded + 1))
- else
- still_failed=$((still_failed + 1))
- fi
- done
- echo " Merged: $merged | Superseded: $superseded | Still unrecovered: $still_failed"
- if (( still_failed > 0 )); then
- echo " WARN: $still_failed record(s) could not be merged even single-threaded โ check $STATE_LOCK_DIR for a genuinely stuck lock. Unmerged records remain in $RECOVERY_DIR." >&2
- else
- rmdir "$RECOVERY_DIR" 2>/dev/null || true
- fi
- }
- # Retry wrapper around update_state. Bare `update_state ...` calls under
- # `set -e` will silently kill the entire background worker subshell if a
- # single lock-timeout failure propagates โ this wrapper retries a few times
- # and, if it still fails, falls back to append_recovery_record so the
- # transition is never actually lost (only delayed until the next run's
- # reconcile step), then logs a clear warning and returns non-zero so the
- # CALLER can still decide whether to skip side effects that assumed success.
- # Return 1 means the transition is durable in recovery; 2 means neither write
- # succeeded, so a human gate must stop and surface the persistence failure
- # (found under --parallel 5 on Git Bash/Windows: ~47 of 50 jobs silently
- # dropped in one run from exactly this before the retry+recovery-log fix).
- update_state_retrying() {
- local attempt=0
- local max_attempts=3
- while (( attempt < max_attempts )); do
- if update_state "$@"; then
- return 0
- fi
- attempt=$((attempt + 1))
- if (( attempt < max_attempts )); then
- echo " โ ๏ธ State update failed (attempt $attempt/$max_attempts), retrying in 2s..." >&2
- sleep 2
- fi
- done
- if append_recovery_record "$@"; then
- echo " โ ๏ธ State update failed after $max_attempts attempts โ offer id=$1 status=$3 recorded to $RECOVERY_DIR for reconciliation on next run." >&2
- else
- echo " โ State update failed after $max_attempts attempts AND recovery-record write also failed โ offer id=$1 status=$3 was NOT recorded anywhere. It will be retried as pending next run." >&2
- return 2
- fi
- return 1
- }
- is_rate_limit_log() {
- local log_file="$1"
- grep -Eiq '(rate limit|rate_limit|too many requests|429|quota exceeded|try again later|temporarily unavailable)' "$log_file"
- }
- is_session_limit_log() {
- local log_file="$1"
- grep -Eiq '(session limit|resets [0-9:]+[ap]m|usage limit|limit[[:space:]]+reached)' "$log_file"
- }
- # Print a delay or "pause". Only standalone delta-seconds headers are trusted;
- # never feed worker text into shell arithmetic. awk compares before formatting,
- # so even an arbitrarily long integer header safely requests a pause.
- rate_limit_delay() {
- local log_file="$1" retry="$2" base delay hint step
- if (( RATE_LIMIT_SLEEP == 0 )); then
- echo pause
- return
- fi
- base=30
- (( base <= RATE_LIMIT_SLEEP )) || base=$RATE_LIMIT_SLEEP
- # A log that exists and is empty has no header to find, and awk would print
- # nothing for it: skip the two processes. A missing log still reaches awk.
- hint=""
- [[ -e "$log_file" && ! -s "$log_file" ]] || hint=$(LC_ALL=C awk -v cap="$RATE_LIMIT_SLEEP" '
- tolower($0) ~ /^[[:space:]]*retry-after:[[:space:]]*[0-9]+[[:space:]]*$/ {
- value=$0
- sub(/^[^:]*:[[:space:]]*/, "", value)
- if (value + 0 > cap) over=1
- if (value + 0 > largest) largest=value + 0
- found=1
- }
- END {
- if (over) print "pause"
- else if (found) printf "%.0f\n", largest
- }
- ' "$log_file")
- if [[ "$hint" == pause ]]; then
- echo pause
- return
- fi
- delay=$base
- if [[ -n "$hint" ]]; then
- (( hint <= delay )) || delay=$hint
- else
- # Saturate before doubling, avoiding exponentiation/overflow even on resume.
- for (( step=0; step<retry && delay<RATE_LIMIT_SLEEP; step++ )); do
- if (( delay > RATE_LIMIT_SLEEP / 2 )); then
- delay=$RATE_LIMIT_SLEEP
- else
- delay=$((delay * 2))
- fi
- done
- fi
- # Upward-only jitter never undercuts Retry-After or the base delay.
- delay=$((delay + delay * (RANDOM % 21) / 100))
- (( delay <= RATE_LIMIT_SLEEP )) || delay=$RATE_LIMIT_SLEEP
- echo "$delay"
- }
- mark_paused_rate_limit() {
- local id="$1" url="$2" started_at="$3" report_num="$4" retries="$5" log_file="$6"
- local completed_at
- completed_at=$(date -u +%Y-%m-%dT%H:%M:%SZ)
- local error_msg
- error_msg=$(tail -5 "$log_file" 2>/dev/null | tr '\n' ' ' | cut -c1-200 || echo "session/rate limit reached")
- update_state_retrying "$id" "$url" "paused_rate_limit" "$started_at" "$completed_at" "$report_num" "-" "$error_msg" "$retries" || true
- printf '%s\t%s\t%s\n' "$id" "$report_num" "$error_msg" > "$PAUSE_FILE"
- BATCH_PAUSED=true
- }
- reserve_report_num_unlocked() {
- local id="$1" url="$2" started="$3" retries="$4"
- # Use the shared, cross-process-atomic reservation system (O_CREAT|O_EXCL
- # sentinel files in reserve-report-num.mjs) instead of the old bash-native
- # max(existing report files, batch-state.tsv numbers)+1 scan. The bash-native
- # version had zero visibility into reservations made by any OTHER process
- # calling `node reserve-report-num.mjs` directly -- e.g. an interactively
- # dispatched Agent evaluating one offer with a browser tool while a batch
- # run is in flight. Both could independently compute the same "next" number
- # and collide on disk. Found 2026-07-30: two separate collisions (report
- # 049, report 051) in one batch run for exactly this reason -- routing every
- # caller through the same node script means they all share one real lock.
- local report_num=""
- report_num=$(node "$PROJECT_DIR/reserve-report-num.mjs" 2>/dev/null | tr -d '[:space:]') || report_num=""
- # No number means the allocator failed (lock timeout, unwritable reports/,
- # exhausted retries). Returning success with an empty number let the caller
- # launch a paid worker with a blank "Report number:" and write an empty
- # report_num cell into batch-state.tsv; the retry wrapper below is what
- # handles a failed reservation, so fail here and let it.
- if [[ ! "$report_num" =~ ^[0-9]+$ ]]; then
- echo " โ reserve-report-num.mjs returned no report number for offer id=$id" >&2
- return 1
- fi
- update_state_unlocked "$id" "$url" "processing" "$started" "-" "$report_num" "-" "-" "$retries"
- printf '%s\n' "$report_num"
- }
- # Release a report-number reservation via the shared atomic system. Safe to
- # call even if the number was never actually reserved this way (e.g. a
- # resumed/paused offer) -- the underlying script no-ops on a missing sentinel.
- release_report_num() {
- local report_num="$1"
- [[ -n "$report_num" && "$report_num" != "-" ]] || return 0
- node "$PROJECT_DIR/reserve-report-num.mjs" --release "$report_num" >/dev/null 2>&1 || true
- }
- reserve_report_num() {
- run_with_state_lock reserve_report_num_unlocked "$@"
- }
- # Succeed if reports/ holds a real report for this number. The reservation
- # sentinel ({num}-RESERVED.md) never counts: reserve-report-num.mjs creates it
- # before the worker runs, so counting it would mark a no-op worker completed.
- #
- # A plain glob loop, not `compgen -G`: compgen exists only when bash is built
- # with programmable completion, and some distro builds (NixOS among them) omit
- # it from non-interactive shells. There, compgen fails, the check sees no
- # report, and every worker is recorded as failed even though its report is on
- # disk. merge-tracker.mjs then refuses those reports as possibly fabricated.
- has_report_file() {
- local report_num="$1" f
- for f in "$REPORTS_DIR/${report_num}"-*.md; do
- [[ -e "$f" ]] || continue
- [[ "$f" == *-RESERVED.md ]] && continue
- return 0
- done
- return 1
- }
- # Retry wrapper โ same rationale as update_state_retrying above. A bare
- # `x=$(reserve_report_num ...)` under `set -e` kills the worker subshell
- # silently on a single lock-timeout; this retries and logs instead.
- reserve_report_num_retrying() {
- local attempt=0
- local max_attempts=3
- local result=""
- while (( attempt < max_attempts )); do
- if result=$(reserve_report_num "$@"); then
- printf '%s\n' "$result"
- return 0
- fi
- attempt=$((attempt + 1))
- if (( attempt < max_attempts )); then
- echo " โ ๏ธ Report-number reservation failed (attempt $attempt/$max_attempts), retrying in 2s..." >&2
- sleep 2
- fi
- done
- echo " โ Report-number reservation failed after $max_attempts attempts for offer id=$1 โ leaving it pending for next run." >&2
- return 1
- }
- # Process a single offer
- process_offer() {
- local id="$1" url="$2" source="$3" notes="$4"
- local started_at
- started_at=$(date -u +%Y-%m-%dT%H:%M:%SZ)
- local retries
- retries=$(get_retries "$id")
- local report_num
- if ! report_num=$(reserve_report_num_retrying "$id" "$url" "$started_at" "$retries"); then
- return 1
- fi
- local date
- date=$(date +%Y-%m-%d)
- # Use mktemp instead of a predictable path: a fixed name like
- # /tmp/batch-jd-${id}.txt is guessable, so an attacker on a shared machine
- # could pre-create it as a symlink and redirect or clobber the write.
- # Always project-local, never ${TMPDIR}: the worker CLI has to be able to read
- # this file, and CLIs that auto-reject external_directory (/tmp/*) โ opencode
- # does โ cannot read anything outside the project. Honouring TMPDIR here would
- # re-open that hole whenever TMPDIR is the usual /tmp. mktemp still randomizes
- # the name, so the symlink-guessability property above is unchanged.
- local jd_tmp_dir="$PROJECT_DIR/batch/tmp"
- mkdir -p "$jd_tmp_dir"
- local jd_file
- jd_file="$(mktemp "$jd_tmp_dir/batch-jd-${id}.XXXXXX")"
- # The worker is a native process. Under Git Bash / MSYS the path above is a
- # POSIX one (/tmp/... or /c/...) that a Windows binary cannot open, so every
- # worker read "JD source unavailable" even when curl had filled the file.
- # cygpath -m yields C:/... which both bash and the worker resolve.
- case "$(uname -s)" in
- MINGW*|MSYS*|CYGWIN*) command -v cygpath >/dev/null 2>&1 && jd_file="$(cygpath -m "$jd_file")" ;;
- esac
- # Pre-populate $jd_file with a static curl fetch so the worker reads HTML
- # directly instead of always falling through to WebFetch (#2492). WebFetch is
- # unreliable on JS-rendered boards (Phenom, Workday, iCIMS) because it hits
- # the rendered JS shell rather than the actual JD text. curl returns the raw
- # HTML in a single round-trip; for static boards that is exactly the JD.
- # For JS-rendered boards the file will be thin (JS shell only), which the
- # sufficiency check below detects โ the file is then truncated to 0 bytes so
- # the worker's Step-1 WebFetch fallback fires exactly as designed.
- # If curl is absent or fails, $jd_file stays empty and WebFetch fires too.
- # Minimum visible word count to treat a fetched page as a real JD rather than
- # a JS shell. A JS app shell (Workday, Phenom, iCIMS) has near-zero visible
- # words after HTML stripping; a real JD has hundreds. 80 is a conservative
- # lower bound โ any genuine posting has at least a title, summary, and a few
- # requirements, which together exceed 80 stripped words.
- local prefetch_min_words=80
- local jd_prefetch_words=0
- local jd_rescue_closed_note=""
- local jd_raw_file=""
- local curl_status=0
- if command -v curl >/dev/null 2>&1; then
- # Reject loopback, link-local, and private-network destinations before curl
- # connects. --proto/--proto-redir restrict schemes but not destination IPs,
- # so a malicious offer URL could reach cloud metadata (169.254.169.254) or
- # internal services without this guard.
- local _url_safe
- local current_url="$url"
- local redirect_count=0
- local redirect_headers
- local redirect_location
- while :; do
- _url_safe=$(node -e "
- try {
- const u = new URL(process.argv[1]);
- const h = u.hostname.toLowerCase().replace(/\.\$/, '');
- const blocked =
- h === 'localhost' || h === 'localhost.localdomain' ||
- h.endsWith('.local') || h.endsWith('.internal') ||
- h.includes(':') ||
- /^127\./.test(h) || /^169\.254\./.test(h) ||
- /^10\./.test(h) || /^172\.(1[6-9]|2[0-9]|3[01])\./.test(h) ||
- /^192\.168\./.test(h) || /^0\./.test(h);
- process.stdout.write(blocked ? '0' : '1');
- } catch (e) { process.stdout.write('0'); }
- " "$current_url" 2>/dev/null)
- if [[ "$_url_safe" != "1" ]]; then
- echo " โน๏ธ JD prefetch: blocked โ private/loopback destination ($current_url)"
- : > "$jd_file"
- break
- else
- # Headers are read by curl/bash only โ never by the worker โ so this one
- # stays on ${TMPDIR:-/tmp} and the expression stays self-contained (the
- # prefetch block is extracted and run in isolation by
- # tests/batch-runner-jd-prefetch.test.mjs, where $jd_tmp_dir is not set).
- redirect_headers="$(mktemp "${TMPDIR:-/tmp}/batch-jd-headers.XXXXXX")"
- curl_status=0
- curl --silent --show-error --location --max-redirs 0 \
- --max-time 20 --connect-timeout 5 \
- --fail --compressed \
- --proto '=http,https' --proto-redir 'https,http' --max-filesize 5000000 \
- --user-agent "Mozilla/5.0 (compatible; career-ops/batch)" \
- --header "Accept: text/html,application/xhtml+xml,*/*;q=0.8" \
- --dump-header "$redirect_headers" \
- --output "$jd_file" \
- -- "$current_url" 2>/dev/null || curl_status=$?
- redirect_location=""
- if [[ "$curl_status" -eq 47 ]]; then
- redirect_location="$(awk 'tolower($0) ~ /^location:[[:space:]]*/ { value=$0; sub(/^[^:]*:[[:space:]]*/, "", value) } END { print value }' "$redirect_headers")"
- fi
- rm -f "$redirect_headers"
- if [[ "$curl_status" -eq 47 && -n "$redirect_location" ]]; then
- if [[ "$redirect_count" -ge 10 ]]; then
- : > "$jd_file"
- echo " โน๏ธ JD prefetch: too many redirects โ worker will WebFetch"
- break
- fi
- current_url="$(node -e "
- try { process.stdout.write(new URL(process.argv[2], process.argv[1]).href); }
- catch (e) { process.stdout.write(''); }
- " "$current_url" "$redirect_location" 2>/dev/null)"
- if [[ -z "$current_url" ]]; then
- : > "$jd_file"
- break
- fi
- redirect_count=$((redirect_count + 1))
- continue
- fi
- if [[ "$curl_status" -ne 0 ]]; then
- : > "$jd_file"
- fi
- break
- fi
- done
- # Keep the bytes curl downloaded: the strip below discards <script> blocks,
- # which is where a JS-rendered board keeps its JD (schema.org JobPosting,
- # hydration payload), so jd-rescue.mjs reads this copy, not the stripped text.
- jd_raw_file="${jd_file}.raw"
- cp "$jd_file" "$jd_raw_file" 2>/dev/null || true
- # Strip HTML tags and count visible words to distinguish a real JD (hundreds
- # of words) from a JS shell (near zero visible text).
- jd_prefetch_words=$(node -e "
- const fs = require('fs');
- try {
- const text = fs.readFileSync(process.argv[1], 'utf-8')
- .replace(/<script[\s\S]*?<\/script>/gi, ' ')
- .replace(/<style[\s\S]*?<\/style>/gi, ' ')
- .replace(/<[^>]+>/g, ' ')
- .replace(/&(nbsp|#160|#xa0);/gi, ' ')
- .replace(/\s+/g, ' ')
- .trim();
- fs.writeFileSync(process.argv[1], text);
- process.stdout.write(String(text.split(' ').filter(Boolean).length));
- } catch (e) { process.stdout.write('0'); }
- " "$jd_file" 2>/dev/null) || jd_prefetch_words=0
- # Ensure jd_prefetch_words is always a non-negative integer. A non-integer
- # (e.g. empty string, "NaN") would cause bash arithmetic to fail or
- # miscompare. Strip everything that is not a digit and default to 0.
- jd_prefetch_words="${jd_prefetch_words//[^0-9]/}"
- jd_prefetch_words="${jd_prefetch_words:-0}"
- # Zero-token JD rescue: read the JD out of the structured data the board
- # already shipped in the bytes above. It only replaces the text when the page
- # is thin or the rescued JD is longer, and it fails open (any failure leaves
- # the stripped text as is, and the checks below decide as before).
- if [[ -f "$BATCH_DIR/jd-rescue.mjs" && -s "$jd_raw_file" ]]; then
- _rs_rc=0
- _rs_line=$(node "$BATCH_DIR/jd-rescue.mjs" "$url" "$jd_raw_file" "$jd_file" "$jd_prefetch_words" "${current_url:-$url}" 2>/dev/null) || _rs_rc=$?
- if [[ "$_rs_rc" -eq 3 ]]; then
- # The ATS says the posting is closed: terminal, even though the page text may look like a JD.
- jd_rescue_closed_note="posting closed (jd-rescue): $(printf '%s' "${_rs_line#*$'\t'}" | tr -c '[:print:]' ' ' | sed 's/^closed://; s/ */ /g')"
- : > "$jd_file"
- jd_prefetch_words=0
- elif [[ "$_rs_rc" -eq 4 ]]; then
- # The download landed on a different page than the offer URL: its text is not this posting's.
- : > "$jd_file"
- jd_prefetch_words=0
- elif [[ "$_rs_rc" -eq 0 ]]; then
- IFS=$'\t' read -r _rs_words _rs_method <<< "$_rs_line"
- _rs_words="${_rs_words//[^0-9]/}"
- _rs_words="${_rs_words:-0}"
- if [[ "$_rs_words" -ge "$prefetch_min_words" ]]; then
- echo " โน๏ธ JD rescue: recovered ${_rs_words} words via ${_rs_method} (visible text had ${jd_prefetch_words})"
- jd_prefetch_words="$_rs_words"
- fi
- fi
- fi
- rm -f "$jd_raw_file"
- if [[ "$jd_prefetch_words" -lt "$prefetch_min_words" ]]; then
- : > "$jd_file"
- echo " โน๏ธ JD prefetch: thin content (${jd_prefetch_words} words) โ worker will WebFetch"
- else
- echo " โน๏ธ JD prefetch: ${jd_prefetch_words} words written to JD file"
- fi
- fi
- # ATS API fallback (#4246). When the static fetch above left $jd_file empty
- # (thin JS shell, curl failure, curl absent), ask the ATS's own public API
- # through fetch-jd.mjs before the worker falls back to WebFetch, which hits
- # the same unrendered shell on Ashby and embedded Greenhouse boards.
- # The call lives here and not in the worker prompt on purpose: the offer URL
- # is untrusted, and here it travels as one argv element ("$url"), never as
- # text inside a shell command the worker composes. fetch-jd.mjs only talks to
- # the known ATS API hosts it resolves from the URL, so a private or unknown
- # host is a miss (exit 1), not a request.
- local jd_closed_note="$jd_rescue_closed_note"
- local jd_api_words=0
- if [[ ! -s "$jd_file" && -z "$jd_closed_note" && -f "$PROJECT_DIR/fetch-jd.mjs" ]]; then
- local jd_api_tmp jd_api_rc=0
- jd_api_tmp="$(mktemp "${TMPDIR:-/tmp}/batch-jd-api.XXXXXX")"
- node "$PROJECT_DIR/fetch-jd.mjs" "$url" > "$jd_api_tmp" 2>/dev/null || jd_api_rc=$?
- if [[ "$jd_api_rc" -eq 3 ]]; then
- # Exit 3: the posting's own data says it is closed (validThrough in the past,
- # closure text on the page, an ATS delisting flag). Do not fall back to
- # WebFetch, which could read the JD from the page and score a closed role.
- # The evidence is page text, so keep it to one short printable line.
- jd_closed_note="posting closed (fetch-jd): $(head -c 160 "$jd_api_tmp" | tr -c '[:print:]' ' ' | sed 's/^CLOSED_POSTING //; s/ */ /g')"
- elif [[ "$jd_api_rc" -eq 0 ]]; then
- jd_api_words=$(wc -w < "$jd_api_tmp" 2>/dev/null) || jd_api_words=0
- jd_api_words="${jd_api_words//[^0-9]/}"
- jd_api_words="${jd_api_words:-0}"
- if grep -q 'Job description truncated here: the posting is longer than' "$jd_api_tmp" 2>/dev/null; then
- # fetch-jd.mjs marked this text as cut short. Scoring and archiving it
- # would treat a partial JD as the whole posting, so leave the JD file
- # empty and let the existing thin-JD handling decide what happens next.
- echo " โน๏ธ JD prefetch: fetch-jd.mjs returned a truncated JD โ not used as the full posting"
- elif [[ "$jd_api_words" -ge "$prefetch_min_words" ]]; then
- cat "$jd_api_tmp" > "$jd_file"
- echo " โน๏ธ JD prefetch: ${jd_api_words} words from the ATS API (fetch-jd.mjs)"
- fi
- fi
- rm -f "$jd_api_tmp"
- fi
- # --thin-action: no usable JD text after the static fetch and the ATS API
- # fallback (a JS-rendered page, an empty shell, a failed fetch). The default,
- # 'webfetch', changes nothing: the worker is told to WebFetch. Detection only
- # here; the skip is recorded after the "Processing offer" line below.
- local jd_thin_skip=0 jd_thin_note="" jd_thin_retryable=0
- if [[ -n "$jd_closed_note" ]]; then
- # A closed posting is terminal ('skipped', not retryable) whatever --thin-action says.
- jd_thin_skip=1
- jd_thin_note="$jd_closed_note"
- elif [[ ! -s "$jd_file" && "$THIN_ACTION" != "webfetch" ]]; then
- if [[ "$THIN_ACTION" == "browser-extract" && -f "$PROJECT_DIR/browser-extract.mjs" ]]; then
- # Real-browser read: zero model tokens, ~5-15s of Chromium. Success is
- # compact JSON on stdout, a failure is exit 1 with a JSON error on stderr.
- # A near-empty or taken-down page is a failure there, so the word floor
- # below only has to stop a login wall or an error page full of navigation.
- local be_out be_err be_words=0 be_code=""
- be_out="$(mktemp "${TMPDIR:-/tmp}/batch-jd-browser.XXXXXX")"
- be_err="$(mktemp "${TMPDIR:-/tmp}/batch-jd-browser-err.XXXXXX")"
- if node "$PROJECT_DIR/browser-extract.mjs" "$url" --mode jd > "$be_out" 2> "$be_err"; then
- node -e '
- let d = "";
- process.stdin.on("data", (c) => d += c);
- process.stdin.on("end", () => {
- try { const o = JSON.parse(d); process.stdout.write(typeof o.text === "string" ? o.text : ""); } catch { /* empty */ }
- });
- ' < "$be_out" > "$jd_file" 2>/dev/null || : > "$jd_file"
- be_words=$(wc -w < "$jd_file" 2>/dev/null) || be_words=0
- be_words="${be_words//[^0-9]/}"; be_words="${be_words:-0}"
- if [[ "$be_words" -ge 200 ]]; then
- echo " ๐ญ Browser extract: recovered ${be_words} words"
- jd_prefetch_words="$be_words"
- else
- : > "$jd_file"
- echo " ๐ญ Browser extract: no readable JD (${be_words} words)"
- fi
- else
- : > "$jd_file"
- be_code=$(node -e '
- let d = "";
- process.stdin.on("data", (c) => d += c);
- process.stdin.on("end", () => { try { process.stdout.write(String(JSON.parse(d).code || "")); } catch { /* none */ } });
- ' < "$be_err" 2>/dev/null) || be_code=""
- echo " ๐ญ Browser extract failed${be_code:+ ($be_code)}"
- # Allowlist of transient failures worth another try. Everything else
- # (unavailable, URL or validation errors, an unknown code) is a verdict
- # about the page or the call and stays terminal.
- case "$be_code" in
- no_playwright)
- jd_thin_retryable=1
- jd_thin_note="browser-extract: Playwright is not installed โ run 'npx playwright install chromium' and retry"
- ;;
- empty_text|navigation_error)
- jd_thin_retryable=1
- jd_thin_note="browser-extract failed ($be_code) โ retryable"
- ;;
- esac
- fi
- rm -f "$be_out" "$be_err"
- fi
- if [[ ! -s "$jd_file" ]]; then
- jd_thin_skip=1
- # curl never got an HTTP response (DNS, refused, timeout, TLS): a transport
- # problem, not a page that cannot be read. 'skipped' is terminal and
- # --retry-failed would never pick it up again, so it stays retryable.
- case "$curl_status" in
- 6|7|28|35|52|55|56)
- jd_thin_retryable=1
- [[ -n "$jd_thin_note" ]] || jd_thin_note="JD prefetch failed to connect (curl exit $curl_status) โ retryable"
- ;;
- esac
- [[ -n "$jd_thin_note" ]] || jd_thin_note="no usable JD text (--thin-action $THIN_ACTION) โ needs-playwright"
- fi
- fi
- echo "--- Processing offer #$id${JOB_POS:+ ($JOB_POS)}: $url (report $report_num, attempt $((retries + 1)))"
- if [[ "$jd_thin_skip" == "1" ]]; then
- local thin_state="skipped" thin_now
- local thin_retries="$retries"
- if [[ "$jd_thin_retryable" == "1" ]]; then
- thin_state="failed"
- # Spend one retry so --retry-failed stops at MAX_RETRIES instead of looping.
- (( thin_retries < MAX_RETRIES )) && thin_retries=$((retries + 1))
- fi
- thin_now=$(date -u +%Y-%m-%dT%H:%M:%SZ)
- update_state_retrying "$id" "$url" "$thin_state" "$started_at" "$thin_now" "-" "-" "$jd_thin_note" "$thin_retries" || true
- release_report_num "$report_num"
- rm -f "$jd_file"
- echo " ๐ซ $jd_thin_note โ no worker spawned"
- return 0
- fi
- # Build the prompt with placeholders replaced
- local prompt
- if [[ "${MODE:-full}" == "match" ]]; then
- prompt="Process this job offer in single-pass CV match mode. Extract CV facts, score the match against the JD requirements in a single pass, output the report and tracker addition, and output final JSON. Do not generate PDF; write โ in the tracker PDF column and set \"pdf\": null in the final JSON."
- elif [[ "$SKIP_PDF" == "true" ]]; then
- prompt="Process this job offer. Run the pipeline: A-G evaluation + report .md + tracker line. Do not generate PDF; write โ in the tracker PDF column and set \"pdf\": null in the final JSON."
- echo " โญ๏ธ --skip-pdf set โ skipping PDF generation for #$id ($url)"
- else
- prompt="Process this job offer. Run the full pipeline: A-G evaluation + report .md + optional PDF + tracker line."
- fi
- prompt="$prompt URL: $url"
- prompt="$prompt JD file: $jd_file"
- prompt="$prompt Report number: $report_num"
- prompt="$prompt Date: $date"
- prompt="$prompt Batch ID: $id"
- local log_file="$LOGS_DIR/${report_num}-${id}.log"
- # Prepare system prompt with placeholders resolved
- local resolved_prompt="$BATCH_DIR/.resolved-prompt-${id}.md"
- # Resolve placeholders to STABLE labels, not per-offer values: the concrete
- # URL / JD file / report number / date / batch ID travel in the per-job user
- # prompt built above. Keeping the system prompt byte-identical across offers
- # lets prompt caching reuse the whole prefix instead of breaking at the first
- # substituted value (~4 KB into a ~41 KB prompt). This also means the URL
- # never enters a sed replacement, so it needs no delimiter/metacharacter
- # escaping (no more esc_url/esc_jd_file/&-in-query-string concerns).
- sed \
- -e "s|{{URL}}|<URL from the job message>|g" \
- -e "s|{{JD_FILE}}|<JD file from the job message>|g" \
- -e "s|{{REPORT_NUM}}|<report number from the job message>|g" \
- -e "s|{{DATE}}|<date from the job message>|g" \
- -e "s|{{ID}}|<batch ID from the job message>|g" \
- "$PROMPT_FILE" > "$resolved_prompt"
- # The prompt names user-layer files by repo-relative path, and the worker
- # resolves those against its working directory, the checkout. With the data
- # root elsewhere that is the wrong tree for everything it reads about the
- # candidate and everything it writes, so say where the user layer is. The
- # text depends only on the data root, so it is the same for every offer in a
- # run, and the default layout adds nothing.
- if [[ "$DATA_ROOT" != "$PROJECT_DIR" ]]; then
- {
- printf '\n\n---\n\n'
- printf '## Runtime paths: data root outside the checkout\n\n'
- if [[ "${MODE:-full}" == "match" ]]; then
- # Match mode names no profile file (#5297), so do not point at one.
- printf 'This install keeps the user layer in a separate data root. Wherever this prompt names one of the paths below, resolve it under the data root, not the working directory:\n\n'
- else
- printf 'This install keeps the user layer in a separate data root. The profile for this run is `%s`. Where this prompt names `config/profile.yml`, read that file instead. Wherever this prompt names one of the paths below, resolve it under the data root, not the working directory:\n\n' "$PROFILE_FILE"
- fi
- printf 'Data root: %s\n\n' "$DATA_ROOT"
- printf '`cv.md`, `article-digest.md`, `modes/_profile.md`, `modes/_custom.md`, `reports/`, `output/`, `data/`, `interview-prep/`, `batch/tracker-additions/`\n\n'
- printf 'Write the report under `reports/` in the data root and the tracker TSV under `batch/tracker-additions/` in the data root, and give scripts absolute data-root paths for these files. Markdown links written inside the report and the tracker TSV keep the relative form this prompt shows (`reports/...`). Every other path (`templates/`, the `.mjs` scripts, the other `modes/` files) stays relative to the working directory.\n'
- } >> "$resolved_prompt"
- fi
- # The .md prompts name config/profile.yml. When CAREER_OPS_PROFILE names a
- # different file, the worker would read the wrong profile for everything
- # except the personalization injected below, so name the file this run uses.
- # With the data root outside the checkout the runtime block above already
- # does that, so this only fills the default layout.
- # The override is checked first, the way hasProfileOverride() does it, and
- # the path comparison only after it. With no override the two paths are the
- # same file, but not always the same string: through a junction or symlink
- # getProfilePath() returns the real path while DATA_ROOT keeps the one pwd
- # gave, and that alone used to add this block.
- local profile_override="${CAREER_OPS_PROFILE:-}"
- if [[ "${MODE:-full}" != "match" && -n "${profile_override//[[:space:]]/}" && "$PROFILE_RESOLVED" == "true" && "$PROFILE_FILE" != "$DATA_ROOT/config/profile.yml" && "$DATA_ROOT" == "$PROJECT_DIR" ]]; then
- {
- printf '\n\n---\n\n'
- printf '## Runtime profile\n\n'
- printf 'The profile for this run is `%s`. Where this prompt names `config/profile.yml`, read that file instead.\n' "$PROFILE_FILE"
- } >> "$resolved_prompt"
- fi
- # Inject user-layer personalization into the temporary worker prompt.
- # The resolved prompt is gitignored runtime state, so user profile data stays
- # out of the system layer while batch scoring matches interactive scoring.
- # Match mode gets none of it, only the output language (#5297).
- if [[ "${MODE:-full}" == "match" ]]; then
- {
- printf '\n\n---\n\n'
- printf '## Runtime output language\n\n'
- printf 'language.output for this run: `%s`\n' "${MATCH_OUTPUT_LANGUAGE:-en}"
- } >> "$resolved_prompt"
- else
- for context_file in "$DATA_ROOT/modes/_profile.md" "$PROFILE_FILE" "$DATA_ROOT/modes/_custom.md"; do
- if [[ -f "$context_file" ]]; then
- {
- printf '\n\n---\n\n'
- printf '## Runtime personalization: %s\n\n' "${context_file#"$DATA_ROOT/"}"
- sed 's/^/ /' "$context_file"
- printf '\n'
- } >> "$resolved_prompt"
- fi
- done
- fi
- # The run-wide evidence freshness result (Step 1.7). The same text for every
- # offer, so it keeps the resolved prompt byte-identical across the run.
- if [[ -n "$EVIDENCE_FRESHNESS_JSON" ]]; then
- {
- printf '\n\n---\n\n'
- printf '## Runtime: GitHub evidence freshness\n\n'
- printf 'The runner ran `node evidence-freshness.mjs` once for this batch run. Its JSON output:\n\n'
- printf '%s\n' "$EVIDENCE_FRESHNESS_JSON" | sed 's/^/ /'
- } >> "$resolved_prompt"
- fi
- # Build the launch command for the configured CLI.
- # For claude the model is resolved once per run from spend_tier unless --model
- # was passed; other CLIs take --model verbatim. Building claude's command in an
- # array keeps quoting safe regardless.
- # --strict-mcp-config (with no --mcp-config) starts workers with no MCP
- # servers: they only evaluate offers and need none. Without it each parallel
- # worker inherits the parent session's MCP (e.g. Playwright) and they deadlock
- # fighting over the single shared browser when --parallel > 1 (issue #506).
- local -a claude_args=(-p --dangerously-skip-permissions --strict-mcp-config)
- if [[ -n "$RESOLVED_MODEL" ]]; then
- claude_args+=(--model "$RESOLVED_MODEL")
- fi
- claude_args+=(--append-system-prompt-file "$resolved_prompt" "$prompt")
- # Non-claude CLIs lack --append-system-prompt-file, so concatenate the
- # resolved system prompt and the per-job prompt into a single argument.
- # model_args is expanded with the ${arr[@]+"${arr[@]}"} idiom below: under
- # `set -u`, bash 3.2 (macOS /bin/bash) treats an empty array expansion as an
- # unbound variable and would abort every worker launched without --model.
- local full_prompt=""
- local -a model_args=()
- if [[ "$CLI" != "claude" ]]; then
- full_prompt="$(cat "$resolved_prompt")"$'\n\n'"$prompt"
- [[ -n "$MODEL" ]] && model_args=(--model "$MODEL")
- fi
- # Non-claude CLIs run a single attempt (explicit break after dispatch); the
- # rate-limit/session retry loop only applies to claude.
- local exit_code=0
- local terminal_failure_recorded=false
- local shim_retries=0
- local max_shim_retries=4
- local match_output_retries=0
- local max_match_output_retries=1
- while true; do
- exit_code=0
- case "$CLI" in
- claude)
- run_worker claude "${claude_args[@]}" > "$log_file" 2>&1 || exit_code=$?
- ;;
- opencode)
- if command -v opencode &>/dev/null; then
- run_worker opencode run ${model_args[@]+"${model_args[@]}"} "$full_prompt" > "$log_file" 2>&1 || exit_code=$?
- else
- run_worker ollama launch opencode ${model_args[@]+"${model_args[@]}"} -y -- run "$full_prompt" > "$log_file" 2>&1 || exit_code=$?
- fi
- ;;
- gemini)
- run_worker gemini ${model_args[@]+"${model_args[@]}"} -p "$full_prompt" > "$log_file" 2>&1 || exit_code=$?
- ;;
- qwen)
- run_worker qwen ${model_args[@]+"${model_args[@]}"} -p "$full_prompt" > "$log_file" 2>&1 || exit_code=$?
- ;;
- grok)
- run_worker grok --always-approve --output-format plain ${model_args[@]+"${model_args[@]}"} -p "$full_prompt" > "$log_file" 2>&1 || exit_code=$?
- ;;
- esac
- # In match mode, check if worker result JSON is malformed or missing required
- # fields (parsed_status or parsed_score). If so, retry once before finalizing
- # so a transient formatting glitch does not permanently fail or bypass gates.
- if [[ "${MODE:-full}" == "match" && $exit_code -eq 0 ]]; then
- local worker_check_json
- worker_check_json=$(awk '
- /^```json[[:space:]]*$/ { in_block=1; block=""; next }
- in_block && /^```[[:space:]]*$/ { in_block=0; last=block; next }
- in_block { block = block $0 "\n" }
- END { printf "%s", last }
- ' "$log_file" 2>/dev/null || true)
- local check_status="" check_score=""
- if [[ -n "$worker_check_json" ]]; then
- local check_parsed
- check_parsed=$(printf '%s' "$worker_check_json" | node -e '
- let data = "";
- process.stdin.on("data", d => data += d);
- process.stdin.on("end", () => {
- try {
- const obj = JSON.parse(data);
- const status = typeof obj.status === "string" ? obj.status : "";
- const score = typeof obj.score === "number" ? String(obj.score) : "";
- process.stdout.write(status + "\x1f" + score);
- } catch {
- process.stdout.write("");
- }
- });
- ' 2>/dev/null || true)
- if [[ -n "$check_parsed" ]]; then
- IFS=$'\x1f' read -r check_status check_score <<< "$check_parsed"
- fi
- fi
- if [[ -z "$check_status" || ( "$check_status" != "needs_confirmation" && "$check_status" != "skipped" && "$check_status" != "failed" && -z "$check_score" ) ]]; then
- if (( match_output_retries < max_match_output_retries && retries < MAX_RETRIES )); then
- match_output_retries=$((match_output_retries + 1))
- retries=$((retries + 1))
- # Move the malformed attempt's artifacts aside first. Left in place,
- # a retry with another slug leaves two reports on one number, and a
- # retry that writes no report would pass the report-file check on the
- # stale one. Same quarantine as the confirmation-hold path below.
- local retry_quarantine="$LOGS_DIR/quarantine" retry_artifact
- shopt -s nullglob
- for retry_artifact in "$REPORTS_DIR/$report_num-"*.md "$TRACKER_DIR/$id.tsv"; do
- [[ "$retry_artifact" == *-RESERVED.md ]] && continue
- mkdir -p "$retry_quarantine"
- mv "$retry_artifact" "$retry_quarantine/$id-match-retry-${retry_artifact##*/}"
- done
- shopt -u nullglob
- echo " โ ๏ธ Malformed or missing match mode JSON summary (attempt $retries/$MAX_RETRIES); retrying once..."
- sleep 2
- continue
- fi
- fi
- fi
- # Non-claude CLIs run a single attempt: the session-limit and rate-limit
- # detection below greps generic phrases (429, quota, session limit) that
- # another CLI's log could match by coincidence and pause or retry the batch.
- if [[ "$CLI" != "claude" ]]; then
- break
- fi
- if [[ $exit_code -eq 0 ]]; then
- break
- fi
- # Check for Claude Code npm shim swap (exit code 127 + command not found)
- if [[ $exit_code -eq 127 ]] && grep -qE "(claude: command not found|claude:.*not found|cannot find.*claude)" "$log_file" && (( shim_retries < max_shim_retries )); then
- shim_retries=$((shim_retries + 1))
- echo " โณ Claude command not found (shim swap detected). Retrying in 30s (attempt $shim_retries/$max_shim_retries)..."
- sleep 30
- continue
- fi
- if is_session_limit_log "$log_file"; then
- mark_paused_rate_limit "$id" "$url" "$started_at" "$report_num" "$retries" "$log_file"
- echo " โธ๏ธ Session/rate limit reached; pausing batch without consuming retry budget."
- terminal_failure_recorded=true
- break
- fi
- if is_rate_limit_log "$log_file" && (( retries < MAX_RETRIES )); then
- if (( RATE_LIMIT_SLEEP <= 0 )); then
- mark_paused_rate_limit "$id" "$url" "$started_at" "$report_num" "$retries" "$log_file"
- echo " โธ๏ธ Rate limited and --rate-limit-sleep is 0; pausing batch without consuming retry budget."
- terminal_failure_recorded=true
- break
- fi
- local retry_delay
- retry_delay=$(rate_limit_delay "$log_file" "$retries")
- if [[ "$retry_delay" == pause ]]; then
- mark_paused_rate_limit "$id" "$url" "$started_at" "$report_num" "$retries" "$log_file"
- echo " โธ๏ธ Retry-After exceeds --rate-limit-sleep; pausing batch without consuming retry budget."
- terminal_failure_recorded=true
- break
- fi
- : > "$THROTTLE_FILE"
- retries=$((retries + 1))
- local retry_completed_at
- retry_completed_at=$(date -u +%Y-%m-%dT%H:%M:%SZ)
- update_state_retrying "$id" "$url" "rate_limited" "$started_at" "$retry_completed_at" "$report_num" "-" "rate-limit; retrying after ${retry_delay}s" "$retries" || true
- echo " โณ Rate limited (attempt $retries/$MAX_RETRIES). Waiting ${retry_delay}s before retry..."
- sleep "$retry_delay"
- continue
- fi
- break
- done
- # Cleanup resolved prompt and pre-fetched JD file
- rm -f "$resolved_prompt" "$jd_file"
- local completed_at
- completed_at=$(date -u +%Y-%m-%dT%H:%M:%SZ)
- if [[ $exit_code -eq 0 ]]; then
- # A worker can exit 0 (no crash) but still self-report failure inside its
- # own JSON summary โ e.g. it correctly declines to fabricate an evaluation
- # when the JD couldn't be extracted (Data Contract: never fabricate).
- # Without this check such offers were silently marked "completed" with no
- # report file on disk and score "-" (found 2026-07-29, offer id 6 / report
- # 019 โ Deepgram JD unextractable in headless mode). Only the downstream
- # reconcile-pipeline.mjs safety net (which leaves an entry in Pending when
- # its report file is missing) prevented the offer from being lost.
- # Extract only the LAST ```json fenced block in the log -- that's the
- # worker's one authoritative final result (batch-prompt.md Step 6), not
- # arbitrary text anywhere else in stdout/stderr -- and parse it as real
- # JSON so an unrelated line merely containing the substring
- # `"status": "failed"` can never falsely flip a successful run.
- local worker_result_json
- worker_result_json=$(awk '
- /^```json[[:space:]]*$/ { in_block=1; block=""; next }
- in_block && /^```[[:space:]]*$/ { in_block=0; last=block; next }
- in_block { block = block $0 "\n" }
- END { printf "%s", last }
- ' "$log_file" 2>/dev/null || true)
- # Parse status, error, AND score from the same authoritative JSON object
- # in one pass -- score extraction used to be a separate sed regex over
- # the whole log (`.*"score":...`), which grabbed the first match
- # anywhere in the log rather than the one from this final result object,
- # producing a spurious score "-" whenever an earlier line in the log
- # (reasoning text, an intermediate example, Block D's "Comp score: 4/5"
- # mention, etc.) matched first. Reading it from the same parsed object
- # as status/error fixes both by construction -- there's only one place
- # left to look.
- local worker_failed_match="" worker_error_match="" score="-"
- local parsed_status="" parsed_error="" parsed_score=""
- if [[ -n "$worker_result_json" ]]; then
- local parsed
- parsed=$(printf '%s' "$worker_result_json" | node -e '
- let data = "";
- process.stdin.on("data", d => data += d);
- process.stdin.on("end", () => {
- try {
- const obj = JSON.parse(data);
- const status = typeof obj.status === "string" ? obj.status : "";
- const message = status === "needs_confirmation" ? obj.question : obj.error;
- const error = typeof message === "string" ? message.replace(/[\x00-\x1f\x7f]/g, " ") : "";
- const score = typeof obj.score === "number" ? String(obj.score) : "";
- process.stdout.write(status + "\x1f" + error + "\x1f" + score);
- } catch {
- process.stdout.write("");
- }
- });
- ' 2>/dev/null || true)
- if [[ -n "$parsed" ]]; then
- # \x1f (US), not \t: tab is IFS *whitespace*, so bash collapses runs of it
- # and strips leading/trailing ones. On the common path -- a worker that
- # succeeded, so `error` is empty -- the two tabs around that empty field
- # collapse into one, `score` slides into parsed_error, and parsed_score
- # comes back empty. The `elif` below then never fires and every
- # successful offer records score "-". A non-whitespace delimiter gets
- # one-field-per-unit splitting with empty fields preserved.
- IFS=$'\x1f' read -r parsed_status parsed_error parsed_score <<< "$parsed"
- if [[ "$parsed_status" == "failed" ]]; then
- worker_failed_match="failed"
- worker_error_match="$parsed_error"
- elif [[ -n "$parsed_score" ]]; then
- score="$parsed_score"
- fi
- fi
- fi
- # In match mode, enforce valid JSON output and required fields. If malformed
- # or missing parsed_status or parsed_score (and not a human-confirmation hold
- # or dealbreaker skip), treat as a failed worker attempt so it cannot be
- # recorded as completed with score "-" or bypass --min-score.
- if [[ "${MODE:-full}" == "match" && -z "$worker_failed_match" ]]; then
- if [[ -z "$worker_result_json" || -z "$parsed" || -z "$parsed_status" ]]; then
- worker_failed_match="failed"
- worker_error_match="malformed or missing match mode JSON summary (parsed_status missing)"
- elif [[ "$parsed_status" != "needs_confirmation" && "$parsed_status" != "skipped" ]] && { [[ -z "$parsed_score" ]] || ! is_decimal_number "$score"; }; then
- worker_failed_match="failed"
- worker_error_match="missing or invalid match mode score: $score"
- fi
- fi
- # A human gate is not a failure or a successful evaluation (#4359).
- # Keep it out of retries and release the unused report reservation.
- if [[ "$parsed_status" == "needs_confirmation" ]]; then
- local question="${parsed_error:-Which agency did this posting come through?}"
- # A malformed/misattributed handoff stays held, never becomes retryable
- # and never supplies an agency answer for a different posting.
- if ! node -e '
- const result = JSON.parse(process.argv[1]);
- process.exit(result.reason === "agency_confirmation" &&
- result.url === process.argv[2] && String(result.id) === process.argv[3] ? 0 : 1);
- ' "$worker_result_json" "$url" "$id"; then
- question="Invalid confirmation handoff identity/reason; parent must inspect the job log and ask for this URL"
- fi
- local tracker_artifact="$TRACKER_DIR/$id.tsv"
- local -a report_artifacts=()
- if [[ -n "$report_num" && "$report_num" != "-" ]]; then
- shopt -s nullglob
- local candidate
- # The reservation sentinel ({num}-RESERVED.md) is the parent's own file,
- # not a worker artifact (same exclusion as the report-file check below).
- for candidate in "$REPORTS_DIR/$report_num-"*.md; do
- [[ "$candidate" == *-RESERVED.md ]] || report_artifacts+=("$candidate")
- done
- shopt -u nullglob
- fi
- if [[ -f "$tracker_artifact" || ${#report_artifacts[@]} -gt 0 ]]; then
- local quarantine_dir="$LOGS_DIR/quarantine"
- mkdir -p "$quarantine_dir"
- if [[ -f "$tracker_artifact" ]]; then
- mv "$tracker_artifact" "$quarantine_dir/$id-tracker.tsv"
- fi
- local report_artifact
- for report_artifact in ${report_artifacts[@]+"${report_artifacts[@]}"}; do
- mv "$report_artifact" "$quarantine_dir/$id-${report_artifact##*/}"
- done
- question="Worker violated the confirmation hold by writing artifacts; inspect the quarantined tracker/report before answering"
- update_state_retrying "$id" "$url" "needs_confirmation" "$started_at" "$completed_at" "-" "-" "$question" "$retries" || true
- echo " ERROR: confirmation hold produced artifacts; reservation kept and tracker merge skipped." >&2
- return 2
- fi
- local hold_rc=0
- update_state_retrying "$id" "$url" "needs_confirmation" "$started_at" "$completed_at" "-" "-" "$question" "$retries" || hold_rc=$?
- release_report_num "$report_num"
- echo " Needs confirmation: $url โ $question (resume in the parent session after an explicit answer)"
- if (( hold_rc == 2 )); then
- echo " ERROR: confirmation hold could not be persisted; stop and repair state before rerunning this posting." >&2
- return 2
- fi
- return 0
- fi
- # A dealbreaker hit is a finished skip, not a hold (batch-prompt.md Step
- # 1.6). The worker writes one SKIP row quoting the rule and the JD line and
- # no report. Accept it only when the handoff names this posting and the row
- # proves it; anything else is quarantined and retried like a failure.
- if [[ "$parsed_status" == "skipped" ]]; then
- local skip_tracker="$TRACKER_DIR/$id.tsv" skip_rule=""
- local -a skip_reports=()
- if [[ -n "$report_num" && "$report_num" != "-" ]]; then
- shopt -s nullglob
- local skip_candidate
- for skip_candidate in "$REPORTS_DIR/$report_num-"*.md; do
- [[ "$skip_candidate" == *-RESERVED.md ]] || skip_reports+=("$skip_candidate")
- done
- shopt -u nullglob
- fi
- if (( ${#skip_reports[@]} == 0 )); then
- skip_rule=$(node -e '
- const fs = require("fs");
- const [json, url, id, tsv, reserved] = process.argv.slice(1);
- const flat = (s) => String(s).replace(/[\x00-\x1f\x7f]+/g, " ").replace(/ +/g, " ").trim();
- const result = JSON.parse(json);
- if (result.reason !== "dealbreaker" || result.url !== url || String(result.id) !== id) process.exit(1);
- if (typeof result.rule !== "string" || !flat(result.rule)) process.exit(1);
- const lines = fs.readFileSync(tsv, "utf8").replace(/\r\n/g, "\n").split("\n").filter(Boolean);
- if (lines.length !== 2) process.exit(1);
- const head = lines[0].split("\t");
- const row = lines[1].split("\t");
- const field = (name) => row[head.indexOf(name)] ?? "";
- if (field("status") !== "SKIP" || field("url") !== url || !field("notes").startsWith("Dealbreaker: \"")) process.exit(1);
- // merge-tracker keeps the TSV num as the row id, so it must be the
- // number this runner reserved (zero padding aside).
- if (!/^\d+$/.test(field("num")) || Number(field("num")) !== Number(reserved)) process.exit(1);
- process.stdout.write(flat(result.rule));
- ' "$worker_result_json" "$url" "$id" "$skip_tracker" "$report_num" 2>/dev/null) || skip_rule=""
- fi
- if [[ -n "$skip_rule" ]]; then
- # Keep the reservation. The SKIP row uses this number and no report
- # file holds it, so releasing now would hand it to the next job before
- # the merge. merge_tracker releases it once the row is in the tracker.
- # rc 2: nothing was recorded and the job runs again as pending, so it
- # is not a discard yet.
- local skip_state_rc=0
- update_state_retrying "$id" "$url" "skipped" "$started_at" "$completed_at" "$report_num" "-" "dealbreaker: \"$skip_rule\"" "$retries" || skip_state_rc=$?
- if (( skip_state_rc != 2 )); then
- log_discard "$id" "$url" "dealbreaker: \"$skip_rule\""
- fi
- echo " โญ๏ธ Skipped (dealbreaker): $url โ \"$skip_rule\""
- return 0
- fi
- local skip_quarantine="$LOGS_DIR/quarantine"
- mkdir -p "$skip_quarantine"
- [[ -f "$skip_tracker" ]] && mv "$skip_tracker" "$skip_quarantine/$id-tracker.tsv"
- local skip_report
- for skip_report in ${skip_reports[@]+"${skip_reports[@]}"}; do
- mv "$skip_report" "$skip_quarantine/$id-${skip_report##*/}"
- done
- if (( retries < MAX_RETRIES )); then
- retries=$((retries + 1))
- fi
- update_state_retrying "$id" "$url" "failed" "$started_at" "$completed_at" "$report_num" "-" "dealbreaker skip without a valid SKIP tracker row for this posting, or with a report; artifacts quarantined" "$retries" || true
- release_report_num "$report_num"
- echo " โ Failed (invalid dealbreaker skip, attempt $retries): artifacts moved to $skip_quarantine"
- return 0
- fi
- if [[ -n "$worker_failed_match" ]]; then
- [[ -z "$worker_error_match" ]] && worker_error_match="worker reported status:failed (exit code 0)"
- if (( retries < MAX_RETRIES )); then
- retries=$((retries + 1))
- fi
- update_state_retrying "$id" "$url" "failed" "$started_at" "$completed_at" "$report_num" "-" "$worker_error_match" "$retries" || true
- release_report_num "$report_num"
- echo " โ Failed (worker-reported, attempt $retries): $worker_error_match"
- return 0
- fi
- # A worker can exit 0, self-report a non-"failed" status (or no parseable
- # JSON at all), and STILL never actually write the report file it claims
- # -- exit code and JSON status alone are not proof a report exists. Found
- # 2026-07-30: offer id 6/report 049 was marked "completed" this way with
- # no file on disk, silently freeing that number for a second, unrelated
- # offer to claim (a real collision). Verify the file before trusting
- # "completed" -- fail closed, not open.
- # The reservation sentinel ({num}-RESERVED.md) must NOT count as the
- # report: it is created before the worker runs, so an unfiltered glob
- # matches it and the guard is defeated every single time (found 2026-09-22:
- # 16 no-op workers recorded as "completed" with score "-").
- if ! has_report_file "$report_num"; then
- if (( retries < MAX_RETRIES )); then
- retries=$((retries + 1))
- fi
- update_state_retrying "$id" "$url" "failed" "$started_at" "$completed_at" "$report_num" "-" "worker exited cleanly but wrote no report file for this report number" "$retries" || true
- release_report_num "$report_num"
- echo " โ Failed (no report file on disk, attempt $retries)"
- return 0
- fi
- # Check min-score gate
- if is_decimal_number "$score" && LC_ALL=C awk -v min="$MIN_SCORE" 'BEGIN{exit !(min > 0)}'; then
- # LC_ALL=C: under a non-English locale awk parses "4.5" as 4, so the
- # MIN_SCORE comparison would silently run on truncated integers.
- if LC_ALL=C awk -v score="$score" -v min="$MIN_SCORE" 'BEGIN{exit !(score < min)}'; then
- local score_state_rc=0
- update_state_retrying "$id" "$url" "skipped" "$started_at" "$completed_at" "$report_num" "$score" "below-min-score" "$retries" || score_state_rc=$?
- if (( score_state_rc != 2 )); then
- log_discard "$id" "$url" "below-min-score"
- fi
- release_report_num "$report_num"
- echo " โญ๏ธ Skipped (score: $score < min-score: $MIN_SCORE)"
- return 0
- fi
- fi
- # Defensive guard: in match mode, an offer must never complete without a valid numeric score.
- if [[ "${MODE:-full}" == "match" ]] && ! is_decimal_number "$score"; then
- if (( retries < MAX_RETRIES )); then
- retries=$((retries + 1))
- fi
- update_state_retrying "$id" "$url" "failed" "$started_at" "$completed_at" "$report_num" "-" "missing or invalid numeric score in match mode: $score" "$retries" || true
- release_report_num "$report_num"
- echo " โ Failed (invalid match mode score: $score, attempt $retries)"
- return 0
- fi
- update_state_retrying "$id" "$url" "completed" "$started_at" "$completed_at" "$report_num" "$score" "-" "$retries" || true
- release_report_num "$report_num"
- echo " โ Completed (score: $score, report: $report_num)"
- elif [[ "$terminal_failure_recorded" == "false" ]]; then
- if (( retries < MAX_RETRIES )); then
- retries=$((retries + 1))
- fi
- local error_msg
- error_msg=$(tail -5 "$log_file" 2>/dev/null | tr '\n' ' ' | cut -c1-200 || echo "Unknown error (exit code $exit_code)")
- update_state_retrying "$id" "$url" "failed" "$started_at" "$completed_at" "$report_num" "-" "$error_msg" "$retries" || true
- release_report_num "$report_num"
- echo " โ Failed (attempt $retries, exit code $exit_code)"
- fi
- }
- # GitHub evidence freshness (#5674): one check for the whole run, not one per
- # worker. The check makes a `gh search prs` call and its answer is the same for
- # every offer, so the runner makes it once and every worker reads the result
- # from its prompt. It is opt-in: with evidence_freshness.enabled not true it
- # reports status "disabled" without any network call, and workers get nothing.
- check_evidence_freshness() {
- [[ -f "$PROJECT_DIR/evidence-freshness.mjs" ]] || return 0
- local out status
- out=$(node "$PROJECT_DIR/evidence-freshness.mjs" 2>/dev/null) || return 0
- status=$(printf '%s\n' "$out" | sed -n 's/^ "status": "\(.*\)",$/\1/p')
- [[ -z "$status" || "$status" == "disabled" ]] && return 0
- EVIDENCE_FRESHNESS_JSON="$out"
- if printf '%s\n' "$out" | grep -q '^ "stale": true,$'; then
- echo "GitHub evidence: stale (checked once for this run; see each report header)"
- else
- echo "GitHub evidence: $status (checked once for this run)"
- fi
- echo ""
- }
- # Merge tracker additions into applications.md
- merge_tracker() {
- echo ""
- echo "=== Merging tracker additions ==="
- node "$PROJECT_DIR/merge-tracker.mjs"
- # A dealbreaker SKIP row holds its report number until it is merged. The
- # tracker row now holds it for reserve-report-num.mjs, so drop the sentinel.
- if [[ -f "$STATE_FILE" ]]; then
- local mid mstatus mnum merror
- while IFS=$'\t' read -r mid _ mstatus _ _ mnum _ merror _; do
- if [[ "$mid" != "id" && "$mstatus" == "skipped" && "$merror" == dealbreaker:* ]]; then
- release_report_num "$mnum"
- fi
- done < "$STATE_FILE"
- fi
- echo ""
- echo "=== Reconciling pipeline.md ==="
- node "$PROJECT_DIR/reconcile-pipeline.mjs" || echo "โ ๏ธ Pipeline reconcile had issues (see above)"
- echo ""
- echo "=== Verifying pipeline integrity ==="
- node "$PROJECT_DIR/verify-pipeline.mjs" || echo "โ ๏ธ Verification found issues (see above)"
- }
- report_progress() {
- echo "=== Progress: $1/$2 offers finished, $((SECONDS - BATCH_START))s elapsed ==="
- }
- # Print summary
- print_summary() {
- echo ""
- echo "=== Batch Summary ==="
- if [[ ! -f "$STATE_FILE" ]]; then
- echo "No state file found."
- return
- fi
- local total=0 completed=0 skipped=0 failed=0 pending=0 needs_confirmation=0
- local score_sum=0 score_count=0
- local -a dealbreaker_lines=()
- while IFS=$'\t' read -r sid surl sstatus _ _ _ sscore serror _; do
- [[ "$sid" == "id" ]] && continue
- total=$((total + 1))
- case "$sstatus" in
- completed) completed=$((completed + 1))
- if is_decimal_number "$sscore"; then
- score_sum=$(LC_ALL=C awk -v sum="$score_sum" -v score="$sscore" 'BEGIN{print sum + score}' 2>/dev/null || echo "$score_sum")
- score_count=$((score_count + 1))
- fi
- ;;
- skipped) skipped=$((skipped + 1))
- [[ "$serror" == dealbreaker:* ]] && dealbreaker_lines+=("#$sid $surl โ $serror")
- ;;
- failed) failed=$((failed + 1)) ;;
- needs_confirmation) needs_confirmation=$((needs_confirmation + 1)) ;;
- *) pending=$((pending + 1)) ;;
- esac
- done < "$STATE_FILE"
- echo "Total: $total | Completed: $completed | Skipped: $skipped | Failed: $failed | Pending: $pending | Needs confirmation: $needs_confirmation"
- if (( ${#dealbreaker_lines[@]} > 0 )); then
- echo "Dealbreaker skips: ${#dealbreaker_lines[@]} (SKIP rows merged; to evaluate one anyway, rerun it in a session and say \"evaluate anyway\")"
- local dealbreaker_line
- for dealbreaker_line in "${dealbreaker_lines[@]}"; do
- echo " $dealbreaker_line"
- done
- fi
- if (( score_count > 0 )); then
- local avg
- # LC_ALL=C: under e.g. a German locale awk formats "%.1f" as "4,5"
- # instead of "4.5", and a decimal comma breaks every downstream parser.
- avg=$(LC_ALL=C awk -v sum="$score_sum" -v count="$score_count" 'BEGIN{printf "%.1f", sum / count}' 2>/dev/null || echo "N/A")
- echo "Average score: $avg/5 ($score_count scored)"
- fi
- if [[ -f "$BATCH_DIR/aggregate-tokens.mjs" ]]; then
- if ! node "$BATCH_DIR/aggregate-tokens.mjs"; then
- echo "Warning: token aggregation failed." >&2
- fi
- fi
- }
- print_status_table() {
- # $1 = "full" forces the per-job table whatever STATUS_FULL says (--watch).
- local force_full="${1:-}"
- if [[ ! -f "$STATE_FILE" ]]; then
- echo "No state file found at $STATE_FILE"
- return
- fi
- local total=0 completed=0 processing=0 failed=0 pending=0 skipped=0 rate_limited=0 paused_rate_limit=0 needs_confirmation=0
- local score_sum=0 score_count=0
- # Read first line to skip header
- local header=true
- while IFS=$'\t' read -r sid surl sstatus sstarted scompleted sreport sscore serror sretries || [[ -n "$sid" ]]; do
- if [[ "$header" == "true" ]]; then
- header=false
- continue
- fi
- [[ -z "$sid" ]] && continue
- sstatus="${sstatus%$'\r'}"
- sscore="${sscore%$'\r'}"
- serror="${serror%$'\r'}"
- sreport="${sreport%$'\r'}"
- total=$((total + 1))
- case "$sstatus" in
- completed)
- completed=$((completed + 1))
- if is_decimal_number "$sscore"; then
- score_sum=$(LC_ALL=C awk -v sum="$score_sum" -v score="$sscore" 'BEGIN{print sum + score}' 2>/dev/null || echo "$score_sum")
- score_count=$((score_count + 1))
- fi
- ;;
- processing) processing=$((processing + 1)) ;;
- needs_confirmation) needs_confirmation=$((needs_confirmation + 1)) ;;
- failed) failed=$((failed + 1)) ;;
- skipped) skipped=$((skipped + 1)) ;;
- rate_limited) rate_limited=$((rate_limited + 1)) ;;
- paused_rate_limit) paused_rate_limit=$((paused_rate_limit + 1)) ;;
- *) pending=$((pending + 1)) ;;
- esac
- done < "$STATE_FILE"
- echo "=== Batch Progress ==="
- echo "Total: $total | Completed: $completed | Processing: $processing | Failed: $failed | Pending: $pending | Skipped: $skipped | Rate Limited: $rate_limited | Paused: $paused_rate_limit | Needs confirmation: $needs_confirmation"
- if (( score_count > 0 )); then
- local avg
- # LC_ALL=C: under e.g. a German locale awk formats "%.1f" as "4,5"
- # instead of "4.5", and a decimal comma breaks every downstream parser.
- avg=$(LC_ALL=C awk -v sum="$score_sum" -v count="$score_count" 'BEGIN{printf "%.1f", sum / count}' 2>/dev/null || echo "N/A")
- echo "Average score: $avg/5 ($score_count scored)"
- fi
- if [[ "$force_full" != "full" && "$STATUS_FULL" != "true" ]]; then
- return
- fi
- echo ""
- # Format the per-job table:
- # Columns: ID, Status, Report, Score, Target (URL or explanation)
- printf "%-4s | %-17s | %-6s | %-5s | %-40s\n" "ID" "Status" "Report" "Score" "URL / Detail"
- printf "%-4s+%-19s+%-8s+%-7s+%-42s\n" "----" "-------------------" "--------" "-------" "------------------------------------------"
- header=true
- while IFS=$'\t' read -r sid surl sstatus sstarted scompleted sreport sscore serror sretries || [[ -n "$sid" ]]; do
- if [[ "$header" == "true" ]]; then
- header=false
- continue
- fi
- [[ -z "$sid" ]] && continue
- sstatus="${sstatus%$'\r'}"
- sscore="${sscore%$'\r'}"
- serror="${serror%$'\r'}"
- sreport="${sreport%$'\r'}"
- local target="$surl"
- if [[ "$sstatus" == "needs_confirmation" ]]; then
- target="$surl โ Needs confirmation: $serror"
- elif [[ "$sstatus" == "failed" && -n "$serror" && "$serror" != "-" ]]; then
- target="Error: $serror"
- fi
- # Trim target to fit nicely (e.g. 50 chars)
- if [[ "$sstatus" != "needs_confirmation" ]] && (( ${#target} > 50 )); then
- target="${target:0:47}..."
- fi
- printf "%-4s | %-17s | %-6s | %-5s | %-50s\n" "$sid" "$sstatus" "$sreport" "$sscore" "$target"
- done < "$STATE_FILE"
- }
- watch_status() {
- local active_pid=""
- if [[ -f "$LOCK_FILE" ]]; then
- active_pid=$(cat "$LOCK_FILE" 2>/dev/null || true)
- fi
- if [[ -n "$active_pid" ]] && kill -0 "$active_pid" 2>/dev/null; then
- echo "Watching batch-runner (PID $active_pid)... Press Ctrl+C to stop."
- while kill -0 "$active_pid" 2>/dev/null; do
- clear || printf "\033[c"
- echo "=== Watching Batch Progress (PID $active_pid) ==="
- print_status_table full
- sleep 2
- done
- echo ""
- echo "=== Batch runner process (PID $active_pid) has finished ==="
- else
- echo "No active batch-runner detected."
- fi
- echo "Showing final status:"
- print_status_table full
- # Chain verify-pipeline.mjs
- if [[ -f "$PROJECT_DIR/verify-pipeline.mjs" ]]; then
- echo ""
- echo "=== Running pipeline verification ==="
- node "$PROJECT_DIR/verify-pipeline.mjs" || echo "โ ๏ธ Verification found issues"
- fi
- }
- # Main
- main() {
- if [[ "$STATUS_ONLY" == "true" ]]; then
- check_status_prerequisites
- print_status_table
- exit 0
- fi
- if [[ "$WATCH_MODE" == "true" ]]; then
- check_status_prerequisites
- watch_status
- exit 0
- fi
- check_prerequisites
- resolve_worker_model
- if [[ "$DRY_RUN" == "false" ]]; then
- acquire_lock
- rm -f "$PAUSE_FILE" "$THROTTLE_FILE"
- select_state_lock
- fi
- init_state
- if [[ "$DRY_RUN" == "false" ]]; then
- reconcile_recovery_records
- fi
- # Count input offers (skip header, ignore blank lines)
- local total_input
- total_input=$(tail -n +2 "$INPUT_FILE" | grep -c '[^[:space:]]' 2>/dev/null || true)
- total_input="${total_input:-0}"
- if (( total_input == 0 )); then
- echo "No offers in $INPUT_FILE. Add offers first."
- exit 0
- fi
- echo "=== career-ops batch runner ==="
- if [[ "${MODE:-full}" == "match" ]]; then
- echo "Mode: match (single-pass CV evaluation)"
- fi
- if (( LIMIT > 0 )); then
- echo "Parallel: $PARALLEL | Max retries: $MAX_RETRIES | Limit: $LIMIT"
- else
- echo "Parallel: $PARALLEL | Max retries: $MAX_RETRIES"
- fi
- if [[ "$RESOLVED_SPEND_TIER" == "override" ]]; then
- echo "CLI: $CLI | Model: $RESOLVED_MODEL (explicit --model override)"
- elif [[ "$RESOLVED_SPEND_TIER" == "cli-default" ]]; then
- echo "CLI: $CLI | Model: $CLI default"
- else
- echo "CLI: $CLI | Model: $RESOLVED_MODEL (spend_tier=${RESOLVED_SPEND_TIER})"
- fi
- echo "Input: $total_input offers"
- echo ""
- # Build list of offers to process
- local -a pending_ids=()
- local -a pending_urls=()
- local -a pending_sources=()
- local -a pending_notes=()
- # --company: resolve the matching ids ONCE (batch/company-scope.mjs), then skip
- # every other row before --limit is counted.
- local scope_ids=""
- if [[ -n "$SCOPE_COMPANY" ]]; then
- if ! scope_ids=$(node "$BATCH_DIR/company-scope.mjs" --input "$INPUT_FILE" --company "$SCOPE_COMPANY"); then
- echo "ERROR: could not resolve --company scope (company-scope.mjs failed)." >&2
- exit 1
- fi
- scope_ids=" $(echo "$scope_ids" | tr '\n' ' ') "
- echo "Scope: company=$SCOPE_COMPANY ($(echo "$scope_ids" | wc -w | tr -d ' ') matching row(s))"
- fi
- while IFS=$'\t' read -r id url source notes; do
- [[ "$id" == "id" ]] && continue # skip header
- [[ -z "$id" || -z "$url" ]] && continue
- # Guard against non-numeric id values
- [[ "$id" =~ ^[0-9]+$ ]] || continue
- # Skip if before start-from
- if (( id < START_FROM )); then
- continue
- fi
- if [[ -n "$SCOPE_COMPANY" && "$scope_ids" != *" $id "* ]]; then
- continue
- fi
- local status
- status=$(get_status "$id")
- # No command-line retry flag constitutes a user's agency answer.
- if [[ "$status" == "needs_confirmation" ]]; then
- echo "HOLD #$id: needs confirmation in the parent session ($url)"
- continue
- fi
- if [[ "$RESUME_PAUSED" == "true" ]]; then
- if [[ "$status" != "paused_rate_limit" ]]; then
- continue
- fi
- elif [[ "$RETRY_FAILED" == "true" ]]; then
- # Only process failed offers
- if [[ "$status" != "failed" ]]; then
- continue
- fi
- # Check retry limit
- local retries
- retries=$(get_retries "$id")
- if (( retries >= MAX_RETRIES )); then
- echo "SKIP #$id: max retries ($MAX_RETRIES) reached"
- continue
- fi
- else
- # Skip terminal offers
- if [[ "$status" == "completed" || "$status" == "skipped" ]]; then
- continue
- fi
- # Paused rate-limit offers resume explicitly with --resume-paused.
- if [[ "$status" == "paused_rate_limit" ]]; then
- continue
- fi
- # Skip failed offers that hit retry limit (unless --retry-failed)
- if [[ "$status" == "failed" ]]; then
- local retries
- retries=$(get_retries "$id")
- if (( retries >= MAX_RETRIES )); then
- echo "SKIP #$id: failed and max retries reached (use --retry-failed to force)"
- continue
- fi
- fi
- fi
- if (( LIMIT > 0 )) && (( ${#pending_ids[@]} >= LIMIT )); then
- break
- fi
- pending_ids+=("$id")
- pending_urls+=("$url")
- pending_sources+=("$source")
- pending_notes+=("$notes")
- done < "$INPUT_FILE"
- local pending_count=${#pending_ids[@]}
- if (( pending_count == 0 )); then
- echo "No offers to process."
- print_summary
- exit 0
- fi
- echo "Pending: $pending_count offers"
- echo ""
- # Dry run: just list
- if [[ "$DRY_RUN" == "true" ]]; then
- echo "=== DRY RUN (no processing) ==="
- for i in "${!pending_ids[@]}"; do
- local status
- status=$(get_status "${pending_ids[$i]}")
- echo " #${pending_ids[$i]}: ${pending_urls[$i]} [${pending_sources[$i]}] (status: $status)"
- done
- echo ""
- echo "Would process $pending_count offers"
- exit 0
- fi
- check_evidence_freshness
- # --skip-pdf must not start a painter, even when the parent shell already opted in.
- if [[ "$SKIP_PDF" == "true" ]]; then
- unset CAREER_OPS_PDF_SHARED
- else
- export CAREER_OPS_PDF_SHARED=1
- PDF_SERVER_OWNER=1
- fi
- # Process offers
- BATCH_START=$SECONDS
- local done_jobs=0
- if (( PARALLEL <= 1 )); then
- # Sequential processing
- for i in "${!pending_ids[@]}"; do
- JOB_POS="$((i + 1))/$pending_count"
- process_offer "${pending_ids[$i]}" "${pending_urls[$i]}" "${pending_sources[$i]}" "${pending_notes[$i]}"
- done_jobs=$((done_jobs + 1))
- report_progress "$done_jobs" "$pending_count"
- if [[ "$BATCH_PAUSED" == "true" || -f "$PAUSE_FILE" ]]; then
- echo "=== Batch paused: session/rate limit reached. Resume later with --resume-paused. ==="
- break
- fi
- done
- else
- # Parallel processing with job control
- local running=0
- local parallel_rc=0 worker_rc=0
- local -a pids=()
- local -a pid_ids=()
- for i in "${!pending_ids[@]}"; do
- if [[ "$BATCH_PAUSED" == "true" || -f "$PAUSE_FILE" ]]; then
- echo "=== Batch paused: session/rate limit reached. Waiting for running workers, not scheduling new offers. ==="
- break
- fi
- # Wait if we're at parallel limit. The throttle check opens every pass,
- # so it also runs after the sleep below: a rate limit recorded while this
- # loop slept drops the run to one worker before the next offer starts,
- # not after it.
- while :; do
- if (( PARALLEL > 1 )) && [[ -f "${THROTTLE_FILE:-}" ]]; then
- PARALLEL=1
- echo "=== Rate limited: running the remaining offers on 1 worker. ==="
- fi
- if (( running < PARALLEL )); then
- break
- fi
- # Wait for any child to finish
- for j in "${!pids[@]}"; do
- if ! kill -0 "${pids[$j]}" 2>/dev/null; then
- worker_rc=0
- wait "${pids[$j]}" 2>/dev/null || worker_rc=$?
- if (( worker_rc != 0 )); then
- parallel_rc=$worker_rc
- fi
- unset 'pids[j]'
- unset 'pid_ids[j]'
- running=$((running - 1))
- done_jobs=$((done_jobs + 1))
- report_progress "$done_jobs" "$pending_count"
- fi
- done
- # Compact arrays
- pids=(${pids[@]+"${pids[@]}"})
- pid_ids=(${pid_ids[@]+"${pid_ids[@]}"})
- if (( parallel_rc != 0 )); then
- break
- fi
- if [[ "$BATCH_PAUSED" == "true" || -f "$PAUSE_FILE" ]]; then
- echo "=== Batch paused: session/rate limit reached. Waiting for running workers, not scheduling new offers. ==="
- break
- fi
- sleep 1
- done
- if (( parallel_rc != 0 )) || [[ "$BATCH_PAUSED" == "true" || -f "$PAUSE_FILE" ]]; then
- break
- fi
- # Launch worker in background
- JOB_POS="$((i + 1))/$pending_count"
- process_offer "${pending_ids[$i]}" "${pending_urls[$i]}" "${pending_sources[$i]}" "${pending_notes[$i]}" &
- pids+=($!)
- pid_ids+=("${pending_ids[$i]}")
- running=$((running + 1))
- done
- # Wait for remaining workers
- for pid in ${pids[@]+"${pids[@]}"}; do
- worker_rc=0
- wait "$pid" 2>/dev/null || worker_rc=$?
- if (( worker_rc != 0 )); then
- parallel_rc=$worker_rc
- fi
- done_jobs=$((done_jobs + 1))
- report_progress "$done_jobs" "$pending_count"
- done
- if (( parallel_rc != 0 )); then
- echo "ERROR: a parallel worker failed fatally; tracker merge skipped. Inspect the worker log and repair state before rerunning." >&2
- return "$parallel_rc"
- fi
- fi
- # Merge tracker additions
- merge_tracker
- # Print summary
- print_summary
- exit 0
- }
- main "$@"
lib/ats-board-url.mjs ยท Bundled script ยท 232 lines ยท 2 findings marked
- // @ts-check
- /**
- * lib/ats-board-url.mjs: the canonical careers URL of an ATS board, in the
- * exact shape that ATS's provider detect() reads.
- *
- * Two callers depend on these being the same functions:
- * - discover-ats.mjs builds every probe candidate with BOARD_URL_BUILDERS,
- * so a board it confirms is one the scanner can later read;
- * - the employer registry (employers/) keys every row on the URL
- * buildBoardUrl() returns, and seeds/employers-validate.mjs rejects a row
- * whose careers_url is anything else.
- * One definition means a registry row and a discovered board cannot drift
- * apart. discover-ats.mjs re-exports buildBoardUrl and SLUG_RE.
- *
- * Dependency-free on purpose (no npm packages, no provider modules): the
- * registry's CI gate runs the validator without `npm install`, the same way
- * the plugin registry gate runs its validator.
- */
- // Safe charset for a slug that will be interpolated into an ATS URL. Mixed
- // case is intentional: Ashby boards are case-sensitive (AlephAlpha, DeepL).
- export const SLUG_RE = /^[A-Za-z0-9._-]+$/;
- const lower = (/** @type {string} */ s) => String(s).toLowerCase();
- /**
- * slug โ careers_url on each vendor's default host. Vendors that put the
- * company in a subdomain are matched lowercase-only by their providers, so
- * their builder lowercases the slug rather than emit a URL the provider would
- * then reject.
- * @type {Readonly<Record<string, (slug: string) => string>>}
- */
- export const BOARD_URL_BUILDERS = Object.freeze({
- greenhouse: (s) => `https://job-boards.greenhouse.io/${s}`,
- ashby: (s) => `https://jobs.ashbyhq.com/${s}`,
- lever: (s) => `https://jobs.lever.co/${s}`,
- workable: (s) => `https://apply.workable.com/${s}`,
- smartrecruiters: (s) => `https://careers.smartrecruiters.com/${s}`,
- rippling: (s) => `https://ats.rippling.com/${s}/jobs`,
- join: (s) => `https://join.com/companies/${s}`,
- recruitee: (s) => `https://${lower(s)}.recruitee.com`,
- breezy: (s) => `https://${lower(s)}.breezy.hr`,
- bamboohr: (s) => `https://${lower(s)}.bamboohr.com`,
- pinpoint: (s) => `https://${lower(s)}.pinpointhq.com`,
- applicantpro: (s) => `https://${lower(s)}.applicantpro.com/jobs/`,
- hibob: (s) => `https://${lower(s)}.careers.hibob.com/jobs`,
- hirehive: (s) => `https://${lower(s)}.hirehive.com`,
- homerun: (s) => `https://${lower(s)}.homerun.co`,
- jazzhr: (s) => `https://${lower(s)}.applytojob.com/apply`,
- keka: (s) => `https://${lower(s)}.keka.com/careers`,
- personio: (s) => `https://${lower(s)}.jobs.personio.de`,
- prevueaps: (s) => `https://${lower(s)}.prevueaps.ca/jobs/`,
- schoolspring: (s) => `https://${lower(s)}.schoolspring.com`,
- taleez: (s) => `https://${lower(s)}.taleez.com`,
- teamtailor: (s) => `https://${lower(s)}.teamtailor.com`,
- trakstar: (s) => `https://${lower(s)}.hire.trakstar.com`,
- turbohire: (s) => `https://${lower(s)}.turbohire.co`,
- 'zoho-recruit': (s) => `https://${lower(s)}.zohorecruit.com/jobs/Careers`,
- });
- /**
- * Vendors that host boards in more than one region. A board in the EU region
- * is a different board from a US one with the same slug, so the region is part
- * of the URL and of the registry key. `us` is the default host above.
- * @type {Readonly<Record<string, Readonly<Record<string, (slug: string) => string>>>>}
- */
- export const REGIONAL_BOARD_URL_BUILDERS = Object.freeze({
- greenhouse: Object.freeze({
- us: BOARD_URL_BUILDERS.greenhouse,
- eu: (s) => `https://job-boards.eu.greenhouse.io/${s}`,
- }),
- lever: Object.freeze({
- us: BOARD_URL_BUILDERS.lever,
- eu: (s) => `https://jobs.eu.lever.co/${s}`,
- }),
- });
- export const REGIONS = Object.freeze(['us', 'eu']);
- /**
- * Vendors whose boards live on one default host (the builder above) or on one
- * of a few other hosts. The region names the other host; it is part of the URL
- * and of the registry key, and a board on the default host has no region.
- * - Zoho Recruit: <tenant>.zohorecruit.com by default; the region names its
- * other data-centre domains, au for .com.au and cn for .com.cn.
- * - Personio: <tenant>.jobs.personio.de by default (the host its provider
- * reads a bare tenant on); com for <tenant>.jobs.personio.com.
- * @type {Readonly<Record<string, Readonly<Record<string, (slug: string) => string>>>>}
- */
- export const OPTIONAL_REGION_BOARD_URL_BUILDERS = Object.freeze({
- personio: Object.freeze({
- com: (s) => `https://${lower(s)}.jobs.personio.com`,
- }),
- 'zoho-recruit': Object.freeze({
- au: (s) => `https://${lower(s)}.zohorecruit.com.au/jobs/Careers`,
- ca: (s) => `https://${lower(s)}.zohorecruit.ca/jobs/Careers`,
- cn: (s) => `https://${lower(s)}.zohorecruit.com.cn/jobs/Careers`,
- eu: (s) => `https://${lower(s)}.zohorecruit.eu/jobs/Careers`,
- in: (s) => `https://${lower(s)}.zohorecruit.in/jobs/Careers`,
- jp: (s) => `https://${lower(s)}.zohorecruit.jp/jobs/Careers`,
- sa: (s) => `https://${lower(s)}.zohorecruit.sa/jobs/Careers`,
- }),
- });
- /**
- * The regions a row of `ats` may name: required ones for greenhouse and lever,
- * optional ones for the vendors above, none elsewhere.
- * @param {string} ats
- * @returns {readonly string[]}
- */
- export function regionsOf(ats) {
- if (Object.hasOwn(REGIONAL_BOARD_URL_BUILDERS, ats)) return REGIONS;
- if (Object.hasOwn(OPTIONAL_REGION_BOARD_URL_BUILDERS, ats)) return Object.keys(OPTIONAL_REGION_BOARD_URL_BUILDERS[ats]);
- return [];
- }
- /** Vendors whose board id is the company's subdomain on the vendor's host. */
- export const SUBDOMAIN_ATS = Object.freeze([
- 'recruitee', 'breezy', 'bamboohr', 'pinpoint', 'applicantpro', 'hibob', 'hirehive', 'homerun', 'jazzhr', 'keka',
- 'personio', 'prevueaps', 'schoolspring', 'taleez', 'teamtailor', 'trakstar', 'turbohire', 'zoho-recruit',
- ]);
- /**
- * The subdomain vendors whose board id the builder and the validator check as
- * one DNS label of at most 63 characters, not one of the vendor's own
- * subdomains. recruitee, breezy, bamboohr and pinpoint, admitted first, keep
- * the rule they were admitted with.
- */
- export const DNS_LABEL_ATS = Object.freeze(
- SUBDOMAIN_ATS.filter((ats) => !['recruitee', 'breezy', 'bamboohr', 'pinpoint'].includes(ats)),
- );
- // One DNS label: letters, digits and inner hyphens, at most 63 characters.
- export const SUBDOMAIN_LABEL_RE = /^[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?$/;
- /**
- * Subdomains a vendor runs for itself, which its provider never reads as a
- * board: each list is the one in providers/<ats>.mjs. The validator and the
- * importer refuse them as board ids, and no URL is built for them.
- * @type {Readonly<Record<string, readonly string[]>>}
- */
- export const SUBDOMAIN_RESERVED_LABELS = Object.freeze({
- applicantpro: Object.freeze(['www', 'feeds']),
- hirehive: Object.freeze(['www', 'app', 'accounts', 'apply']),
- homerun: Object.freeze(['www', 'app', 'feed', 'help', 'static', 'cdn', 'api', 'developers', 'blog', '404']),
- keka: Object.freeze(['www', 'app', 'api', 'help', 'support', 'blog', 'docs', 'status', 'mail', 'developers']),
- schoolspring: Object.freeze(['www', 'api']),
- taleez: Object.freeze(['www', 'app', 'api', 'blog', 'files', 'static', 'help', 'aide', 'status']),
- trakstar: Object.freeze(['www', 'jsapi', 'api', 'app', 'mail', 'support', 'help', 'status', 'blog', 'docs']),
- turbohire: Object.freeze(['app', 'api', 'www']),
- });
- /**
- * Whether `board` is a board id a subdomain vendor can serve: one DNS label
- * (any letter case; the builders lowercase it) that is not one of the vendor's
- * own subdomains.
- * @param {string} ats
- * @param {string} board
- */
- export function isSubdomainBoard(ats, board) {
- const label = lower(board);
- if (!SUBDOMAIN_LABEL_RE.test(label)) return false;
- return !(Object.hasOwn(SUBDOMAIN_RESERVED_LABELS, ats) && SUBDOMAIN_RESERVED_LABELS[ats].includes(label));
- }
- // Workday needs three coordinates, not a slug: tenant, instance (wdN) and site.
- // The same board can be served from two host forms, so the form is recorded too:
- // jobs: https://<tenant>.<wdN>.myworkdayjobs.com/<site>
- // site: https://<wdN>.myworkdaysite.com/recruiting/<tenant>/<site>
- export const WORKDAY_HOSTS = Object.freeze(['jobs', 'site']);
- export const WORKDAY_SEGMENT_RE = /^[A-Za-z0-9_-]+$/;
- export const WORKDAY_INSTANCE_RE = /^wd[0-9]{1,3}$/;
- /** The closed list of ATS ids a registry row may name. */
- export const BOARD_ATS = Object.freeze([...Object.keys(BOARD_URL_BUILDERS), 'workday']);
- /**
- * Build a Workday careers URL from its coordinates.
- * @param {{tenant: string, instance: string, site: string, host?: string}} coords
- * @returns {string|null} null when a coordinate is malformed or the host form is unknown.
- */
- export function workdayBoardUrl({ tenant, instance, site, host = 'jobs' }) {
- if (typeof tenant !== 'string' || !WORKDAY_SEGMENT_RE.test(tenant)) return null;
- if (typeof site !== 'string' || !WORKDAY_SEGMENT_RE.test(site)) return null;
- if (typeof instance !== 'string' || !WORKDAY_INSTANCE_RE.test(instance)) return null;
- if (host === 'jobs') return `https://${tenant}.${instance}.myworkdayjobs.com/${site}`;
- Dismissed after review: Looks up a DNS name built from data, which carries it out past web filters
- Line 186 on GitHub
- if (host === 'site') return `https://${instance}.myworkdaysite.com/recruiting/${tenant}/${site}`;
- Dismissed after review: Looks up a DNS name built from data, which carries it out past web filters
- Line 187 on GitHub
- return null;
- }
- /**
- * The canonical careers URL for a registry row.
- *
- * `region` is required on the regional vendors (greenhouse, lever), optional on
- * personio and zoho-recruit (absent on the default host) and must be absent
- * elsewhere; `workday_host` is required on workday and must be absent
- * elsewhere; a Workday `board` is `tenant/wdN/site`; a subdomain board id
- * must be a host label its vendor serves. Anything that does not fit returns
- * null rather than a URL, so a caller never probes or stores a guess. The
- * builder checks the shape of the address only: the registry's admission rules
- * (length, numeric-only ids, placeholders, letter case) are the validator's.
- *
- * @param {{ats?: unknown, region?: unknown, board?: unknown, workday_host?: unknown}} row
- * @returns {string|null}
- */
- export function buildBoardUrl({ ats, region, board, workday_host } = {}) {
- if (typeof ats !== 'string' || typeof board !== 'string' || !board) return null;
- if (ats === 'workday') {
- if (region !== undefined || typeof workday_host !== 'string') return null;
- const parts = board.split('/');
- if (parts.length !== 3) return null;
- const [tenant, instance, site] = parts;
- return workdayBoardUrl({ tenant, instance, site, host: workday_host });
- }
- // Own-property lookups only: a plain index would find Object.prototype keys
- // ("constructor", "toString") and call them as builders.
- if (!Object.hasOwn(BOARD_URL_BUILDERS, ats)) return null;
- if (workday_host !== undefined || !SLUG_RE.test(board)) return null;
- if (DNS_LABEL_ATS.includes(ats) && !isSubdomainBoard(ats, board)) return null;
- if (Object.hasOwn(REGIONAL_BOARD_URL_BUILDERS, ats)) {
- const byRegion = REGIONAL_BOARD_URL_BUILDERS[ats];
- if (typeof region !== 'string' || !Object.hasOwn(byRegion, region)) return null;
- return byRegion[region](board);
- }
- if (region !== undefined && Object.hasOwn(OPTIONAL_REGION_BOARD_URL_BUILDERS, ats)) {
- const byRegion = OPTIONAL_REGION_BOARD_URL_BUILDERS[ats];
- if (typeof region !== 'string' || !Object.hasOwn(byRegion, region)) return null;
- return byRegion[region](board);
- }
- if (region !== undefined) return null;
- return BOARD_URL_BUILDERS[ats](board);
- }
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If itโs helped, pass it on.