Back to search
classifier-dev
SkillGitHubIncomplete
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Checked today
Badge for your README
Markdown
[](/listings/mrmps--classifier-dev)HTML
<a href="/listings/mrmps--classifier-dev"><img src="/api/listings/mrmps--classifier-dev/badge.svg" alt="findsafeskills safety rating" /></a>Install
i
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
.claude-plugin/marketplace.json · Manifest · 27 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
- {
- "$schema": "https://json.schemastore.org/claude-code-marketplace.json",
- "name": "classifier-dev",
- "owner": {
- "name": "Michael Ryaboy",
- "email": "contact@classifier.dev",
- "url": "https://classifier.dev"
- },
- "metadata": {
- "description": "classifier.dev classification tools and skills for coding agents.",
- "version": "1.1.0"
- },
- "plugins": [
- {
- "name": "classifier",
- "source": "./plugins/classifier",
- "description": "Classify batches of text into your labels, tag multiple topics, count categories and review uncertain results. Public access needs no API key.",
- "version": "1.1.0",
- "author": { "name": "Michael Ryaboy", "email": "contact@classifier.dev", "url": "https://classifier.dev" },
- "homepage": "https://classifier.dev/mcp-setup",
- "repository": "https://github.com/mrmps/classifier-dev",
- "license": "MIT",
- "category": "productivity",
- "keywords": ["classification", "triage", "filtering", "routing", "labels", "zero-shot"]
- }
- ]
- }
README.md · README · 531 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
- # classifier.dev
- Zero-shot text classification. Plain text in, a label and a calibrated
- confidence out. No key, no signup. Up to a thousand texts per request.
- curl https://classifier.dev/spam,not+spam/Win+a+free+iPhone
- spam
- curl "https://classifier.dev/?labels=spam,not+spam&text=Win+a+free+iPhone" # same call, query form
- spam
- Single Cloudflare Worker. No database, no framework, no build step beyond esbuild.
- ## CLI
- npm i -g classifier-dev
- classify bug,feature,praise < feedback.txt
- `cli/` is a separate npm package (`classifier-dev`, bin `classify`): one
- dependency-free Node file, tests against a mock API (`npm test`), semver with
- its own CHANGELOG, released with `npm run release patch|minor|major` which
- tags `cli-v<version>` and lets `.github/workflows/publish-cli.yml` publish
- (needs an `NPM_TOKEN` repo secret). It talks to the API exactly like curl does.
- ## Layout
- src/index.ts routing, validation, tiers, LLM fallback chain, analytics
- src/query.ts the GET query form, read and written with nuqs; the URL an error suggests
- src/jev.ts TypeSafe's Jev: packs inputs into requests, reads probabilities
- src/limiter.ts Durable Object: per-IP rate limiting
- src/report.ts digest — Analytics Engine SQL -> Resend, flags model fallbacks
- src/alerts.ts every 15 minutes; emails only when something is wrong
- src/feedback.ts agent feedback, feedback.now protocol -> email
- src/privacy.ts keyed pseudonyms: nothing kept points back at a caller
- src/cost.ts per-request upstream spend, from the providers' own accounting
- src/docs.ts the site (GET / and GET /benchmark), plain text
- src/home.ts the same two documents rendered, for browsers only
- src/ui.ts the shared look: markdown in a terminal
- cli/ the `classify` command, published to npm as classifier-dev
- eval/ benchmarks; read eval/README.md before quoting a number
- finish-dns.sh one-shot DNS wiring, see below
- wrangler.example.toml the Worker config, minus the account-specific ids
- ## The site
- `curl classifier.dev` prints plain text, exactly as it always has. A browser
- sends `Accept: text/html` and gets the same document rendered — headings,
- bracketed links, copy buttons — from `src/home.ts`. Nothing is duplicated: the
- page is generated from `DOCS` and `BENCHMARK` at request time, so the text
- stays canonical and the two cannot drift. `?format=text` opts out by hand, and
- both responses carry `Vary: accept`.
- ## Agent feedback
- Implements the [feedback.now](https://feedback.now) protocol (schema 1.1), so
- any agent that speaks it can report a problem without being told how:
- GET /.well-known/agent-feedback.json what this host accepts
- GET /api/v1/policy categories, severities, limits
- POST /api/v1/feedback full structured report
- POST /api/v1/observations lighter signal
- POST /api/v1/feedback/{id}/attachments more evidence, later
- GET /api/v1/receipts/{id} did it land, and was it any good
- Accepted submissions are emailed to `REPORT_TO`. Reports are kept in KV for 90
- days. A repeat of the same domain + surface + category + title is stored and
- acknowledged as a duplicate but not emailed again, so one looping agent cannot
- empty itself into the inbox; the hourly budget is 100 per IP and the remainder
- comes back on every receipt.
- Testimonials use `signal.category: "testimonial"`. They require
- `reporter.agent_type` and `reporter.agent_description`, so praise arrives with
- enough context to understand what kind of agent benefited and how it works.
- `quality_score` is a deterministic function of how complete the report is — an
- agent can read the rule and write a better one next time. Nothing here calls
- the classifier or Analytics Engine: this is where reports arrive saying those
- are broken, so it must work when they do not.
- ## Deploy
- Merging to `main` deploys. `.github/workflows/deploy.yml` typechecks, runs the
- Worker and CLI tests, runs `wrangler deploy`, and then asks the live service for
- `/v1/health` and one classification, so a deploy that uploads a broken Worker
- fails in CI rather than in somebody's terminal.
- By hand, to try something before it is merged:
- cp wrangler.example.toml wrangler.toml # once, then fill in your own ids
- npx wrangler deploy
- `wrangler.toml` is gitignored and holds the two values that are specific to one
- Cloudflare account: `account_id`, and the `STATS` KV namespace id that
- `npx wrangler kv namespace create STATS` hands back. The tracked
- `wrangler.example.toml` carries everything else — crons, bindings, migrations —
- so the deployment shape is in the repository and only the identifiers are not.
- CI has no `wrangler.toml`, so `.github/render-wrangler.mjs` writes one from the
- example and three repository secrets. That makes the example the deployed shape
- rather than a copy of it: change a binding in `wrangler.toml` alone and CI keeps
- deploying the old one.
- Repository secrets the deploy needs:
- CLOUDFLARE_API_TOKEN dash.cloudflare.com > My Profile > API Tokens >
- Create Token > "Edit Cloudflare Workers"
- CLOUDFLARE_ACCOUNT_ID the account_id from wrangler.toml
- STATS_KV_ID the STATS namespace id from wrangler.toml
- REPORT_TO where the daily digest goes
- Secrets set with `wrangler secret put` live on the Worker, not in the script
- bundle, so a deploy leaves them alone and CI never needs to know them.
- Secrets the Worker reads: `TYPESAFE_API_KEY`, `AI_GATEWAY_API_KEY` (Vercel's
- AI Gateway, which serves Jev on a free monthly credit; when set it is asked
- first and TypeSafe catches what it refuses), `OPENROUTER_API_KEY`,
- `CONTEXT_API_KEY` (context.dev, the chat's web search and page reads),
- `RESEND_API_KEY`, `CF_ANALYTICS_TOKEN`, `REPORT_KEY`, `PRIVACY_SALT`. Add one
- with `npx wrangler secret put NAME`; none of them are ever read from the
- repository. `src/index.ts` lists the rest in the `Env` interface.
- Secrets are compared with `secretEquals` (src/secrets.ts), never `===`: a
- plain comparison returns on the first wrong byte and tells a caller how much
- of a guess was right.
- ## The model
- Both tiers answer from [TypeSafe's Jev](https://docs.typesafe.ai), a decision
- model rather than a language model: it takes a state and typed questions and
- returns a calibrated probability per option, in ~150ms. That shape is why the
- API can do three things the LLM version could not.
- **A thousand inputs per request.** State is an array of `{id, text}` and each
- input gets its own question, so the whole batch is one upstream call. The
- documented limit is 64k tokens per request; `jev.ts` packs to a conservative
- budget and runs the resulting requests eight at a time. Measured: 400 news
- headlines classified in 650ms end to end, and packing 100 items scored the
- same as sending them one at a time.
- **Confidence that means something.** On 400 six-way emotion items, answers at
- >= 0.9 confidence were right 82% of the time and answers below 0.5 were right
- 29%. The previous model's logprob "confidence" put 87% of news items above 0.9
- and was right on 68% of those. So `tier: "smart"` now means: re-ask the
- single-label answers below 0.7 of a fast reasoning model and replace them,
- marked `escalated: true`. Nothing else changes. Which model matters: on
- exactly the items Jev is unsure about, deepseek-v4-flash, qwen3.7-flash and
- mercury-2.5 were no better than Jev; gemini-3.8-flash took news topics from
- 87.5% to 90.0% and emotion from 61.8% to 63.7%, so that is the chain. A
- frontier model (claude-fable-5.1) gets 72.3% / 90.7% at ~3x the price; the
- numbers are on /benchmark if that trade ever looks worth it.
- **Multi-label in one pass.** One yes/no question per label, labels at >= 0.7
- returned most-likely-first with the full score map. F1 0.887 on the seven-case
- set against 0.799 for the sweep-and-verify LLM cascade it replaced, in 230ms
- instead of 1.5s. Re-judging its candidates with the reasoning model made it
- worse (and took 23s), so multi-label ignores the tier.
- The LLM chains in `index.ts` remain as the fallback when TypeSafe is
- unavailable, limited to twenty inputs because they are one call per input.
- The digest reports which model actually answered, with a `FALLBACK` marker,
- because the previous primary was delisted upstream and served its backup for
- weeks at F1 0.546 without anything saying so.
- ## Updates list
- The form and `POST /subscribe` send a confirmation email through Resend.
- The API returns `202 {"ok":true,"status":"pending_confirmation"}`. Nothing is
- written to the newsletter database until the emailed token is submitted to
- `POST /subscribe/confirm` as `{"token":"..."}` (or with the confirmation form).
- GET only renders the form, so mail scanners cannot confirm subscriptions.
- Tokens are signed with `NEWSLETTER_CONFIRMATION_SECRET`, expire within 24 hours,
- and are never returned from signup. Resend's idempotency key deduplicates repeat
- requests for the same inbox within each clock hour. Existing per-IP limits also
- apply. Signing-key rotation invalidates outstanding links.
- Apply `migrations/postgres/` with `npm run db:migrate` before deploying. The
- subscriber table shares the application database. Existing unconfirmed subscribers stay unconfirmed; do not
- backfill `confirmed_at` or send them updates until they confirm. An existing
- unsubscribe is never cleared by confirmation or by replaying an old token.
- Required Worker secrets: `DATABASE_URL`, `NEWSLETTER_RESEND_API_KEY`, and a
- random `NEWSLETTER_CONFIRMATION_SECRET` of at least 32 bytes. `NEWSLETTER_FROM`
- in `wrangler.example.toml` must use a verified Resend sending domain. `REPORT_TO`
- is the reply address and receives notifications only for newly confirmed rows.
- The subscriber table holds email, source, signup/confirmation/unsubscribe dates, which
- roadmap items were ticked (`wants text[]`, holding `ROADMAP` keys; the ticks
- ride in the confirmation token and are written on confirmation), the requested
- latency when faster inference is selected, and an internal id. It holds no IP,
- request id, or classification traffic. Pending
- signups are not stored. Tokens and mail-provider error bodies must not be logged.
- `migrations/postgres/0007_newsletter.sql` defines the table; a repeat confirmation
- replaces the ticks only when it ticked something, and never clears an
- unsubscribe or moves the first confirmation date.
- Read only confirmed, active recipients when sending updates:
- SELECT email, wants, desired_latency_ms FROM subscriber
- WHERE confirmed_at IS NOT NULL AND unsubscribed_at IS NULL;
- What people asked for first, to order the work by:
- SELECT unnest(wants) AS item, count(*) FROM subscriber
- WHERE unsubscribed_at IS NULL GROUP BY 1 ORDER BY 2 DESC;
- The Worker uses the application `DATABASE_URL` for newsletter and account data.
- Subscriber rows have no account foreign key or API traffic identifier; sharing
- storage does not subscribe account holders or change existing consent. The old
- newsletter project is retained as a read-only migration archive. See
- `docs/postgres-setup.md` for the verified cutover and recovery procedure.
- The copy is one constant — `ROADMAP` in `src/newsletter.ts`. The plain text at
- `curl classifier.dev`, the form on the rendered page and `index.md` all read it,
- so a change to the roadmap changes all three or none.
- ## Jev long context
- Default Jev and explicit `model: "jev"` inputs over 32,000 characters use
- Chonkie RecursiveChunker with 600-token `cl100k_base` chunks, parallel Jev
- screening for relevant or uncertain evidence (including opposing evidence and
- exceptions), then final Jev over whole eligible chunks in source order.
- The final evidence budget is 20,000 `cl100k_base` tokens plus a conservative
- provider context check. Eligible chunks can be omitted when that budget fills;
- `usage.long_context` reports selection counts, tokens, calls and timing.
- This does not guarantee full-document final reading or universal accuracy.
- Use Fast with a workspace key backed by paid balance or an active paid
- subscription; anonymous access and signup credit do not qualify. Synchronous limits:
- 250,000 original context tokens summed across inputs, 20 documents, 32 decisions,
- and a 1 MB body. Retail is $0.084/M original `cl100k_base` context tokens,
- counted once across inputs regardless of dimensions and actual inference usage.
- Each continuous whitespace or non-whitespace run is limited to 8,192 UTF-16
- code units. Longer runs return `400 long_context_input` before document
- tokenization or chunking, bounding tokenizer work on pathological inputs.
- No eligible evidence returns `422 long_context_no_evidence` without charge.
- Explicit `model: "chunklaya"` keeps its separate legacy opt-in behavior.
- Send one whole document of up to 10 million input tokens (100 MB) to
- `POST /v1/classify` with a funded workspace key. Use the normal JSON input and
- labels, or send `text/plain` with a `labels` query parameter. Large uploads
- return `202` with a `status_url`; `Prefer: respond-async` also works for smaller
- documents. The server splits, screens, retries and finishes automatically.
- The repository CLI handles upload and polling with
- `node cli/classify.js a,b --document document.txt --json`.
- The exact whole-document token count sets the reservation and $0.084/M charge.
- Source text is stored privately while queued and deleted as it is screened.
- Final Jev reads at most 20,000 selected evidence tokens. Completion, failure,
- cancellation and 24-hour expiry delete remaining source and evidence; failed
- or canceled work is refunded. Results remain available until expiry.
- ## Analytics
- Every request writes one Analytics Engine datapoint (tier, label-set fingerprint,
- country, status, count, latency). No request text is written to analytics.
- Long-context aggregates also go to `classifier_long_context_events`, with
- context/document totals, chunk screening/selection/omission counts, phase input
- tokens (including unknown-count flags), answered-call counts and phase timings.
- Account events append the corresponding fields; their ordered column contract
- is in [`src/server/analytics/schema.ts`](src/server/analytics/schema.ts).
- Documents and original context tokens count once; chunk and phase counters
- accumulate across dimension passes. No document text or caller identifiers are
- written to the dedicated dataset.
- Every request also records what it cost us: OpenRouter returns the charge for
- a call when asked, and Jev is billed on the input tokens it reports, at the
- rate `eval/bench.py` prices the benchmarks with. Spend accumulates in a
- per-request meter (`src/cost.ts`) and lands in `double3`. That column was added
- after launch, so it reads 0 for anything older than that deploy.
- A cron at 15:00 UTC queries it and emails a digest via Resend.
- ### Alerts
- A separate cron runs every fifteen minutes and stays silent unless something
- fires. It only watches conditions with an action attached: the Jev key being
- refused, Jev not answering (the fallback chain serving quietly, which has
- happened), 5xx rates, smart-tier escalations failing (the shape an exhausted
- `OPENROUTER_API_KEY` takes), mean latency, a spend spike against the trailing
- day, and traffic stopping outright. 4xx is ignored — that is scanners probing
- for `/wp-admin`, not a fault.
- **Jev credits.** TypeSafe publishes no balance endpoint — its API is
- `/v1/systemone` and `/v1/models`, nothing else — so there is no number to
- watch. Instead the check calls `/v1/models` with the key every fifteen
- minutes and reports back whatever TypeSafe says: a 401, 402 or 403 there means
- out of credit, revoked or wrong, and raises a critical alert quoting TypeSafe's
- own message rather than guessing which status means what. Because it probes
- rather than waiting for traffic, it fires on a quiet host before any caller
- meets the fallback chain, and it runs even when Analytics Engine is down.
- Each condition emails once when it starts, again every six hours while it
- lasts, and once when it clears, with the state in KV under `alert:`. Thresholds
- are the `T` object at the top of `src/alerts.ts`.
- curl -H "authorization: Bearer $REPORT_KEY" https://classifier.dev/alerts
- curl -H "authorization: Bearer $REPORT_KEY" "https://classifier.dev/alerts?demo=1&send=1"
- The first previews without sending or touching state; the second emails a
- sample through the real path, to prove delivery works.
- Preview the digest any time without sending it:
- curl -H "authorization: Bearer $REPORT_KEY" https://classifier.dev/report
- Jev provider attempts are stored separately in `classifier_jev_attempts` through
- `JEV_AE`, including recovered failures, retries and gateway cooldown skips.
- The report includes status, reason, count and mean latency for each provider;
- `/alerts` shows current incidents even when their notification is suppressed.
- The existing 15-minute alert check warns when at least three attempts fail and
- failures exceed 5% for either provider. Counts account for Analytics Engine
- sampling. No input text, caller identifiers or upstream messages are stored.
- Successful simple and multi-label classifier names are retained for 90 days in
- a separate aggregate KV record with no caller or source-text fields. Its keyed
- fingerprint also appears in per-request analytics, allowing operators to resolve
- label names in those pseudonymous records.
- `AI_GATEWAY_DISABLED = "true"` in `wrangler.example.toml` keeps production on
- TypeSafe directly after the gateway repeatedly returned 429 on September 19.
- The gateway key is retained. To restore gateway-first routing, verify gateway
- capacity, change this variable to `"false"` in the example and local config,
- and deploy. Check provider attempts and the live API tests after re-enabling.
- The header is the only way in. A query string lands in access logs, in browser
- history and in the Referer header of whatever gets clicked next, so `?key=` is
- gone. Append `?send=1` to actually email it.
- Cloudflare's Analytics Engine SQL is a narrow ClickHouse subset — no `uniq()`,
- no `SELECT DISTINCT`, and a bare `SELECT col ... GROUP BY col` is rejected.
- Distinct counts therefore use `SELECT col, count() ... GROUP BY col` and count
- the returned rows. Each query is isolated so one failure cannot blank the report.
- ### Privacy
- Two columns in that dataset used to be the caller: the IP address, and the
- label set, joined and lowercased. Both are keyed hashes now (`src/privacy.ts`),
- so the figures still count distinct callers and distinct classifiers and
- nothing can be read back into an address or into somebody's wording. The caller
- hash takes the UTC day as well, so it stops being the same value tomorrow —
- which is why a unique-caller count over 7d or 30d is really caller-days.
- Set the key once, and treat it as a secret like any other:
- npx wrangler secret put PRIVACY_SALT # 32 random bytes
- Rotating it renumbers every fingerprint, so distinct counts double-count across
- the rotation. It falls back to `ADMIN_SIGNING_KEY`, then `REPORT_KEY`, then a
- per-isolate random value, because an unkeyed hash of an IPv4 address or of a
- common label set inverts in seconds.
- ## Eval
- npm run bench # multi-label, 7 cases: jev vs any OpenRouter model
- npm run single -- --dataset emotion --backend jev
- npm run single -- --dataset ag_news --backend openrouter:qwen/qwen3.7-flash
- python3 eval/escalate.py --dataset emotion # what the smart tier buys
- `single.py` downloads AG News and dair-ai/emotion test rows on first use and
- caches raw results under `eval/data/results/` so `escalate.py` can combine
- backends without re-spending. `eval/README.md` lists the caveats.
- ## Rate limiting
- Per IP in a Durable Object, counted in classifications: 3,000/min and
- 20,000/day on fast, 200/min and 2,000/day on smart.
- Two other approaches were tried and rejected:
- - Cloudflare's native `ratelimit` binding registers fine but never decremented
- (70 calls against a limit of 60 all returned `success: true`).
- - KV is edge-cached and eventually consistent, so a counter written this second
- is invisible to the next read — every request saw `remaining: 59`.
- A Durable Object is single-threaded and strongly consistent, which is what a
- counter needs. Verified at the original 60/min: 75 requests -> 60 × 200, 15 × 429.
- ## DNS
- The domain is registered at Porkbun; the Worker is on Cloudflare. Cloudflare
- Workers custom domains require the zone to live in Cloudflare, and neither API
- token here has `zone.create`, so that one step is manual:
- 1. https://dash.cloudflare.com -> Add a domain -> `classifier.dev` -> Free plan
- 2. `./finish-dns.sh` — reads the assigned nameservers, points Porkbun at them
- via the Porkbun API, and attaches the Worker to the apex and `www`.
- ## Agent skill
- npx skills add https://classifier.dev
- Served from this domain over RFC 8615 well-known discovery, so there is no
- repository in the middle:
- src/SKILL.md the skill, bundled as a Text module
- GET /skill.md the artifact
- GET /.well-known/agent-skills/index.json discovery, schema v0.2.0
- The index must carry a sha256 of the artifact, and an index that disagrees with
- the file makes the skill uninstallable. Rather than commit a digest that a later
- edit would silently invalidate, `src/skill.ts` hashes the bytes it actually
- serves, once per isolate. Editing SKILL.md is therefore enough; nothing else
- needs updating.
- The skill teaches the case the API pitch misses: you are already a model and can
- classify anything you can see, so the reason to call out is context, not
- capability — filtering forty search results down to six without reading forty.
- Two things it documents because testing found them the hard way. Cloudflare
- 403s Python's stdlib `urllib` User-Agent before the request reaches the Worker,
- so the recipe sets one explicitly. And filters should be told "when in doubt,
- keep it": on a ten-snippet research filter that took signal kept from 4/6 to
- 6/6 with no extra noise, where adding a third "possibly relevant" label did
- nothing.
- ## Discovery surfaces
- GET /openapi.json OpenAPI 3.1, also at /.well-known/openapi.json
- GET /llms.txt short index for agents, linked from robots.txt
- GET /benchmark measured accuracy, cost, latency
- Both are linked from the third paragraph of `GET /` so an agent reading the
- landing page finds them immediately.
- ## Known issue: Cloudflare's managed robots.txt
- Adding the zone enabled Cloudflare AI Crawl Control, which prepends a managed
- block to `/robots.txt` disallowing GPTBot, ClaudeBot, CCBot, Google-Extended,
- Bytespider, Amazonbot and meta-externalagent. The Worker's own robots.txt is
- appended after it and cannot override it.
- This blocks *training* crawlers, not runtime API consumers — any agent can still
- call the API. But it does keep the docs out of future model training data, which
- works against discovery. The toggles at
- dash.cloudflare.com -> classifier.dev -> AI Crawl Control -> Security did not
- persist when flipped, so this likely needs a plan-level change or support.
- ## Operator agent access
- For operator-owned bulk agent work, set a dedicated `AGENT_API_KEY` Worker secret
- and send it as `Authorization: Bearer ...`. Classification quotas are unmetered,
- but inference uses a separate $2/day operator allowance (`OPERATOR_DAILY_USD`),
- shared across IPs, with the same request and concurrency limits as free usage.
- It neither consumes nor expands the anonymous allowance and does not replace
- `ENTERPRISE_API_KEY`. The CLI accepts it
- through `CLASSIFY_API_KEY` or `CLASSIFIER_API_KEY`. It does not authorize private
- reports or admin access. Keep it in an ignored secret file; never give it to
- public clients. Anonymous quotas continue to apply to unauthenticated traffic.
- For a funded workspace key with an arranged bulk quota, set the
- `API_KEY_RATE_LIMIT_MULTIPLIERS` Worker secret to a JSON object mapping the
- SHA-256 hex digest of each key to an integer multiplier (1–1,000). Overrides only
- raise the admission quota for that key; account counters, balance charging,
- model capacity limits, and other keys remain unchanged. Remove the entry to
- restore the plan quota.
- ## Finding the classification code
- The domain terms are defined in [CONTEXT.md](CONTEXT.md).
- | Module | Responsibility |
- | --- | --- |
- | `src/index.ts` | HTTP validation, quota, model selection, smart escalation, fallback, and response formatting. |
- | `src/jev.ts` | Jev questions and answers, shared batch budgets and recovery, gateway/direct transport, and provider validation. |
- | `src/dimensions.ts` | Dimension definitions and the mapping from input–dimension decisions to Jev questions and back. |
- Both ordinary and dimension classification use the same Jev batching module.
- A question group is the smallest part of a request that recovery keeps together:
- one input's questions for ordinary classification, one decision for dimensions.
- Batch preparation accounts for shared input text once, checks both provider
- budgets, and runs before quota charging for dimensions. Execution limits
- concurrent requests and splits an oversized batch between question groups.
- Change provider budgets and recovery in `src/jev.ts`; keep dimension meaning in
- `src/dimensions.ts`. Test the posted requests and returned classifications through
- `jevClassify` and the Worker HTTP interface, without depending on packing internals.
- ## Multiple dimensions
- `POST /v1/classify` also accepts `items` and `dimensions`:
- ```json
- {
- "items": ["Checkout charges me twice"],
- "dimensions": {
- "team": ["billing", "identity", "platform"],
- "urgency": {
- "labels": ["immediate", "normal", "low"],
- "instructions": "Active financial harm is immediate."
- },
- "kind": ["bug", "request", "question"]
- }
- }
- ```
- Each `results[i].dimensions[name]` carries its own label, confidence, scores,
- model and latency. Jev shares state across item–dimension questions, packing
- both context limits. Smart escalation is per field and clears the original
- scores; unavailable Jev falls back only up to 20 decisions. The API accepts
- up to 20 dimensions and 1,000 decisions, with each decision charged to quota.
- The `classify_dimensions` MCP tool uses this same path.
- Analytics adds `blob9` (single/multi/dimensions), `double6` (successful input
- items), `double7` (dimension count), `double8` (uncertain or unscored fields),
- and `double9` (fields served by fallback). `double1` counts successful decisions.
- Dimension configurations are keyed fingerprints, never stored verbatim. The
- admin panel queries dimension traffic separately, including failures and
- caller-days. The existing alert cron flags three or more dimension 5xx in an
- hour above 10% of dimension requests, plus any dimension fallback usage.
- Run `npm run test:e2e` with Node 22.18+ against the real production API, or set
- `CLASSIFIER_BASE_URL` to a staging origin. These are named TypeScript tests
- using `node:test` and real HTTP/model calls, with no fetch or inference mocks.
- Set `CLASSIFIER_API_KEY` to an authorized key when the shared IP quota is used
- up; it is optional. The suite uses inference and counts toward normal quotas.
- The tests check exact decisions and ordering for a 300-decision batch, every
- field's allowed labels and probability distribution, real smart escalation,
- provider score preservation, aliases, validation errors, MCP, OpenAPI, and
- legacy calls.
- Responses start as `unknown` and are validated before becoming typed matrices.
- Smart tests require an actual escalation; model drift that makes every fixture
- confident fails the test instead of silently skipping the reasoning provider.
- `npm run typecheck` checks the Worker and the TypeScript live tests. Deployment
- CI runs the live suite after publishing; `npm test` stays offline. Fault
- injection remains in `test/dimensions.test.ts` and
- `test/dimensions-observability.test.ts`.
- Run the **live API tests** workflow manually from GitHub Actions (or
- `gh workflow run e2e.yml`) to check production without deploying a Worker.
src/skillscan.ts · Bundled script · 373 lines · 1 finding marked
- /**
- * The static pass over a submitted skill: the cleaners.
- *
- * A SKILL.md is instructions an agent will follow with a shell in its hand, so
- * the first review is the one no model can be talked out of. The rules mirror
- * what the open-source skill scanners look for (Cisco's skill-scanner, NVIDIA's
- * SkillSpector, Snyk's agent-scan, kurtpayne/skillscan-security): instruction
- * override and hidden text, credential harvesting and exfiltration, remote
- * code that is fetched and run, destructive and persistence commands, embedded
- * secrets, obfuscated blobs and suspicious links. Each rule is a regular
- * expression with a severity: `block` ends the review before a model sees the
- * text, `warn` is handed to the judge as context and shown on the page.
- *
- * Deliberately blunt. A rule that fires on a benign skill costs the submitter
- * one rewrite; a rule that misses a malicious one costs somebody a laptop.
- *
- * Nothing here is a substitute for the two model passes that follow; it is the
- * pass that makes those passes safe to run on untrusted text.
- */
- export type Severity = "block" | "warn";
- export type Finding = {
- /** Stable id, in the vocabulary the open-source scanners use. */
- rule: string;
- severity: Severity;
- /** One sentence, addressed to the submitter. */
- message: string;
- /** 1-based line of the first hit, when the rule is line-scoped. */
- line?: number;
- /** The matched text, trimmed to something a page can show. */
- excerpt?: string;
- };
- export const MAX_SKILL_CHARS = 24_000;
- export const MIN_SKILL_CHARS = 200;
- export const MAX_LINE_CHARS = 2_000;
- type Rule = { id: string; severity: Severity; re: RegExp; message: string };
- /**
- * Line-scoped rules. Every regex is case-insensitive and applied to one line at
- * a time, so a hit has a line number and an excerpt.
- */
- const RULES: Rule[] = [
- // ---- instruction override and hidden text (prompt injection) ----------
- {
- id: "prompt_injection.override",
- severity: "block",
- re: /\b(ignore|disregard|forget|override)\b[^.\n]{0,40}\b(previous|prior|earlier|above|all|any|system|developer)\b[^.\n]{0,30}\b(instructions?|prompts?|rules?|guidelines?|constraints?)\b/i,
- message: "Tells the agent to ignore or override its existing instructions.",
- },
- {
- id: "prompt_injection.persona",
- severity: "block",
- re: /\b(you are now|from now on you are|act as if you have no|pretend (you|that you) (are|have) no|DAN mode|jailbreak|developer mode enabled)\b/i,
- message: "Tries to replace the agent's identity or switch off its safeguards.",
- },
- {
- id: "prompt_injection.secrecy",
- severity: "block",
- re: /\b(do not|don't|never)\b[^.\n]{0,30}\b(tell|inform|mention|reveal|show|disclose|alert)\b[^.\n]{0,30}\b(the )?(user|human|operator|owner)\b/i,
- message: "Asks the agent to hide what it is doing from the user.",
- },
- {
- id: "prompt_injection.no_confirm",
- severity: "warn",
- re: /\b(without|skip|bypass|no need for)\b[^.\n]{0,20}\b(asking|confirmation|confirming|permission|approval|review)\b/i,
- message: "Asks the agent to skip confirmation or permission steps.",
- },
- {
- id: "prompt_injection.hidden_html",
- severity: "block",
- re: /<!--[\s\S]{0,200}\b(instruction|ignore|must|always|never|secret|run|execute|curl|send)\b/i,
- message: "An HTML comment carries instructions a reader of the rendered page would not see.",
- },
- {
- id: "prompt_injection.hidden_markup",
- severity: "block",
- re: /<(script|iframe|object|embed|img|svg|style|meta|link|form)\b/i,
- message: "Active HTML is not allowed in a skill; write Markdown.",
- },
- // ---- credential harvesting and exfiltration --------------------------
- {
- id: "exfiltration.credential_paths",
- severity: "block",
- re: /(~|\$HOME|\/home\/\w+|\/Users\/\w+|\/root)\/\.(ssh|aws|gnupg|gcloud|azure|kube|docker|npmrc|netrc|pypirc|git-credentials|config\/gh|claude\/(settings|credentials))\b|\/etc\/(shadow|passwd)\b|id_(rsa|ed25519|ecdsa)\b|\.env\b[^.\n]{0,40}\b(cat|read|send|upload|post|curl|print|echo)\b|\b(cat|read|send|upload|post|curl|print|echo)\b[^.\n]{0,40}\.env\b/i,
- message: "Reads or sends credential files (~/.ssh, ~/.aws, .env, /etc/shadow and the like).",
- },
- {
- id: "exfiltration.env_dump",
- severity: "block",
- re: /\b(printenv|env\s*\|\s*(curl|nc|base64)|set\s*\|\s*curl|export -p|Get-ChildItem Env:|process\.env\b[^.\n]{0,40}\b(fetch|post|send|http))\b/i,
- message: "Dumps environment variables, which is where keys live.",
- },
- {
- id: "exfiltration.send_secret",
- severity: "block",
- re: /\b(api[_ -]?key|secret|token|password|credential|private key|cookie|session)s?\b[^.\n]{0,60}\b(send|post|upload|submit|transmit|forward|exfil|paste)\b[^.\n]{0,40}\b(to|at|via)\b|\b(send|post|upload|submit|transmit|forward|paste)\b[^.\n]{0,40}\b(api[_ -]?key|secret|token|password|credential|private key|cookie|session)s?\b[^.\n]{0,40}\b(to|at|via)\b/i,
- message: "Sends keys, tokens or passwords somewhere.",
- },
- {
- id: "exfiltration.webhook",
- severity: "block",
- re: /https?:\/\/[^\s)"'`]*\b(webhook\.site|requestbin|pipedream\.net|ngrok(-free)?\.(app|io|dev)|burpcollaborator|interact\.sh|oast\.(fun|live|me|online|pro|site)|beeceptor|hookbin|postb\.in|localtunnel|trycloudflare\.com|serveo\.net)\b/i,
- message: "Links to a request-capture or tunnel service, the usual destination for exfiltrated data.",
- },
- {
- id: "exfiltration.discord_telegram",
- severity: "block",
- re: /https?:\/\/(discord(app)?\.com\/api\/webhooks|api\.telegram\.org\/bot|hooks\.slack\.com\/services)\//i,
- message: "Posts to a chat webhook; a skill has no reason to phone home.",
- },
- {
- id: "exfiltration.ip_literal",
- severity: "block",
- re: /https?:\/\/(\d{1,3}\.){3}\d{1,3}(?![\d.])(?!.*\b(127\.0\.0\.1|0\.0\.0\.0)\b)/i,
- message: "Links to a bare IP address rather than a named host.",
- },
- {
- id: "exfiltration.shortener",
- severity: "warn",
- re: /https?:\/\/(bit\.ly|tinyurl\.com|t\.co|goo\.gl|is\.gd|cutt\.ly|rb\.gy|shorturl\.at|tiny\.cc|ow\.ly)\//i,
- message: "Uses a URL shortener; link to the real destination.",
- },
- {
- id: "exfiltration.punycode",
- severity: "block",
- re: /https?:\/\/[^\s/]*xn--/i,
- message: "Links to a punycode host, which is how look-alike domains are spelled.",
- },
- {
- id: "exfiltration.curl_upload",
- severity: "warn",
- re: /\b(curl|wget|http|https)\b[^\n]{0,80}(\s-(d|F|T|X\s*POST|-data|-upload-file|-form)\b)[^\n]{0,80}(@\/|@~|\$\(|`)/i,
- message: "Uploads a local file or command output to a remote host.",
- },
- // ---- remote code that is fetched and run -----------------------------
- {
- id: "remote_exec.pipe_to_shell",
- severity: "block",
- re: /\b(curl|wget|fetch|iwr|Invoke-WebRequest|irm|Invoke-RestMethod)\b[^\n|]{0,160}\|\s*(sudo\s+)?(ba|z|da|k|c|tc)?sh\b|\|\s*(sudo\s+)?(python3?|perl|ruby|node|bash|sh|iex|Invoke-Expression)\b[^\n]{0,20}$/i,
- message: "Pipes something downloaded straight into a shell or interpreter.",
- },
- {
- id: "remote_exec.eval_download",
- severity: "block",
- re: /\b(eval|exec|source|\.)\s*[("'`]?\s*\$?\(?(curl|wget|fetch)\b|\beval\s*\(\s*(atob|Buffer\.from|base64)/i,
- message: "Evaluates downloaded or decoded text as code.",
- },
- {
- id: "remote_exec.decode_run",
- severity: "block",
- re: /\b(base64|b64decode|xxd\s+-r|openssl\s+enc\s+-d)\b[^\n]{0,60}\|\s*(sudo\s+)?(ba|z|da)?sh\b|echo\s+["']?[A-Za-z0-9+/=]{40,}["']?\s*\|\s*base64\s+(-d|--decode)/i,
- message: "Decodes an encoded blob and runs it.",
- },
- {
- id: "remote_exec.binary_download",
- severity: "warn",
- re: /\b(curl|wget)\b[^\n]{0,120}\b(-o|-O|--output)\b[^\n]{0,80}\b(chmod\s+\+x|\.\/)/i,
- message: "Downloads a binary and marks it executable; pin the source and say what it is.",
- },
- // ---- destructive and persistence commands ----------------------------
- {
- id: "destructive.rm_root",
- severity: "block",
- re: /\brm\s+(-[a-z]*r[a-z]*f?[a-z]*|-[a-z]*f[a-z]*r[a-z]*)\s+(\/|~|\$HOME|\*|\.\s*$|\.\.|\/\*)/i,
- message: "Deletes the filesystem, the home directory or the working tree.",
- },
- {
- id: "destructive.disk",
- severity: "block",
- re: /\b(mkfs(\.\w+)?|dd\s+if=|shred\b|diskutil\s+erase|format\s+[a-z]:|fdisk\b|wipefs)\b/i,
- message: "Writes to or erases a disk.",
- },
- {
- id: "destructive.git",
- severity: "warn",
- re: /\bgit\s+(push\s+[^\n]*--force|push\s+-f\b|reset\s+--hard\s+origin|clean\s+-[a-z]*f[a-z]*d|branch\s+-D\s+(main|master))/i,
- message: "Runs a git command that discards history; say when and why.",
- },
- {
- id: "destructive.fork_bomb",
- severity: "block",
- re: /:\s*\(\s*\)\s*\{\s*:\s*\|\s*:\s*&\s*\}\s*;|\bwhile\s+true.*\bfork\b/i,
- message: "A fork bomb.",
- },
- {
- id: "persistence.startup",
- severity: "block",
- re: /(>>|tee\s+-a)\s*(~|\$HOME)?\/?\.?(bashrc|zshrc|profile|bash_profile|zprofile)\b|\bcrontab\s+(-e|-|\S+\.txt)|\/etc\/cron\.|systemctl\s+enable|launchctl\s+load|LaunchAgents\/|\\Startup\\|HKLM\\.*\\Run\b|schtasks\s+\/create/i,
- message: "Installs something that runs again after this session.",
- },
- {
- id: "privilege.sudo",
- severity: "warn",
- re: /\bsudo\s+(?!apt|apt-get|brew|npm|pip|dnf|yum|pacman)\S|\bchmod\s+(-R\s+)?(777|a\+rwx|o\+w)\b|\bchown\s+(-R\s+)?root\b|setuid|\bsu\s+-\b/i,
- message: "Escalates privilege; a skill should not need root.",
- },
- {
- id: "privilege.security_off",
- severity: "block",
- // No \b before a flag: a hyphen is not a word character, so the boundary
- // would be looked for between the space and the dash and never found.
- re: /\b(csrutil\s+disable|spctl\s+--master-disable|setenforce\s+0|ufw\s+disable|iptables\s+-F|Set-MpPreference\s+-Disable|GIT_SSL_NO_VERIFY|NODE_TLS_REJECT_UNAUTHORIZED\s*=\s*0)|(?<![\w-])(--no-verify|--insecure)\b|(?<![\w-])-k\s+https/i,
- message: "Turns off a security control (TLS verification, a firewall, SIP, Defender, commit hooks).",
- },
- {
- id: "supply_chain.package_hijack",
- severity: "warn",
- re: /\b(npm|pip|pnpm|yarn|gem|cargo)\s+(install|add|i)\b[^\n]{0,80}(--registry\s|--index-url\s|-i\s+https?:\/\/|--extra-index-url)/i,
- message: "Installs packages from a non-default registry; say which and why.",
- },
- {
- id: "supply_chain.unpinned_dependency",
- severity: "warn",
- re: /\b(npx|bunx|pipx\s+run|uvx)\s+(?!skills\b|@modelcontextprotocol\/inspector\b)[a-z@][\w@./-]*(?![^\n]*@\d)/i,
- message: "Runs a package straight from the registry without a pinned version.",
- },
- // ---- embedded secrets --------------------------------------------------
- {
- id: "secret.api_key",
- severity: "block",
- re: /\b(sk-[A-Za-z0-9]{20,}|sk-ant-[A-Za-z0-9_-]{20,}|AKIA[0-9A-Z]{16}|ghp_[A-Za-z0-9]{36}|gho_[A-Za-z0-9]{36}|github_pat_[A-Za-z0-9_]{22,}|xox[abprs]-[A-Za-z0-9-]{10,}|AIza[0-9A-Za-z_-]{35}|-----BEGIN (RSA |EC |OPENSSH |DSA )?PRIVATE KEY-----|glpat-[A-Za-z0-9_-]{20}|npm_[A-Za-z0-9]{36}|re_[A-Za-z0-9]{20,}|sq0[a-z]{3}-[A-Za-z0-9_-]{20,})\b/,
- message: "Contains what looks like a live credential. Remove it and rotate it.",
- },
- // ---- obfuscation -------------------------------------------------------
- {
- id: "obfuscation.base64_blob",
- severity: "block",
- re: /(?<![A-Za-z0-9+/])[A-Za-z0-9+/]{120,}={0,2}(?![A-Za-z0-9+/])/,
- message: "A long encoded blob; a skill is meant to be read.",
- },
- {
- id: "obfuscation.hex_escapes",
- severity: "block",
- re: /(\\x[0-9a-f]{2}){8,}|(\\u[0-9a-f]{4}){6,}|(%[0-9a-f]{2}){12,}/i,
- message: "A run of escaped bytes; write the text in the open.",
- },
- {
- id: "obfuscation.reversed",
- severity: "warn",
- re: /\|\s*rev\s*\|\s*(ba)?sh\b|\.split\(['"]{2}\)\.reverse\(\)\.join/i,
- message: "Reverses a string before running it.",
- },
- ];
- /**
- * Characters that are invisible on a page and meaningful to a model: zero
- * width joiners and spaces, bidi overrides, soft hyphens, the Unicode tag
- * block (U+E0000 to U+E007F, which is how text is smuggled past a reader),
- * and the private-use areas. Any one of them is a block: a skill has no
- * legitimate use for text that a person cannot see.
- */
- const INVISIBLE = /[-----]|[\uDB40][\uDC00-\uDC7F]/;
- Note: Usual in code (characters a sanitizer strips): Contains 4 bidirectional-override character(s), which make text display in a different order from how it's read
- Line 260 on GitHub
- /**
- * Letters from scripts that have look-alikes in Latin (Cyrillic, Greek) inside
- * an otherwise Latin word: "аdmin" with a Cyrillic а. A whole word in another
- * script is fine; a mixed word is a homoglyph.
- */
- const MIXED_SCRIPT = /(?<![\p{L}\p{N}])(?=[\p{L}\p{N}]*[A-Za-z])(?=[\p{L}\p{N}]*[\u0400-\u04FF\u0370-\u03FF])[\p{L}\p{N}]+(?![\p{L}\p{N}])/u;
- const clip = (s: string, n = 96) => (s.length > n ? `${s.slice(0, n - 1)}…` : s);
- /** Line endings, BOM and trailing whitespace normalised; more than two blank lines collapsed. */
- export function clean(raw: string): string {
- return raw
- .replace(/^/, "")
- .replace(/\r\n?/g, "\n")
- .split("\n")
- .map((l) => l.replace(/[ \t]+$/, ""))
- .join("\n")
- .replace(/\n{3,}/g, "\n\n")
- .trim() + "\n";
- }
- export type Frontmatter = { name: string; description: string; body: string };
- /** The `---` block at the top, or null when there is none. Only name and description are read. */
- export function frontmatter(text: string): Frontmatter | null {
- const m = text.match(/^---\n([\s\S]*?)\n---\n?([\s\S]*)$/);
- if (!m) return null;
- const fields: Record<string, string> = {};
- for (const line of m[1].split("\n")) {
- const kv = line.match(/^([A-Za-z_][\w-]*):\s*(.*)$/);
- if (kv) fields[kv[1].toLowerCase()] = kv[2].trim().replace(/^(["'])(.*)\1$/, "$2");
- }
- return { name: fields.name ?? "", description: fields.description ?? "", body: m[2] };
- }
- /** name: lowercase, digits and hyphens, the way `npx skills` and Claude Code want a directory called. */
- export const SLUG = /^[a-z0-9](?:[a-z0-9-]{0,62}[a-z0-9])?$/;
- /**
- * Run every cleaner over a skill. Returns the findings in document order; the
- * caller decides what a `block` means (it means the review stops).
- */
- export function scan(text: string): Finding[] {
- const out: Finding[] = [];
- const lines = text.split("\n");
- if (text.length > MAX_SKILL_CHARS) {
- out.push({ rule: "structure.too_long", severity: "block", message: `A skill is at most ${MAX_SKILL_CHARS.toLocaleString("en-US")} characters; this is ${text.length.toLocaleString("en-US")}.` });
- }
- if (text.length < MIN_SKILL_CHARS) {
- out.push({ rule: "structure.too_short", severity: "block", message: `A skill needs at least ${MIN_SKILL_CHARS} characters of instructions to be worth reviewing.` });
- }
- const fm = frontmatter(text);
- if (!fm) {
- out.push({ rule: "structure.no_frontmatter", severity: "block", message: "Start with a YAML front matter block: ---, name:, description:, ---." });
- } else {
- if (!SLUG.test(fm.name)) {
- out.push({ rule: "structure.bad_name", severity: "block", message: "name must be 1 to 64 lowercase letters, digits and hyphens, like `bulk-classify`." });
- }
- if (fm.description.length < 20 || fm.description.length > 1024) {
- out.push({ rule: "structure.bad_description", severity: "block", message: "description must say in 20 to 1,024 characters what the skill does and when to use it." });
- }
- if (!/^#{1,3}\s|\n#{1,3}\s|\n- |\n\d+\. |\n {4}\S/.test(fm.body)) {
- out.push({ rule: "structure.no_steps", severity: "warn", message: "No headings, lists or code blocks: a skill reads better as steps than as a paragraph." });
- }
- }
- const invisible = text.match(INVISIBLE);
- if (invisible) {
- const at = text.indexOf(invisible[0]);
- out.push({
- rule: "hidden_text.invisible_characters",
- severity: "block",
- message: `Contains an invisible character (U+${invisible[0].codePointAt(0)!.toString(16).toUpperCase().padStart(4, "0")}). Remove zero-width, bidi and tag characters.`,
- line: text.slice(0, at).split("\n").length,
- });
- }
- const mixed = text.match(MIXED_SCRIPT);
- if (mixed) {
- out.push({ rule: "hidden_text.homoglyph", severity: "block", message: "A word mixes Latin with Cyrillic or Greek letters, which is how look-alike text is smuggled in.", excerpt: clip(mixed[0]) });
- }
- lines.forEach((line, i) => {
- if (line.length > MAX_LINE_CHARS) {
- out.push({ rule: "structure.long_line", severity: "block", message: `Line ${i + 1} is ${line.length} characters; nothing readable is that long.`, line: i + 1 });
- return;
- }
- for (const r of RULES) {
- const m = line.match(r.re);
- if (!m) continue;
- // One report per rule, at its first hit; the judge and the page get a summary, not a log.
- if (out.some((f) => f.rule === r.id)) continue;
- out.push({ rule: r.id, severity: r.severity, message: r.message, line: i + 1, excerpt: clip(m[0].trim()) });
- }
- });
- // Links: count and shape. A skill that is mostly links is an advert.
- const links = [...text.matchAll(/https?:\/\/[^\s)"'`<>]+/g)].map((m) => m[0]);
- if (links.length > 40) {
- out.push({ rule: "structure.link_farm", severity: "block", message: `${links.length} links; a skill is instructions, not a directory.` });
- }
- const insecure = links.find((l) => l.startsWith("http://") && !/^http:\/\/(localhost|127\.0\.0\.1)/.test(l));
- if (insecure) {
- out.push({ rule: "exfiltration.plain_http", severity: "warn", message: "Links over plain http; use https.", excerpt: clip(insecure) });
- }
- return out;
- }
- export const blocked = (findings: Finding[]) => findings.filter((f) => f.severity === "block");
- export const warnings = (findings: Finding[]) => findings.filter((f) => f.severity === "warn");
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.