Skip to content
Back to search

classifier-dev

SkillGitHubIncomplete

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/mrmps--classifier-dev/badge.svg)](/listings/mrmps--classifier-dev)

HTML

<a href="/listings/mrmps--classifier-dev"><img src="/api/listings/mrmps--classifier-dev/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/marketplace.json · Manifest · 27 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. {
  2. "$schema": "https://json.schemastore.org/claude-code-marketplace.json",
  3. "name": "classifier-dev",
  4. "owner": {
  5. "name": "Michael Ryaboy",
  6. "email": "contact@classifier.dev",
  7. "url": "https://classifier.dev"
  8. },
  9. "metadata": {
  10. "description": "classifier.dev classification tools and skills for coding agents.",
  11. "version": "1.1.0"
  12. },
  13. "plugins": [
  14. {
  15. "name": "classifier",
  16. "source": "./plugins/classifier",
  17. "description": "Classify batches of text into your labels, tag multiple topics, count categories and review uncertain results. Public access needs no API key.",
  18. "version": "1.1.0",
  19. "author": { "name": "Michael Ryaboy", "email": "contact@classifier.dev", "url": "https://classifier.dev" },
  20. "homepage": "https://classifier.dev/mcp-setup",
  21. "repository": "https://github.com/mrmps/classifier-dev",
  22. "license": "MIT",
  23. "category": "productivity",
  24. "keywords": ["classification", "triage", "filtering", "routing", "labels", "zero-shot"]
  25. }
  26. ]
  27. }
README.md · README · 531 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. # classifier.dev
  2. Zero-shot text classification. Plain text in, a label and a calibrated
  3. confidence out. No key, no signup. Up to a thousand texts per request.
  4. curl https://classifier.dev/spam,not+spam/Win+a+free+iPhone
  5. spam
  6. curl "https://classifier.dev/?labels=spam,not+spam&text=Win+a+free+iPhone" # same call, query form
  7. spam
  8. Single Cloudflare Worker. No database, no framework, no build step beyond esbuild.
  9. ## CLI
  10. npm i -g classifier-dev
  11. classify bug,feature,praise < feedback.txt
  12. `cli/` is a separate npm package (`classifier-dev`, bin `classify`): one
  13. dependency-free Node file, tests against a mock API (`npm test`), semver with
  14. its own CHANGELOG, released with `npm run release patch|minor|major` which
  15. tags `cli-v<version>` and lets `.github/workflows/publish-cli.yml` publish
  16. (needs an `NPM_TOKEN` repo secret). It talks to the API exactly like curl does.
  17. ## Layout
  18. src/index.ts routing, validation, tiers, LLM fallback chain, analytics
  19. src/query.ts the GET query form, read and written with nuqs; the URL an error suggests
  20. src/jev.ts TypeSafe's Jev: packs inputs into requests, reads probabilities
  21. src/limiter.ts Durable Object: per-IP rate limiting
  22. src/report.ts digest — Analytics Engine SQL -> Resend, flags model fallbacks
  23. src/alerts.ts every 15 minutes; emails only when something is wrong
  24. src/feedback.ts agent feedback, feedback.now protocol -> email
  25. src/privacy.ts keyed pseudonyms: nothing kept points back at a caller
  26. src/cost.ts per-request upstream spend, from the providers' own accounting
  27. src/docs.ts the site (GET / and GET /benchmark), plain text
  28. src/home.ts the same two documents rendered, for browsers only
  29. src/ui.ts the shared look: markdown in a terminal
  30. cli/ the `classify` command, published to npm as classifier-dev
  31. eval/ benchmarks; read eval/README.md before quoting a number
  32. finish-dns.sh one-shot DNS wiring, see below
  33. wrangler.example.toml the Worker config, minus the account-specific ids
  34. ## The site
  35. `curl classifier.dev` prints plain text, exactly as it always has. A browser
  36. sends `Accept: text/html` and gets the same document rendered — headings,
  37. bracketed links, copy buttons — from `src/home.ts`. Nothing is duplicated: the
  38. page is generated from `DOCS` and `BENCHMARK` at request time, so the text
  39. stays canonical and the two cannot drift. `?format=text` opts out by hand, and
  40. both responses carry `Vary: accept`.
  41. ## Agent feedback
  42. Implements the [feedback.now](https://feedback.now) protocol (schema 1.1), so
  43. any agent that speaks it can report a problem without being told how:
  44. GET /.well-known/agent-feedback.json what this host accepts
  45. GET /api/v1/policy categories, severities, limits
  46. POST /api/v1/feedback full structured report
  47. POST /api/v1/observations lighter signal
  48. POST /api/v1/feedback/{id}/attachments more evidence, later
  49. GET /api/v1/receipts/{id} did it land, and was it any good
  50. Accepted submissions are emailed to `REPORT_TO`. Reports are kept in KV for 90
  51. days. A repeat of the same domain + surface + category + title is stored and
  52. acknowledged as a duplicate but not emailed again, so one looping agent cannot
  53. empty itself into the inbox; the hourly budget is 100 per IP and the remainder
  54. comes back on every receipt.
  55. Testimonials use `signal.category: "testimonial"`. They require
  56. `reporter.agent_type` and `reporter.agent_description`, so praise arrives with
  57. enough context to understand what kind of agent benefited and how it works.
  58. `quality_score` is a deterministic function of how complete the report is — an
  59. agent can read the rule and write a better one next time. Nothing here calls
  60. the classifier or Analytics Engine: this is where reports arrive saying those
  61. are broken, so it must work when they do not.
  62. ## Deploy
  63. Merging to `main` deploys. `.github/workflows/deploy.yml` typechecks, runs the
  64. Worker and CLI tests, runs `wrangler deploy`, and then asks the live service for
  65. `/v1/health` and one classification, so a deploy that uploads a broken Worker
  66. fails in CI rather than in somebody's terminal.
  67. By hand, to try something before it is merged:
  68. cp wrangler.example.toml wrangler.toml # once, then fill in your own ids
  69. npx wrangler deploy
  70. `wrangler.toml` is gitignored and holds the two values that are specific to one
  71. Cloudflare account: `account_id`, and the `STATS` KV namespace id that
  72. `npx wrangler kv namespace create STATS` hands back. The tracked
  73. `wrangler.example.toml` carries everything else — crons, bindings, migrations —
  74. so the deployment shape is in the repository and only the identifiers are not.
  75. CI has no `wrangler.toml`, so `.github/render-wrangler.mjs` writes one from the
  76. example and three repository secrets. That makes the example the deployed shape
  77. rather than a copy of it: change a binding in `wrangler.toml` alone and CI keeps
  78. deploying the old one.
  79. Repository secrets the deploy needs:
  80. CLOUDFLARE_API_TOKEN dash.cloudflare.com > My Profile > API Tokens >
  81. Create Token > "Edit Cloudflare Workers"
  82. CLOUDFLARE_ACCOUNT_ID the account_id from wrangler.toml
  83. STATS_KV_ID the STATS namespace id from wrangler.toml
  84. REPORT_TO where the daily digest goes
  85. Secrets set with `wrangler secret put` live on the Worker, not in the script
  86. bundle, so a deploy leaves them alone and CI never needs to know them.
  87. Secrets the Worker reads: `TYPESAFE_API_KEY`, `AI_GATEWAY_API_KEY` (Vercel's
  88. AI Gateway, which serves Jev on a free monthly credit; when set it is asked
  89. first and TypeSafe catches what it refuses), `OPENROUTER_API_KEY`,
  90. `CONTEXT_API_KEY` (context.dev, the chat's web search and page reads),
  91. `RESEND_API_KEY`, `CF_ANALYTICS_TOKEN`, `REPORT_KEY`, `PRIVACY_SALT`. Add one
  92. with `npx wrangler secret put NAME`; none of them are ever read from the
  93. repository. `src/index.ts` lists the rest in the `Env` interface.
  94. Secrets are compared with `secretEquals` (src/secrets.ts), never `===`: a
  95. plain comparison returns on the first wrong byte and tells a caller how much
  96. of a guess was right.
  97. ## The model
  98. Both tiers answer from [TypeSafe's Jev](https://docs.typesafe.ai), a decision
  99. model rather than a language model: it takes a state and typed questions and
  100. returns a calibrated probability per option, in ~150ms. That shape is why the
  101. API can do three things the LLM version could not.
  102. **A thousand inputs per request.** State is an array of `{id, text}` and each
  103. input gets its own question, so the whole batch is one upstream call. The
  104. documented limit is 64k tokens per request; `jev.ts` packs to a conservative
  105. budget and runs the resulting requests eight at a time. Measured: 400 news
  106. headlines classified in 650ms end to end, and packing 100 items scored the
  107. same as sending them one at a time.
  108. **Confidence that means something.** On 400 six-way emotion items, answers at
  109. >= 0.9 confidence were right 82% of the time and answers below 0.5 were right
  110. 29%. The previous model's logprob "confidence" put 87% of news items above 0.9
  111. and was right on 68% of those. So `tier: "smart"` now means: re-ask the
  112. single-label answers below 0.7 of a fast reasoning model and replace them,
  113. marked `escalated: true`. Nothing else changes. Which model matters: on
  114. exactly the items Jev is unsure about, deepseek-v4-flash, qwen3.7-flash and
  115. mercury-2.5 were no better than Jev; gemini-3.8-flash took news topics from
  116. 87.5% to 90.0% and emotion from 61.8% to 63.7%, so that is the chain. A
  117. frontier model (claude-fable-5.1) gets 72.3% / 90.7% at ~3x the price; the
  118. numbers are on /benchmark if that trade ever looks worth it.
  119. **Multi-label in one pass.** One yes/no question per label, labels at >= 0.7
  120. returned most-likely-first with the full score map. F1 0.887 on the seven-case
  121. set against 0.799 for the sweep-and-verify LLM cascade it replaced, in 230ms
  122. instead of 1.5s. Re-judging its candidates with the reasoning model made it
  123. worse (and took 23s), so multi-label ignores the tier.
  124. The LLM chains in `index.ts` remain as the fallback when TypeSafe is
  125. unavailable, limited to twenty inputs because they are one call per input.
  126. The digest reports which model actually answered, with a `FALLBACK` marker,
  127. because the previous primary was delisted upstream and served its backup for
  128. weeks at F1 0.546 without anything saying so.
  129. ## Updates list
  130. The form and `POST /subscribe` send a confirmation email through Resend.
  131. The API returns `202 {"ok":true,"status":"pending_confirmation"}`. Nothing is
  132. written to the newsletter database until the emailed token is submitted to
  133. `POST /subscribe/confirm` as `{"token":"..."}` (or with the confirmation form).
  134. GET only renders the form, so mail scanners cannot confirm subscriptions.
  135. Tokens are signed with `NEWSLETTER_CONFIRMATION_SECRET`, expire within 24 hours,
  136. and are never returned from signup. Resend's idempotency key deduplicates repeat
  137. requests for the same inbox within each clock hour. Existing per-IP limits also
  138. apply. Signing-key rotation invalidates outstanding links.
  139. Apply `migrations/postgres/` with `npm run db:migrate` before deploying. The
  140. subscriber table shares the application database. Existing unconfirmed subscribers stay unconfirmed; do not
  141. backfill `confirmed_at` or send them updates until they confirm. An existing
  142. unsubscribe is never cleared by confirmation or by replaying an old token.
  143. Required Worker secrets: `DATABASE_URL`, `NEWSLETTER_RESEND_API_KEY`, and a
  144. random `NEWSLETTER_CONFIRMATION_SECRET` of at least 32 bytes. `NEWSLETTER_FROM`
  145. in `wrangler.example.toml` must use a verified Resend sending domain. `REPORT_TO`
  146. is the reply address and receives notifications only for newly confirmed rows.
  147. The subscriber table holds email, source, signup/confirmation/unsubscribe dates, which
  148. roadmap items were ticked (`wants text[]`, holding `ROADMAP` keys; the ticks
  149. ride in the confirmation token and are written on confirmation), the requested
  150. latency when faster inference is selected, and an internal id. It holds no IP,
  151. request id, or classification traffic. Pending
  152. signups are not stored. Tokens and mail-provider error bodies must not be logged.
  153. `migrations/postgres/0007_newsletter.sql` defines the table; a repeat confirmation
  154. replaces the ticks only when it ticked something, and never clears an
  155. unsubscribe or moves the first confirmation date.
  156. Read only confirmed, active recipients when sending updates:
  157. SELECT email, wants, desired_latency_ms FROM subscriber
  158. WHERE confirmed_at IS NOT NULL AND unsubscribed_at IS NULL;
  159. What people asked for first, to order the work by:
  160. SELECT unnest(wants) AS item, count(*) FROM subscriber
  161. WHERE unsubscribed_at IS NULL GROUP BY 1 ORDER BY 2 DESC;
  162. The Worker uses the application `DATABASE_URL` for newsletter and account data.
  163. Subscriber rows have no account foreign key or API traffic identifier; sharing
  164. storage does not subscribe account holders or change existing consent. The old
  165. newsletter project is retained as a read-only migration archive. See
  166. `docs/postgres-setup.md` for the verified cutover and recovery procedure.
  167. The copy is one constant — `ROADMAP` in `src/newsletter.ts`. The plain text at
  168. `curl classifier.dev`, the form on the rendered page and `index.md` all read it,
  169. so a change to the roadmap changes all three or none.
  170. ## Jev long context
  171. Default Jev and explicit `model: "jev"` inputs over 32,000 characters use
  172. Chonkie RecursiveChunker with 600-token `cl100k_base` chunks, parallel Jev
  173. screening for relevant or uncertain evidence (including opposing evidence and
  174. exceptions), then final Jev over whole eligible chunks in source order.
  175. The final evidence budget is 20,000 `cl100k_base` tokens plus a conservative
  176. provider context check. Eligible chunks can be omitted when that budget fills;
  177. `usage.long_context` reports selection counts, tokens, calls and timing.
  178. This does not guarantee full-document final reading or universal accuracy.
  179. Use Fast with a workspace key backed by paid balance or an active paid
  180. subscription; anonymous access and signup credit do not qualify. Synchronous limits:
  181. 250,000 original context tokens summed across inputs, 20 documents, 32 decisions,
  182. and a 1 MB body. Retail is $0.084/M original `cl100k_base` context tokens,
  183. counted once across inputs regardless of dimensions and actual inference usage.
  184. Each continuous whitespace or non-whitespace run is limited to 8,192 UTF-16
  185. code units. Longer runs return `400 long_context_input` before document
  186. tokenization or chunking, bounding tokenizer work on pathological inputs.
  187. No eligible evidence returns `422 long_context_no_evidence` without charge.
  188. Explicit `model: "chunklaya"` keeps its separate legacy opt-in behavior.
  189. Send one whole document of up to 10 million input tokens (100 MB) to
  190. `POST /v1/classify` with a funded workspace key. Use the normal JSON input and
  191. labels, or send `text/plain` with a `labels` query parameter. Large uploads
  192. return `202` with a `status_url`; `Prefer: respond-async` also works for smaller
  193. documents. The server splits, screens, retries and finishes automatically.
  194. The repository CLI handles upload and polling with
  195. `node cli/classify.js a,b --document document.txt --json`.
  196. The exact whole-document token count sets the reservation and $0.084/M charge.
  197. Source text is stored privately while queued and deleted as it is screened.
  198. Final Jev reads at most 20,000 selected evidence tokens. Completion, failure,
  199. cancellation and 24-hour expiry delete remaining source and evidence; failed
  200. or canceled work is refunded. Results remain available until expiry.
  201. ## Analytics
  202. Every request writes one Analytics Engine datapoint (tier, label-set fingerprint,
  203. country, status, count, latency). No request text is written to analytics.
  204. Long-context aggregates also go to `classifier_long_context_events`, with
  205. context/document totals, chunk screening/selection/omission counts, phase input
  206. tokens (including unknown-count flags), answered-call counts and phase timings.
  207. Account events append the corresponding fields; their ordered column contract
  208. is in [`src/server/analytics/schema.ts`](src/server/analytics/schema.ts).
  209. Documents and original context tokens count once; chunk and phase counters
  210. accumulate across dimension passes. No document text or caller identifiers are
  211. written to the dedicated dataset.
  212. Every request also records what it cost us: OpenRouter returns the charge for
  213. a call when asked, and Jev is billed on the input tokens it reports, at the
  214. rate `eval/bench.py` prices the benchmarks with. Spend accumulates in a
  215. per-request meter (`src/cost.ts`) and lands in `double3`. That column was added
  216. after launch, so it reads 0 for anything older than that deploy.
  217. A cron at 15:00 UTC queries it and emails a digest via Resend.
  218. ### Alerts
  219. A separate cron runs every fifteen minutes and stays silent unless something
  220. fires. It only watches conditions with an action attached: the Jev key being
  221. refused, Jev not answering (the fallback chain serving quietly, which has
  222. happened), 5xx rates, smart-tier escalations failing (the shape an exhausted
  223. `OPENROUTER_API_KEY` takes), mean latency, a spend spike against the trailing
  224. day, and traffic stopping outright. 4xx is ignored — that is scanners probing
  225. for `/wp-admin`, not a fault.
  226. **Jev credits.** TypeSafe publishes no balance endpoint — its API is
  227. `/v1/systemone` and `/v1/models`, nothing else — so there is no number to
  228. watch. Instead the check calls `/v1/models` with the key every fifteen
  229. minutes and reports back whatever TypeSafe says: a 401, 402 or 403 there means
  230. out of credit, revoked or wrong, and raises a critical alert quoting TypeSafe's
  231. own message rather than guessing which status means what. Because it probes
  232. rather than waiting for traffic, it fires on a quiet host before any caller
  233. meets the fallback chain, and it runs even when Analytics Engine is down.
  234. Each condition emails once when it starts, again every six hours while it
  235. lasts, and once when it clears, with the state in KV under `alert:`. Thresholds
  236. are the `T` object at the top of `src/alerts.ts`.
  237. curl -H "authorization: Bearer $REPORT_KEY" https://classifier.dev/alerts
  238. curl -H "authorization: Bearer $REPORT_KEY" "https://classifier.dev/alerts?demo=1&send=1"
  239. The first previews without sending or touching state; the second emails a
  240. sample through the real path, to prove delivery works.
  241. Preview the digest any time without sending it:
  242. curl -H "authorization: Bearer $REPORT_KEY" https://classifier.dev/report
  243. Jev provider attempts are stored separately in `classifier_jev_attempts` through
  244. `JEV_AE`, including recovered failures, retries and gateway cooldown skips.
  245. The report includes status, reason, count and mean latency for each provider;
  246. `/alerts` shows current incidents even when their notification is suppressed.
  247. The existing 15-minute alert check warns when at least three attempts fail and
  248. failures exceed 5% for either provider. Counts account for Analytics Engine
  249. sampling. No input text, caller identifiers or upstream messages are stored.
  250. Successful simple and multi-label classifier names are retained for 90 days in
  251. a separate aggregate KV record with no caller or source-text fields. Its keyed
  252. fingerprint also appears in per-request analytics, allowing operators to resolve
  253. label names in those pseudonymous records.
  254. `AI_GATEWAY_DISABLED = "true"` in `wrangler.example.toml` keeps production on
  255. TypeSafe directly after the gateway repeatedly returned 429 on September 19.
  256. The gateway key is retained. To restore gateway-first routing, verify gateway
  257. capacity, change this variable to `"false"` in the example and local config,
  258. and deploy. Check provider attempts and the live API tests after re-enabling.
  259. The header is the only way in. A query string lands in access logs, in browser
  260. history and in the Referer header of whatever gets clicked next, so `?key=` is
  261. gone. Append `?send=1` to actually email it.
  262. Cloudflare's Analytics Engine SQL is a narrow ClickHouse subset — no `uniq()`,
  263. no `SELECT DISTINCT`, and a bare `SELECT col ... GROUP BY col` is rejected.
  264. Distinct counts therefore use `SELECT col, count() ... GROUP BY col` and count
  265. the returned rows. Each query is isolated so one failure cannot blank the report.
  266. ### Privacy
  267. Two columns in that dataset used to be the caller: the IP address, and the
  268. label set, joined and lowercased. Both are keyed hashes now (`src/privacy.ts`),
  269. so the figures still count distinct callers and distinct classifiers and
  270. nothing can be read back into an address or into somebody's wording. The caller
  271. hash takes the UTC day as well, so it stops being the same value tomorrow —
  272. which is why a unique-caller count over 7d or 30d is really caller-days.
  273. Set the key once, and treat it as a secret like any other:
  274. npx wrangler secret put PRIVACY_SALT # 32 random bytes
  275. Rotating it renumbers every fingerprint, so distinct counts double-count across
  276. the rotation. It falls back to `ADMIN_SIGNING_KEY`, then `REPORT_KEY`, then a
  277. per-isolate random value, because an unkeyed hash of an IPv4 address or of a
  278. common label set inverts in seconds.
  279. ## Eval
  280. npm run bench # multi-label, 7 cases: jev vs any OpenRouter model
  281. npm run single -- --dataset emotion --backend jev
  282. npm run single -- --dataset ag_news --backend openrouter:qwen/qwen3.7-flash
  283. python3 eval/escalate.py --dataset emotion # what the smart tier buys
  284. `single.py` downloads AG News and dair-ai/emotion test rows on first use and
  285. caches raw results under `eval/data/results/` so `escalate.py` can combine
  286. backends without re-spending. `eval/README.md` lists the caveats.
  287. ## Rate limiting
  288. Per IP in a Durable Object, counted in classifications: 3,000/min and
  289. 20,000/day on fast, 200/min and 2,000/day on smart.
  290. Two other approaches were tried and rejected:
  291. - Cloudflare's native `ratelimit` binding registers fine but never decremented
  292. (70 calls against a limit of 60 all returned `success: true`).
  293. - KV is edge-cached and eventually consistent, so a counter written this second
  294. is invisible to the next read — every request saw `remaining: 59`.
  295. A Durable Object is single-threaded and strongly consistent, which is what a
  296. counter needs. Verified at the original 60/min: 75 requests -> 60 × 200, 15 × 429.
  297. ## DNS
  298. The domain is registered at Porkbun; the Worker is on Cloudflare. Cloudflare
  299. Workers custom domains require the zone to live in Cloudflare, and neither API
  300. token here has `zone.create`, so that one step is manual:
  301. 1. https://dash.cloudflare.com -> Add a domain -> `classifier.dev` -> Free plan
  302. 2. `./finish-dns.sh` — reads the assigned nameservers, points Porkbun at them
  303. via the Porkbun API, and attaches the Worker to the apex and `www`.
  304. ## Agent skill
  305. npx skills add https://classifier.dev
  306. Served from this domain over RFC 8615 well-known discovery, so there is no
  307. repository in the middle:
  308. src/SKILL.md the skill, bundled as a Text module
  309. GET /skill.md the artifact
  310. GET /.well-known/agent-skills/index.json discovery, schema v0.2.0
  311. The index must carry a sha256 of the artifact, and an index that disagrees with
  312. the file makes the skill uninstallable. Rather than commit a digest that a later
  313. edit would silently invalidate, `src/skill.ts` hashes the bytes it actually
  314. serves, once per isolate. Editing SKILL.md is therefore enough; nothing else
  315. needs updating.
  316. The skill teaches the case the API pitch misses: you are already a model and can
  317. classify anything you can see, so the reason to call out is context, not
  318. capability — filtering forty search results down to six without reading forty.
  319. Two things it documents because testing found them the hard way. Cloudflare
  320. 403s Python's stdlib `urllib` User-Agent before the request reaches the Worker,
  321. so the recipe sets one explicitly. And filters should be told "when in doubt,
  322. keep it": on a ten-snippet research filter that took signal kept from 4/6 to
  323. 6/6 with no extra noise, where adding a third "possibly relevant" label did
  324. nothing.
  325. ## Discovery surfaces
  326. GET /openapi.json OpenAPI 3.1, also at /.well-known/openapi.json
  327. GET /llms.txt short index for agents, linked from robots.txt
  328. GET /benchmark measured accuracy, cost, latency
  329. Both are linked from the third paragraph of `GET /` so an agent reading the
  330. landing page finds them immediately.
  331. ## Known issue: Cloudflare's managed robots.txt
  332. Adding the zone enabled Cloudflare AI Crawl Control, which prepends a managed
  333. block to `/robots.txt` disallowing GPTBot, ClaudeBot, CCBot, Google-Extended,
  334. Bytespider, Amazonbot and meta-externalagent. The Worker's own robots.txt is
  335. appended after it and cannot override it.
  336. This blocks *training* crawlers, not runtime API consumers — any agent can still
  337. call the API. But it does keep the docs out of future model training data, which
  338. works against discovery. The toggles at
  339. dash.cloudflare.com -> classifier.dev -> AI Crawl Control -> Security did not
  340. persist when flipped, so this likely needs a plan-level change or support.
  341. ## Operator agent access
  342. For operator-owned bulk agent work, set a dedicated `AGENT_API_KEY` Worker secret
  343. and send it as `Authorization: Bearer ...`. Classification quotas are unmetered,
  344. but inference uses a separate $2/day operator allowance (`OPERATOR_DAILY_USD`),
  345. shared across IPs, with the same request and concurrency limits as free usage.
  346. It neither consumes nor expands the anonymous allowance and does not replace
  347. `ENTERPRISE_API_KEY`. The CLI accepts it
  348. through `CLASSIFY_API_KEY` or `CLASSIFIER_API_KEY`. It does not authorize private
  349. reports or admin access. Keep it in an ignored secret file; never give it to
  350. public clients. Anonymous quotas continue to apply to unauthenticated traffic.
  351. For a funded workspace key with an arranged bulk quota, set the
  352. `API_KEY_RATE_LIMIT_MULTIPLIERS` Worker secret to a JSON object mapping the
  353. SHA-256 hex digest of each key to an integer multiplier (1–1,000). Overrides only
  354. raise the admission quota for that key; account counters, balance charging,
  355. model capacity limits, and other keys remain unchanged. Remove the entry to
  356. restore the plan quota.
  357. ## Finding the classification code
  358. The domain terms are defined in [CONTEXT.md](CONTEXT.md).
  359. | Module | Responsibility |
  360. | --- | --- |
  361. | `src/index.ts` | HTTP validation, quota, model selection, smart escalation, fallback, and response formatting. |
  362. | `src/jev.ts` | Jev questions and answers, shared batch budgets and recovery, gateway/direct transport, and provider validation. |
  363. | `src/dimensions.ts` | Dimension definitions and the mapping from input–dimension decisions to Jev questions and back. |
  364. Both ordinary and dimension classification use the same Jev batching module.
  365. A question group is the smallest part of a request that recovery keeps together:
  366. one input's questions for ordinary classification, one decision for dimensions.
  367. Batch preparation accounts for shared input text once, checks both provider
  368. budgets, and runs before quota charging for dimensions. Execution limits
  369. concurrent requests and splits an oversized batch between question groups.
  370. Change provider budgets and recovery in `src/jev.ts`; keep dimension meaning in
  371. `src/dimensions.ts`. Test the posted requests and returned classifications through
  372. `jevClassify` and the Worker HTTP interface, without depending on packing internals.
  373. ## Multiple dimensions
  374. `POST /v1/classify` also accepts `items` and `dimensions`:
  375. ```json
  376. {
  377. "items": ["Checkout charges me twice"],
  378. "dimensions": {
  379. "team": ["billing", "identity", "platform"],
  380. "urgency": {
  381. "labels": ["immediate", "normal", "low"],
  382. "instructions": "Active financial harm is immediate."
  383. },
  384. "kind": ["bug", "request", "question"]
  385. }
  386. }
  387. ```
  388. Each `results[i].dimensions[name]` carries its own label, confidence, scores,
  389. model and latency. Jev shares state across item–dimension questions, packing
  390. both context limits. Smart escalation is per field and clears the original
  391. scores; unavailable Jev falls back only up to 20 decisions. The API accepts
  392. up to 20 dimensions and 1,000 decisions, with each decision charged to quota.
  393. The `classify_dimensions` MCP tool uses this same path.
  394. Analytics adds `blob9` (single/multi/dimensions), `double6` (successful input
  395. items), `double7` (dimension count), `double8` (uncertain or unscored fields),
  396. and `double9` (fields served by fallback). `double1` counts successful decisions.
  397. Dimension configurations are keyed fingerprints, never stored verbatim. The
  398. admin panel queries dimension traffic separately, including failures and
  399. caller-days. The existing alert cron flags three or more dimension 5xx in an
  400. hour above 10% of dimension requests, plus any dimension fallback usage.
  401. Run `npm run test:e2e` with Node 22.18+ against the real production API, or set
  402. `CLASSIFIER_BASE_URL` to a staging origin. These are named TypeScript tests
  403. using `node:test` and real HTTP/model calls, with no fetch or inference mocks.
  404. Set `CLASSIFIER_API_KEY` to an authorized key when the shared IP quota is used
  405. up; it is optional. The suite uses inference and counts toward normal quotas.
  406. The tests check exact decisions and ordering for a 300-decision batch, every
  407. field's allowed labels and probability distribution, real smart escalation,
  408. provider score preservation, aliases, validation errors, MCP, OpenAPI, and
  409. legacy calls.
  410. Responses start as `unknown` and are validated before becoming typed matrices.
  411. Smart tests require an actual escalation; model drift that makes every fixture
  412. confident fails the test instead of silently skipping the reasoning provider.
  413. `npm run typecheck` checks the Worker and the TypeScript live tests. Deployment
  414. CI runs the live suite after publishing; `npm test` stays offline. Fault
  415. injection remains in `test/dimensions.test.ts` and
  416. `test/dimensions-observability.test.ts`.
  417. Run the **live API tests** workflow manually from GitHub Actions (or
  418. `gh workflow run e2e.yml`) to check production without deploying a Worker.
src/skillscan.ts · Bundled script · 373 lines · 1 finding marked
  1. /**
  2. * The static pass over a submitted skill: the cleaners.
  3. *
  4. * A SKILL.md is instructions an agent will follow with a shell in its hand, so
  5. * the first review is the one no model can be talked out of. The rules mirror
  6. * what the open-source skill scanners look for (Cisco's skill-scanner, NVIDIA's
  7. * SkillSpector, Snyk's agent-scan, kurtpayne/skillscan-security): instruction
  8. * override and hidden text, credential harvesting and exfiltration, remote
  9. * code that is fetched and run, destructive and persistence commands, embedded
  10. * secrets, obfuscated blobs and suspicious links. Each rule is a regular
  11. * expression with a severity: `block` ends the review before a model sees the
  12. * text, `warn` is handed to the judge as context and shown on the page.
  13. *
  14. * Deliberately blunt. A rule that fires on a benign skill costs the submitter
  15. * one rewrite; a rule that misses a malicious one costs somebody a laptop.
  16. *
  17. * Nothing here is a substitute for the two model passes that follow; it is the
  18. * pass that makes those passes safe to run on untrusted text.
  19. */
  20. export type Severity = "block" | "warn";
  21. export type Finding = {
  22. /** Stable id, in the vocabulary the open-source scanners use. */
  23. rule: string;
  24. severity: Severity;
  25. /** One sentence, addressed to the submitter. */
  26. message: string;
  27. /** 1-based line of the first hit, when the rule is line-scoped. */
  28. line?: number;
  29. /** The matched text, trimmed to something a page can show. */
  30. excerpt?: string;
  31. };
  32. export const MAX_SKILL_CHARS = 24_000;
  33. export const MIN_SKILL_CHARS = 200;
  34. export const MAX_LINE_CHARS = 2_000;
  35. type Rule = { id: string; severity: Severity; re: RegExp; message: string };
  36. /**
  37. * Line-scoped rules. Every regex is case-insensitive and applied to one line at
  38. * a time, so a hit has a line number and an excerpt.
  39. */
  40. const RULES: Rule[] = [
  41. // ---- instruction override and hidden text (prompt injection) ----------
  42. {
  43. id: "prompt_injection.override",
  44. severity: "block",
  45. re: /\b(ignore|disregard|forget|override)\b[^.\n]{0,40}\b(previous|prior|earlier|above|all|any|system|developer)\b[^.\n]{0,30}\b(instructions?|prompts?|rules?|guidelines?|constraints?)\b/i,
  46. message: "Tells the agent to ignore or override its existing instructions.",
  47. },
  48. {
  49. id: "prompt_injection.persona",
  50. severity: "block",
  51. re: /\b(you are now|from now on you are|act as if you have no|pretend (you|that you) (are|have) no|DAN mode|jailbreak|developer mode enabled)\b/i,
  52. message: "Tries to replace the agent's identity or switch off its safeguards.",
  53. },
  54. {
  55. id: "prompt_injection.secrecy",
  56. severity: "block",
  57. re: /\b(do not|don't|never)\b[^.\n]{0,30}\b(tell|inform|mention|reveal|show|disclose|alert)\b[^.\n]{0,30}\b(the )?(user|human|operator|owner)\b/i,
  58. message: "Asks the agent to hide what it is doing from the user.",
  59. },
  60. {
  61. id: "prompt_injection.no_confirm",
  62. severity: "warn",
  63. re: /\b(without|skip|bypass|no need for)\b[^.\n]{0,20}\b(asking|confirmation|confirming|permission|approval|review)\b/i,
  64. message: "Asks the agent to skip confirmation or permission steps.",
  65. },
  66. {
  67. id: "prompt_injection.hidden_html",
  68. severity: "block",
  69. re: /<!--[\s\S]{0,200}\b(instruction|ignore|must|always|never|secret|run|execute|curl|send)\b/i,
  70. message: "An HTML comment carries instructions a reader of the rendered page would not see.",
  71. },
  72. {
  73. id: "prompt_injection.hidden_markup",
  74. severity: "block",
  75. re: /<(script|iframe|object|embed|img|svg|style|meta|link|form)\b/i,
  76. message: "Active HTML is not allowed in a skill; write Markdown.",
  77. },
  78. // ---- credential harvesting and exfiltration --------------------------
  79. {
  80. id: "exfiltration.credential_paths",
  81. severity: "block",
  82. re: /(~|\$HOME|\/home\/\w+|\/Users\/\w+|\/root)\/\.(ssh|aws|gnupg|gcloud|azure|kube|docker|npmrc|netrc|pypirc|git-credentials|config\/gh|claude\/(settings|credentials))\b|\/etc\/(shadow|passwd)\b|id_(rsa|ed25519|ecdsa)\b|\.env\b[^.\n]{0,40}\b(cat|read|send|upload|post|curl|print|echo)\b|\b(cat|read|send|upload|post|curl|print|echo)\b[^.\n]{0,40}\.env\b/i,
  83. message: "Reads or sends credential files (~/.ssh, ~/.aws, .env, /etc/shadow and the like).",
  84. },
  85. {
  86. id: "exfiltration.env_dump",
  87. severity: "block",
  88. re: /\b(printenv|env\s*\|\s*(curl|nc|base64)|set\s*\|\s*curl|export -p|Get-ChildItem Env:|process\.env\b[^.\n]{0,40}\b(fetch|post|send|http))\b/i,
  89. message: "Dumps environment variables, which is where keys live.",
  90. },
  91. {
  92. id: "exfiltration.send_secret",
  93. severity: "block",
  94. re: /\b(api[_ -]?key|secret|token|password|credential|private key|cookie|session)s?\b[^.\n]{0,60}\b(send|post|upload|submit|transmit|forward|exfil|paste)\b[^.\n]{0,40}\b(to|at|via)\b|\b(send|post|upload|submit|transmit|forward|paste)\b[^.\n]{0,40}\b(api[_ -]?key|secret|token|password|credential|private key|cookie|session)s?\b[^.\n]{0,40}\b(to|at|via)\b/i,
  95. message: "Sends keys, tokens or passwords somewhere.",
  96. },
  97. {
  98. id: "exfiltration.webhook",
  99. severity: "block",
  100. re: /https?:\/\/[^\s)"'`]*\b(webhook\.site|requestbin|pipedream\.net|ngrok(-free)?\.(app|io|dev)|burpcollaborator|interact\.sh|oast\.(fun|live|me|online|pro|site)|beeceptor|hookbin|postb\.in|localtunnel|trycloudflare\.com|serveo\.net)\b/i,
  101. message: "Links to a request-capture or tunnel service, the usual destination for exfiltrated data.",
  102. },
  103. {
  104. id: "exfiltration.discord_telegram",
  105. severity: "block",
  106. re: /https?:\/\/(discord(app)?\.com\/api\/webhooks|api\.telegram\.org\/bot|hooks\.slack\.com\/services)\//i,
  107. message: "Posts to a chat webhook; a skill has no reason to phone home.",
  108. },
  109. {
  110. id: "exfiltration.ip_literal",
  111. severity: "block",
  112. re: /https?:\/\/(\d{1,3}\.){3}\d{1,3}(?![\d.])(?!.*\b(127\.0\.0\.1|0\.0\.0\.0)\b)/i,
  113. message: "Links to a bare IP address rather than a named host.",
  114. },
  115. {
  116. id: "exfiltration.shortener",
  117. severity: "warn",
  118. re: /https?:\/\/(bit\.ly|tinyurl\.com|t\.co|goo\.gl|is\.gd|cutt\.ly|rb\.gy|shorturl\.at|tiny\.cc|ow\.ly)\//i,
  119. message: "Uses a URL shortener; link to the real destination.",
  120. },
  121. {
  122. id: "exfiltration.punycode",
  123. severity: "block",
  124. re: /https?:\/\/[^\s/]*xn--/i,
  125. message: "Links to a punycode host, which is how look-alike domains are spelled.",
  126. },
  127. {
  128. id: "exfiltration.curl_upload",
  129. severity: "warn",
  130. re: /\b(curl|wget|http|https)\b[^\n]{0,80}(\s-(d|F|T|X\s*POST|-data|-upload-file|-form)\b)[^\n]{0,80}(@\/|@~|\$\(|`)/i,
  131. message: "Uploads a local file or command output to a remote host.",
  132. },
  133. // ---- remote code that is fetched and run -----------------------------
  134. {
  135. id: "remote_exec.pipe_to_shell",
  136. severity: "block",
  137. re: /\b(curl|wget|fetch|iwr|Invoke-WebRequest|irm|Invoke-RestMethod)\b[^\n|]{0,160}\|\s*(sudo\s+)?(ba|z|da|k|c|tc)?sh\b|\|\s*(sudo\s+)?(python3?|perl|ruby|node|bash|sh|iex|Invoke-Expression)\b[^\n]{0,20}$/i,
  138. message: "Pipes something downloaded straight into a shell or interpreter.",
  139. },
  140. {
  141. id: "remote_exec.eval_download",
  142. severity: "block",
  143. re: /\b(eval|exec|source|\.)\s*[("'`]?\s*\$?\(?(curl|wget|fetch)\b|\beval\s*\(\s*(atob|Buffer\.from|base64)/i,
  144. message: "Evaluates downloaded or decoded text as code.",
  145. },
  146. {
  147. id: "remote_exec.decode_run",
  148. severity: "block",
  149. re: /\b(base64|b64decode|xxd\s+-r|openssl\s+enc\s+-d)\b[^\n]{0,60}\|\s*(sudo\s+)?(ba|z|da)?sh\b|echo\s+["']?[A-Za-z0-9+/=]{40,}["']?\s*\|\s*base64\s+(-d|--decode)/i,
  150. message: "Decodes an encoded blob and runs it.",
  151. },
  152. {
  153. id: "remote_exec.binary_download",
  154. severity: "warn",
  155. re: /\b(curl|wget)\b[^\n]{0,120}\b(-o|-O|--output)\b[^\n]{0,80}\b(chmod\s+\+x|\.\/)/i,
  156. message: "Downloads a binary and marks it executable; pin the source and say what it is.",
  157. },
  158. // ---- destructive and persistence commands ----------------------------
  159. {
  160. id: "destructive.rm_root",
  161. severity: "block",
  162. re: /\brm\s+(-[a-z]*r[a-z]*f?[a-z]*|-[a-z]*f[a-z]*r[a-z]*)\s+(\/|~|\$HOME|\*|\.\s*$|\.\.|\/\*)/i,
  163. message: "Deletes the filesystem, the home directory or the working tree.",
  164. },
  165. {
  166. id: "destructive.disk",
  167. severity: "block",
  168. re: /\b(mkfs(\.\w+)?|dd\s+if=|shred\b|diskutil\s+erase|format\s+[a-z]:|fdisk\b|wipefs)\b/i,
  169. message: "Writes to or erases a disk.",
  170. },
  171. {
  172. id: "destructive.git",
  173. severity: "warn",
  174. re: /\bgit\s+(push\s+[^\n]*--force|push\s+-f\b|reset\s+--hard\s+origin|clean\s+-[a-z]*f[a-z]*d|branch\s+-D\s+(main|master))/i,
  175. message: "Runs a git command that discards history; say when and why.",
  176. },
  177. {
  178. id: "destructive.fork_bomb",
  179. severity: "block",
  180. re: /:\s*\(\s*\)\s*\{\s*:\s*\|\s*:\s*&\s*\}\s*;|\bwhile\s+true.*\bfork\b/i,
  181. message: "A fork bomb.",
  182. },
  183. {
  184. id: "persistence.startup",
  185. severity: "block",
  186. re: /(>>|tee\s+-a)\s*(~|\$HOME)?\/?\.?(bashrc|zshrc|profile|bash_profile|zprofile)\b|\bcrontab\s+(-e|-|\S+\.txt)|\/etc\/cron\.|systemctl\s+enable|launchctl\s+load|LaunchAgents\/|\\Startup\\|HKLM\\.*\\Run\b|schtasks\s+\/create/i,
  187. message: "Installs something that runs again after this session.",
  188. },
  189. {
  190. id: "privilege.sudo",
  191. severity: "warn",
  192. re: /\bsudo\s+(?!apt|apt-get|brew|npm|pip|dnf|yum|pacman)\S|\bchmod\s+(-R\s+)?(777|a\+rwx|o\+w)\b|\bchown\s+(-R\s+)?root\b|setuid|\bsu\s+-\b/i,
  193. message: "Escalates privilege; a skill should not need root.",
  194. },
  195. {
  196. id: "privilege.security_off",
  197. severity: "block",
  198. // No \b before a flag: a hyphen is not a word character, so the boundary
  199. // would be looked for between the space and the dash and never found.
  200. re: /\b(csrutil\s+disable|spctl\s+--master-disable|setenforce\s+0|ufw\s+disable|iptables\s+-F|Set-MpPreference\s+-Disable|GIT_SSL_NO_VERIFY|NODE_TLS_REJECT_UNAUTHORIZED\s*=\s*0)|(?<![\w-])(--no-verify|--insecure)\b|(?<![\w-])-k\s+https/i,
  201. message: "Turns off a security control (TLS verification, a firewall, SIP, Defender, commit hooks).",
  202. },
  203. {
  204. id: "supply_chain.package_hijack",
  205. severity: "warn",
  206. re: /\b(npm|pip|pnpm|yarn|gem|cargo)\s+(install|add|i)\b[^\n]{0,80}(--registry\s|--index-url\s|-i\s+https?:\/\/|--extra-index-url)/i,
  207. message: "Installs packages from a non-default registry; say which and why.",
  208. },
  209. {
  210. id: "supply_chain.unpinned_dependency",
  211. severity: "warn",
  212. re: /\b(npx|bunx|pipx\s+run|uvx)\s+(?!skills\b|@modelcontextprotocol\/inspector\b)[a-z@][\w@./-]*(?![^\n]*@\d)/i,
  213. message: "Runs a package straight from the registry without a pinned version.",
  214. },
  215. // ---- embedded secrets --------------------------------------------------
  216. {
  217. id: "secret.api_key",
  218. severity: "block",
  219. re: /\b(sk-[A-Za-z0-9]{20,}|sk-ant-[A-Za-z0-9_-]{20,}|AKIA[0-9A-Z]{16}|ghp_[A-Za-z0-9]{36}|gho_[A-Za-z0-9]{36}|github_pat_[A-Za-z0-9_]{22,}|xox[abprs]-[A-Za-z0-9-]{10,}|AIza[0-9A-Za-z_-]{35}|-----BEGIN (RSA |EC |OPENSSH |DSA )?PRIVATE KEY-----|glpat-[A-Za-z0-9_-]{20}|npm_[A-Za-z0-9]{36}|re_[A-Za-z0-9]{20,}|sq0[a-z]{3}-[A-Za-z0-9_-]{20,})\b/,
  220. message: "Contains what looks like a live credential. Remove it and rotate it.",
  221. },
  222. // ---- obfuscation -------------------------------------------------------
  223. {
  224. id: "obfuscation.base64_blob",
  225. severity: "block",
  226. re: /(?<![A-Za-z0-9+/])[A-Za-z0-9+/]{120,}={0,2}(?![A-Za-z0-9+/])/,
  227. message: "A long encoded blob; a skill is meant to be read.",
  228. },
  229. {
  230. id: "obfuscation.hex_escapes",
  231. severity: "block",
  232. re: /(\\x[0-9a-f]{2}){8,}|(\\u[0-9a-f]{4}){6,}|(%[0-9a-f]{2}){12,}/i,
  233. message: "A run of escaped bytes; write the text in the open.",
  234. },
  235. {
  236. id: "obfuscation.reversed",
  237. severity: "warn",
  238. re: /\|\s*rev\s*\|\s*(ba)?sh\b|\.split\(['"]{2}\)\.reverse\(\)\.join/i,
  239. message: "Reverses a string before running it.",
  240. },
  241. ];
  242. /**
  243. * Characters that are invisible on a page and meaningful to a model: zero
  244. * width joiners and spaces, bidi overrides, soft hyphens, the Unicode tag
  245. * block (U+E0000 to U+E007F, which is how text is smuggled past a reader),
  246. * and the private-use areas. Any one of them is a block: a skill has no
  247. * legitimate use for text that a person cannot see.
  248. */
  249. const INVISIBLE = /[​-‏‪-‮⁠-⁤⁦-⁩­᠎-]|[\uDB40][\uDC00-\uDC7F]/;
    • Note: Usual in code (characters a sanitizer strips): Contains 4 bidirectional-override character(s), which make text display in a different order from how it's read
    • Line 260 on GitHub
  250. /**
  251. * Letters from scripts that have look-alikes in Latin (Cyrillic, Greek) inside
  252. * an otherwise Latin word: "аdmin" with a Cyrillic а. A whole word in another
  253. * script is fine; a mixed word is a homoglyph.
  254. */
  255. const MIXED_SCRIPT = /(?<![\p{L}\p{N}])(?=[\p{L}\p{N}]*[A-Za-z])(?=[\p{L}\p{N}]*[\u0400-\u04FF\u0370-\u03FF])[\p{L}\p{N}]+(?![\p{L}\p{N}])/u;
  256. const clip = (s: string, n = 96) => (s.length > n ? `${s.slice(0, n - 1)}…` : s);
  257. /** Line endings, BOM and trailing whitespace normalised; more than two blank lines collapsed. */
  258. export function clean(raw: string): string {
  259. return raw
  260. .replace(/^/, "")
  261. .replace(/\r\n?/g, "\n")
  262. .split("\n")
  263. .map((l) => l.replace(/[ \t]+$/, ""))
  264. .join("\n")
  265. .replace(/\n{3,}/g, "\n\n")
  266. .trim() + "\n";
  267. }
  268. export type Frontmatter = { name: string; description: string; body: string };
  269. /** The `---` block at the top, or null when there is none. Only name and description are read. */
  270. export function frontmatter(text: string): Frontmatter | null {
  271. const m = text.match(/^---\n([\s\S]*?)\n---\n?([\s\S]*)$/);
  272. if (!m) return null;
  273. const fields: Record<string, string> = {};
  274. for (const line of m[1].split("\n")) {
  275. const kv = line.match(/^([A-Za-z_][\w-]*):\s*(.*)$/);
  276. if (kv) fields[kv[1].toLowerCase()] = kv[2].trim().replace(/^(["'])(.*)\1$/, "$2");
  277. }
  278. return { name: fields.name ?? "", description: fields.description ?? "", body: m[2] };
  279. }
  280. /** name: lowercase, digits and hyphens, the way `npx skills` and Claude Code want a directory called. */
  281. export const SLUG = /^[a-z0-9](?:[a-z0-9-]{0,62}[a-z0-9])?$/;
  282. /**
  283. * Run every cleaner over a skill. Returns the findings in document order; the
  284. * caller decides what a `block` means (it means the review stops).
  285. */
  286. export function scan(text: string): Finding[] {
  287. const out: Finding[] = [];
  288. const lines = text.split("\n");
  289. if (text.length > MAX_SKILL_CHARS) {
  290. out.push({ rule: "structure.too_long", severity: "block", message: `A skill is at most ${MAX_SKILL_CHARS.toLocaleString("en-US")} characters; this is ${text.length.toLocaleString("en-US")}.` });
  291. }
  292. if (text.length < MIN_SKILL_CHARS) {
  293. out.push({ rule: "structure.too_short", severity: "block", message: `A skill needs at least ${MIN_SKILL_CHARS} characters of instructions to be worth reviewing.` });
  294. }
  295. const fm = frontmatter(text);
  296. if (!fm) {
  297. out.push({ rule: "structure.no_frontmatter", severity: "block", message: "Start with a YAML front matter block: ---, name:, description:, ---." });
  298. } else {
  299. if (!SLUG.test(fm.name)) {
  300. out.push({ rule: "structure.bad_name", severity: "block", message: "name must be 1 to 64 lowercase letters, digits and hyphens, like `bulk-classify`." });
  301. }
  302. if (fm.description.length < 20 || fm.description.length > 1024) {
  303. out.push({ rule: "structure.bad_description", severity: "block", message: "description must say in 20 to 1,024 characters what the skill does and when to use it." });
  304. }
  305. if (!/^#{1,3}\s|\n#{1,3}\s|\n- |\n\d+\. |\n {4}\S/.test(fm.body)) {
  306. out.push({ rule: "structure.no_steps", severity: "warn", message: "No headings, lists or code blocks: a skill reads better as steps than as a paragraph." });
  307. }
  308. }
  309. const invisible = text.match(INVISIBLE);
  310. if (invisible) {
  311. const at = text.indexOf(invisible[0]);
  312. out.push({
  313. rule: "hidden_text.invisible_characters",
  314. severity: "block",
  315. message: `Contains an invisible character (U+${invisible[0].codePointAt(0)!.toString(16).toUpperCase().padStart(4, "0")}). Remove zero-width, bidi and tag characters.`,
  316. line: text.slice(0, at).split("\n").length,
  317. });
  318. }
  319. const mixed = text.match(MIXED_SCRIPT);
  320. if (mixed) {
  321. out.push({ rule: "hidden_text.homoglyph", severity: "block", message: "A word mixes Latin with Cyrillic or Greek letters, which is how look-alike text is smuggled in.", excerpt: clip(mixed[0]) });
  322. }
  323. lines.forEach((line, i) => {
  324. if (line.length > MAX_LINE_CHARS) {
  325. out.push({ rule: "structure.long_line", severity: "block", message: `Line ${i + 1} is ${line.length} characters; nothing readable is that long.`, line: i + 1 });
  326. return;
  327. }
  328. for (const r of RULES) {
  329. const m = line.match(r.re);
  330. if (!m) continue;
  331. // One report per rule, at its first hit; the judge and the page get a summary, not a log.
  332. if (out.some((f) => f.rule === r.id)) continue;
  333. out.push({ rule: r.id, severity: r.severity, message: r.message, line: i + 1, excerpt: clip(m[0].trim()) });
  334. }
  335. });
  336. // Links: count and shape. A skill that is mostly links is an advert.
  337. const links = [...text.matchAll(/https?:\/\/[^\s)"'`<>]+/g)].map((m) => m[0]);
  338. if (links.length > 40) {
  339. out.push({ rule: "structure.link_farm", severity: "block", message: `${links.length} links; a skill is instructions, not a directory.` });
  340. }
  341. const insecure = links.find((l) => l.startsWith("http://") && !/^http:\/\/(localhost|127\.0\.0\.1)/.test(l));
  342. if (insecure) {
  343. out.push({ rule: "exfiltration.plain_http", severity: "warn", message: "Links over plain http; use https.", excerpt: clip(insecure) });
  344. }
  345. return out;
  346. }
  347. export const blocked = (findings: Finding[]) => findings.filter((f) => f.severity === "block");
  348. export const warnings = (findings: Finding[]) => findings.filter((f) => f.severity === "warn");

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.