Skip to content
Back to search

digitally-twin-anything

MarketplaceWorks with Claude CodeGitHub

The digitally-twin-anything plugin: build, accept and hand over a digital twin of a real system with the dta command line tool.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

Its scripts were only partly scanned.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/DOWingard/digitally-twin-anything with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/DOWingard/digitally-twin-anything'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/dowingard--digitally-twin-anything/badge.svg)](/listings/dowingard--digitally-twin-anything)

HTML

<a href="/listings/dowingard--digitally-twin-anything"><img src="/api/listings/dowingard--digitally-twin-anything/badge.svg" alt="findsafeskills safety rating" /></a>

Install

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

The repository's root is a Claude Code plugin, and a marketplace that lists it. The plugin gives the agent the `build-twin` skill and the `dta` command; `dta` runs the toolkit in a container, so no Python is needed on the machine.

Requirements:

- Linux. It is developed and tested on x86-64 Linux only. On macOS and Windows the launcher refuses to run: there Docker runs containers in a virtual machine, and the toolkit cannot reach a twin the way it does on Linux (`DTA_UNSUPPORTED_OS=1` tries it anyway, untested). - Docker: the engine, running, and its command line with the buildx plugin (Docker Desktop has both; on Linux, the `docker-buildx-plugin` package or your distribution's equivalent). The user who runs the agent must be allowed to use Docker's socket. Compose is not needed on the machine itself: the toolkit's image carries its own docker client with compose. - A POSIX shell (`sh`) and the usual command line tools. - For the first use, network access to Docker Hub (`registry-1.docker.io`, `auth.docker.io`, `production.cloudfront.docker.com`) and the Python package index (`pypi.org`, `files.pythonhosted.org`), and about 2 GB of free disk.

Add the plugin from a clone of this repository:

git clone <this repository> digitally-twin-anything

# for one session
claude --plugin-dir ./digitally-twin-anything

# or for good, through the repository's marketplace
claude plugin marketplace add ./digitally-twin-anything
claude plugin install digitally-twin-anything@digitally-twin-anything

Inside a session, `/plugin marketplace add <path>` and `/plugin install digitally-twin-anything@digitally-twin-anything` do the same. A marketplace added from a local path loads the plugin in place, so a `git pull` in the clone takes effect at the next session. Once the repository is hosted on GitHub, `<owner>/<repo>` replaces the path.

What the plugin can do on your machine

Read this before installing. The `dta` launcher starts the toolkit's container with Docker's socket, the host's network and the working directory mounted, running as you. Access to Docker's socket is equivalent to root on the machine: the toolkit uses it to build images and to start and stop the twin's own containers, and anything that can run `dta` can do as much. Install the plugin only where you would let the agent use Docker.

- A twin's services listen on the loopback address only (`127.0.0.1`), behind one API token; nothing is published to the network. - Secrets (source credentials, the token, the store's password) live in the workspace's `.env`, which the toolkit never prints, never puts into an export and reads values of by name only. - A twin calls the real system only through actions you declared and signed, and only when the executors add-on is switched on in the contract.

First use

The images are built on your machine, not downloaded ready-made. The first `dta` command, whatever it is, builds `dta-toolkit:<version>` (the toolkit) and `dta-runtime:<version>` (what a twin's own services run from), the version being `bin/VERSION`, then runs. The agent says so and waits; you see docker's progress on standard error, and all of it goes to `~/.local/state/dta/build.log` (`$XDG_STATE_HOME/dta/build.log` when that is set). Commands started while it builds wait for it instead of building again.

Measured on one machine:

| What | Measured | |---|---| | Downloads of a first build | about 0.6 GB, counted from the registry's manifests and the lock file: two base images (110 MB), the Python packages of both images and the installer | | A first build | 49 s on a fast connection with the base images already present; expect several minutes on an ordinary one | | A build of unchanged sources with docker's layer cache | about 2 s for both | | Size | `dta-toolkit` 712 MB and `dta-runtime` 1.21 GB as docker lists them; 1.8 GB on disk together, since they share their 120 MB base |

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/marketplace.json · Manifest · 15 lines
  1. {
  2. "name": "digitally-twin-anything",
  3. "description": "The digitally-twin-anything plugin: build, accept and hand over a digital twin of a real system with the dta command line tool.",
  4. "owner": {
  5. "name": "Derek Wingard"
  6. },
  7. "plugins": [
  8. {
  9. "name": "digitally-twin-anything",
  10. "source": "./",
  11. "description": "Build, accept and hand over a digital twin of any system whose data lives in databases, APIs or files, with the dta command line tool: a typed graph with full history, served by an API, a browser UI and an operator skill.",
  12. "category": "development"
  13. }
  14. ]
  15. }
README.md · README · 690 lines
  1. <h1 align="center">DTA: Digitally Twin Anything</h1>
  2. <h3 align="center">One agent plugin for building a working digital twin of anything a dataset can describe.</h3>
  3. <p align="center"><b>Point an agent at your data. Get a twin that answers your questions within the error you accept, checks itself, and corrects itself.</b></p>
  4. <p align="center">
  5. <img alt="version 0.1.0" src="https://img.shields.io/badge/version-v0.1.0-cb3837?style=flat-square">
  6. <img alt="Claude Code plugin" src="https://img.shields.io/badge/Claude%20Code-plugin-5a4fcf?style=flat-square">
  7. <img alt="skill: build-twin" src="https://img.shields.io/badge/skill-build--twin-000000?style=flat-square">
  8. <img alt="requires Docker" src="https://img.shields.io/badge/Docker-required-2ea44f?style=flat-square&logo=docker&logoColor=white">
  9. <img alt="platform: Linux" src="https://img.shields.io/badge/platform-Linux-555555?style=flat-square&logo=linux&logoColor=white">
  10. <img alt="license: Apache-2.0" src="https://img.shields.io/badge/license-Apache--2.0-f5c518?style=flat-square">
  11. </p>
  12. ---
  13. An agent plugin that builds a working digital twin of anything that can be
  14. described by a large enough dataset, then keeps it accurate and manages it.
  15. A twin built by this system is:
  16. - a typed graph of the target (objects, links, properties) with its full
  17. version history, so every read can ask "as of" any past version;
  18. - synced from the target's data sources on a schedule (a live twin) or loaded
  19. once (a static twin);
  20. - able to answer a declared set of questions within declared error tolerances,
  21. and to correct itself or raise a failure when it cannot;
  22. - served through one HTTP API, a browser UI and a generated skill file that
  23. tells an agent how to operate it;
  24. - one container stack that can be exported and moved to another host.
  25. Status: version 0.1.0, a first release for Linux. A static twin (loaded once
  26. from recorded data) and a live twin (kept in step with its sources, one
  27. version per sync interval) can be built, accepted and handed over, with
  28. functions, models selected by backtest, questions whose accuracy the twin
  29. measures, and what-if scenarios. A live twin checks its own fidelity, corrects itself or says that
  30. it failed, and can act on the real system through typed actions and control
  31. loops. A twin can be exported into one file and imported on another host
  32. with the same history. The repository is itself the agent plugin: it carries
  33. the skill by which an agent builds a twin, and a launcher that runs the
  34. toolkit from its image.
  35. ## Install
  36. The repository's root is a Claude Code plugin, and a marketplace that lists
  37. it. The plugin gives the agent the `build-twin` skill and the `dta` command;
  38. `dta` runs the toolkit in a container, so no Python is needed on the machine.
  39. Requirements:
  40. - Linux. It is developed and tested on x86-64 Linux only. On macOS and
  41. Windows the launcher refuses to run: there Docker runs containers in a
  42. virtual machine, and the toolkit cannot reach a twin the way it does on
  43. Linux (`DTA_UNSUPPORTED_OS=1` tries it anyway, untested).
  44. - Docker: the engine, running, and its command line with the buildx plugin
  45. (Docker Desktop has both; on Linux, the `docker-buildx-plugin` package or
  46. your distribution's equivalent). The user who runs the agent must be allowed
  47. to use Docker's socket. Compose is not needed on the machine itself: the
  48. toolkit's image carries its own docker client with compose.
  49. - A POSIX shell (`sh`) and the usual command line tools.
  50. - For the first use, network access to Docker Hub (`registry-1.docker.io`,
  51. `auth.docker.io`, `production.cloudfront.docker.com`) and the Python package
  52. index (`pypi.org`, `files.pythonhosted.org`), and about 2 GB of free disk.
  53. Add the plugin from a clone of this repository:
  54. ```sh
  55. git clone <this repository> digitally-twin-anything
  56. # for one session
  57. claude --plugin-dir ./digitally-twin-anything
  58. # or for good, through the repository's marketplace
  59. claude plugin marketplace add ./digitally-twin-anything
  60. claude plugin install digitally-twin-anything@digitally-twin-anything
  61. ```
  62. Inside a session, `/plugin marketplace add <path>` and `/plugin install
  63. digitally-twin-anything@digitally-twin-anything` do the same. A marketplace
  64. added from a local path loads the plugin in place, so a `git pull` in the
  65. clone takes effect at the next session. Once the repository is hosted on
  66. GitHub, `<owner>/<repo>` replaces the path.
  67. ### What the plugin can do on your machine
  68. Read this before installing. The `dta` launcher starts the toolkit's container
  69. with Docker's socket, the host's network and the working directory mounted,
  70. running as you. Access to Docker's socket is equivalent to root on the
  71. machine: the toolkit uses it to build images and to start and stop the twin's
  72. own containers, and anything that can run `dta` can do as much. Install the
  73. plugin only where you would let the agent use Docker.
  74. - A twin's services listen on the loopback address only (`127.0.0.1`), behind
  75. one API token; nothing is published to the network.
  76. - Secrets (source credentials, the token, the store's password) live in the
  77. workspace's `.env`, which the toolkit never prints, never puts into an
  78. export and reads values of by name only.
  79. - A twin calls the real system only through actions you declared and signed,
  80. and only when the executors add-on is switched on in the contract.
  81. ### First use
  82. The images are built on your machine, not downloaded ready-made. The first
  83. `dta` command, whatever it is, builds `dta-toolkit:<version>` (the toolkit)
  84. and `dta-runtime:<version>` (what a twin's own services run from), the
  85. version being `bin/VERSION`, then runs. The agent says so and waits; you see
  86. docker's progress on standard error, and all of it goes to
  87. `~/.local/state/dta/build.log` (`$XDG_STATE_HOME/dta/build.log` when that is
  88. set). Commands started while it builds wait for it instead of building again.
  89. Measured on one machine:
  90. | What | Measured |
  91. |---|---|
  92. | Downloads of a first build | about 0.6 GB, counted from the registry's manifests and the lock file: two base images (110 MB), the Python packages of both images and the installer |
  93. | A first build | 49 s on a fast connection with the base images already present; expect several minutes on an ordinary one |
  94. | A build of unchanged sources with docker's layer cache | about 2 s for both |
  95. | Size | `dta-toolkit` 712 MB and `dta-runtime` 1.21 GB as docker lists them; 1.8 GB on disk together, since they share their 120 MB base |
  96. | Time the launcher adds to every later command | about 30 ms before the container starts, the check for the two images included |
  97. The builds need only the repository's committed files; local files of a clone
  98. (`.env`, environments, caches, notes, tests) never enter the build. Base
  99. images are pinned by digest, Python packages by the lock file's versions and
  100. hashes, the installer by version and hash, and the build backend of the
  101. project's own packages by version.
  102. To build both images again, after changing the sources or to start over:
  103. ```sh
  104. DTA_REBUILD=1 ./digitally-twin-anything/bin/dta status
  105. ```
  106. To remove them (a later `dta` command builds them again):
  107. ```sh
  108. docker image rm dta-toolkit:0.1.0 dta-runtime:0.1.0
  109. ```
  110. Docker keeps the layers it built in its build cache (`docker builder du`
  111. shows how much); `docker builder prune` empties that cache, for every project
  112. on the machine, not this one alone. Twins you built have their own containers,
  113. volumes and images: `dta down --destroy-data` in a twin's workspace removes
  114. them.
  115. `bin/dta` documents every variable it reads in its header; `DTA_TOOLKIT_IMAGE`
  116. runs an image of your choosing instead, which is then never built.
  117. ### Try it
  118. `examples/bike-share` holds a data generator and the manifest of a static twin
  119. of what it writes: 21 days of a fictional bike-share operator (80 stations,
  120. 40,275 trips, a status row per station every 15 minutes; 19 MB).
  121. With the agent, which is what the plugin is for: generate the data (step 1
  122. below), start Claude Code with the plugin in an empty folder and say what you
  123. want, for instance "Build a digital twin of the bike-share export in
  124. ~/bikes/export; its description is in description.md there." The agent asks
  125. for the purpose, the questions and the error you accept for each, shows a
  126. summary to sign, builds the twin and reports whether it passed acceptance.
  127. By hand, with the example's manifest in place of the agent's work:
  128. ```sh
  129. REPO=/path/to/digitally-twin-anything
  130. export PATH="$REPO/bin:$PATH"
  131. dta --help # the first command builds the two images
  132. # 1. the data
  133. mkdir -p ~/bikes/twin
  134. docker run --rm -u "$(id -u):$(id -g)" \
  135. -v "$REPO/examples/bike-share:/example:ro" -v "$HOME/bikes:/out" \
  136. --entrypoint python dta-runtime:0.1.0 /example/generate.py /out/export
  137. # 2. the workspace and the manifest
  138. cd ~/bikes/twin
  139. dta init
  140. cp "$REPO/examples/bike-share/manifest.yaml" manifest.yaml
  141. cp "$REPO/examples/bike-share/ontology/cykelbo.yaml" ontology/
  142. dta ontology check --write
  143. dta env template
  144. # in .env: CYKELBO_EXPORT_DIR=<absolute path of ~/bikes/export>, two free
  145. # ports as DTA_API_PORT and DTA_UI_PORT, DTA_BASE_URL=http://127.0.0.1:<API port>
  146. dta env check
  147. # 3. the stages
  148. dta source probe
  149. dta profile
  150. dta manifest validate
  151. dta manifest confirm --all-user
  152. dta manifest validate
  153. dta manifest sign --non-interactive --approved-by "<your name>"
  154. dta build
  155. dta accept
  156. # 4. the twin
  157. dta questions list
  158. dta questions answer q_forecast
  159. dta down # stop it; --destroy-data also removes its data
  160. ```
  161. Run this way from an export of the committed files, on a machine that was busy
  162. with other work: every command before the build took 1 to 4 s; `dta build`
  163. 8 min 25 s (loading the data 105 s, backtesting three candidate models for 80
  164. stations 364 s); `dta accept` 23 s, all 26 checks passing. The forecast one
  165. hour ahead was measured at 0.55 bikes (MAE) against the 2 the manifest accepts,
  166. and the UI answered on `DTA_UI_PORT`. `dta manifest sign` without the two
  167. options shows the summary and asks for your name instead.
  168. ## Layout
  169. | Path | What it is |
  170. |---|---|
  171. | `.claude-plugin` | The plugin's manifest and the marketplace that lists it |
  172. | `bin/dta` | The launcher: builds the two images when they are absent, then runs the toolkit's image on the working directory |
  173. | `skills/build-twin` | The skill by which an agent builds, accepts and hands over a twin |
  174. | `tests/plugin` | Tests of the launcher (against real docker) and of the skill's texts against the tool |
  175. | `packages/core` | Shared library: structured logging, environment configuration, the manifest schema and its checks, ontology loading (LinkML), the versioned store, the connector protocol |
  176. | `packages/toolkit` | The `dta` command line tool: manifest checks, the gate each build stage ends in, delegated tasks, the twin's container stack derived from its manifest, build, acceptance and the operator skill |
  177. | `packages/runtime` | What runs inside a twin's stack: installing and opening a twin, reading its sources, loading them into the graph and keeping it in step with them, the HTTP API, the scheduler, the worker that runs functions and models, training frames, the shipped model families, scenarios and the browser UI |
  178. | `docker/toolkit.Dockerfile` | Image of the toolkit, with a docker client and its compose and buildx plugins |
  179. | `docker/runtime.Dockerfile` | Image of the twin's services: one image, one command per service |
  180. | `examples/bike-share` | A data generator and the manifest of a static twin of what it writes |
  181. | `spikes/storage` | Benchmark that selected Dolt as the versioned store |
  182. ## The store
  183. A twin's graph lives in a [Dolt](https://github.com/dolthub/dolt) database, one
  184. table per object or link type, generated from the twin's ontology.
  185. - A version is one commit. Its time is data time (the latest data timestamp the
  186. twin had seen), never the wall clock, so history can be replayed exactly.
  187. - Every stored value carries its provenance: where it came from, whether it
  188. was observed, inferred, assumed or simulated, how confident, and when it
  189. was true.
  190. - A change set is validated against the ontology and written whole or not at
  191. all. A committed version never changes.
  192. - Reads take a selector (`head`, a version id, `#<n>`, a time, or a range)
  193. and return Arrow tables.
  194. - Branches hold scenarios; simulated facts cannot reach the live branch.
  195. ## The manifest
  196. A twin is fully specified by one file, `manifest.yaml`, in its workspace. Each
  197. required field has a status:
  198. | Status | Meaning |
  199. |---|---|
  200. | unknown | no value yet |
  201. | proposed | a value is set and waits for the user to confirm it |
  202. | derived | computed by the toolkit from measurements |
  203. | confirmed | accepted by the field's owner (the user or the building agent) |
  204. The build may start only when no required field is unknown or proposed.
  205. `dta manifest validate` reports what is inconsistent in the manifest as
  206. written; `dta manifest gaps` lists every required field still unknown or
  207. waiting for confirmation; `dta status` shows the stage the build stands at
  208. and the result of each stage's gate.
  209. ## Building a twin
  210. A build moves through stages; each ends in a gate, and `dta status` shows where
  211. the build stands. All commands act on a workspace (`--workspace`, default the
  212. current directory) and print one JSON document with `--json`.
  213. | Stage | Gate passes when | Command |
  214. |---|---|---|
  215. | 0 intake | the workspace and a manifest exist | `dta init` |
  216. | 1 contract | the user has confirmed the contract fields | `dta manifest confirm` |
  217. | 2 access | every source can be read from inside the twin's stack | `dta source probe` |
  218. | 3 profiling | every source has a profile in the manifest | `dta profile` |
  219. | 4 design | the manifest and its ontology are consistent | `dta manifest validate` |
  220. | 5 gaps | no required field is open, and the user has signed | `dta manifest sign` |
  221. | 6 build | the stack runs and serves this manifest | `dta build` |
  222. | 7 acceptance | every acceptance check passes | `dta accept` |
  223. | 8 handover | the operator skill is installed | `dta install-skill` |
  224. ```sh
  225. dta init
  226. # write manifest.yaml and the ontology file it names (LinkML)
  227. dta ontology check --write # confirm the ontology's types in the manifest
  228. dta env template # .env.example and .env; fill in what is not generated
  229. dta source probe # reach every source from inside the stack
  230. dta profile # measure the sources; profiles go into the manifest
  231. dta sync derive --write # a live twin: its sync interval, from the contract and the profiles
  232. dta manifest validate
  233. dta manifest confirm --all-user
  234. dta manifest validate # confirming changes the manifest; the design gate is of the result
  235. dta manifest sign
  236. dta build # image, store, install, load, API, contract, sync (live), code,
  237. # functions, models, questions, UI, skill
  238. dta accept # acceptance report, or a deficit report
  239. dta install-skill # the skill an agent operates the twin with
  240. ```
  241. - Sources are only ever read from inside the twin's stack, never from the host,
  242. so an address, its credentials and its network mean the same at build time
  243. and at run time. A source in another compose project is reached through
  244. `runtime.networks` in the manifest.
  245. - `dta build` can be run again at any time. Every step changes only what
  246. differs; the load is not repeated while the twin is where the last load left
  247. it. A failing step stops the build and leaves what was done in place.
  248. - `dta accept` runs every check and reports all failures together: the API
  249. against its own OpenAPI document (generated requests, fixed seed), the five
  250. data-readiness requirements, constraint violations in the load, coverage of
  251. every property the contract names, the UI, the skill and that the skill
  252. lists every endpoint the API serves; that every question is answered, that
  253. none is measured outside its tolerance, that every model specification has
  254. an accepted model, and that the worker runs.
  255. - `dta install-skill` refuses a twin that has not been accepted unless
  256. `--provisional` is given; the handover report then lists what has not passed.
  257. - `dta up`, `dta down` and `dta logs` manage the stack. `dta down
  258. --destroy-data` removes the twin's data and takes back the build and
  259. acceptance gates.
  260. - No value of `.env` is printed, logged or written to a report.
  261. ### A live twin
  262. A twin whose contract says `tier: live` keeps reading its sources after the
  263. load. The manifest's `sync.interval` (Δ) is how often; `dta sync derive`
  264. computes it as the longest of three: the sources' refresh period, their
  265. rate-limit period, and what the questions need, which is the shorter of the
  266. tightest staleness less the source's own delay and half the fastest change
  267. the profiles measured.
  268. - Time is the twin's clock, never the machine's: `runtime.clock` is `wall` or
  269. `source:<id>`, a source that tells the time. Intervals are counted from the
  270. Unix epoch; a run takes every interval that has closed and writes one
  271. version for each, so a twin that was stopped catches up interval by interval.
  272. - A scheduler service in the stack starts the runs through the API. One run
  273. at a time; a run asked for twice with the same key is started once.
  274. - Each run records itself in the twin (`TwinSyncRun`): what it read and wrote
  275. per source, the rows it set aside with the rule each broke, the rows that
  276. arrived for an interval already written. A source that fails does not stop
  277. the others; the run is then `partial`.
  278. - A dataset declared under `sources.<id>.incremental` is read on from where
  279. the last run stopped instead of whole.
  280. - Freshness is measured per question: the age of the newest data of the
  281. sources the question is answered from, against the question's staleness.
  282. - `dta build` loads history once, only while no run has written; it stops the
  283. scheduler before a load and its `sync` step starts the scheduler, makes one
  284. run and fails unless that run synced every source. `dta accept` adds: the
  285. scheduler runs, the latest run succeeded, no source is failing, every
  286. question is fresh.
  287. ```sh
  288. dta sync status # the clock, freshness per question, the sources
  289. dta sync run # one run now; waits and prints its report
  290. dta sync run --source scada # only these sources
  291. dta sync run --replay scada.readings # read one dataset again, whole
  292. ```
  293. ### Functions, models and questions
  294. The manifest declares what the twin computes; `dta build` registers it in the
  295. twin, where it is versioned like everything else.
  296. - A function answers a question from the graph. Eight are built in
  297. (`latest`, `below`, `above`, `integrate`, `paths`, `aggregate`, `series`,
  298. and `forecast`, which asks a question's model); others are Python functions
  299. of the twin's own, kept in the directory `runtime.code` names. Extra
  300. packages they need are listed in `runtime.worker_requirements`; the build
  301. makes a worker image with them.
  302. - A model specification names a question, a target property, candidate
  303. families and a backtest. Shipped families: `persistence`, `seasonal_naive`,
  304. `theta`, `ets`, `arima`, `mstl`, `ridge_lags`, `gbm_lags`; a twin can bring
  305. families of its own (a hydraulic simulation of a pipe network, say).
  306. The build backtests every candidate over the twin's history, on a rolling
  307. origin, and makes the best one that is within the question's tolerance and
  308. better than the baseline the question's active model.
  309. - A question is `validated` (measured within its tolerance), `provisional`
  310. (not measured, or too little history) or `failing`. Every answer says which.
  311. - Models and code of the twin's own run in the worker, a separate container
  312. with limits on memory and time that reads the twin only through the API;
  313. code of the twin's own runs there in a process of its own per call.
  314. - A build evaluates a model again only when its definition, its code or its
  315. question changed; `dta models evaluate` does it on request, for instance
  316. after new data.
  317. ```sh
  318. dta register # functions, models and questions of the manifest
  319. dta models evaluate [--spec ID]
  320. dta models list
  321. dta questions list
  322. dta questions measure
  323. dta questions answer ID [--arg name=value] [--at VERSION]
  324. dta scenarios list
  325. ```
  326. ### Fidelity, actions and control
  327. A live twin measures itself on its own clock. At every `fidelity.interval` it
  328. scores what its models predicted for each predictive question against what was
  329. then observed, and is in one state: `unknown` (no check yet), `healthy`,
  330. `correcting` or `failed`.
  331. - **Breach.** A question's error at an origin o is its metric over (o, o + h].
  332. Its window error E is the mean over the origins scored in the last
  333. `fidelity.window`; with at least 3 of them and E above the question's
  334. tolerance ε an incident opens.
  335. - **Correction, rung by rung, without a human.** Recalibrate the active model
  336. on recent data; reselect among the candidates the manifest declares; ask the
  337. managing agent to re-model (a notification), and fail when nobody did within
  338. `correction.remodel_timeout`. A correction is accepted only if it passes the
  339. specification's acceptance on the recent history, and an incident resolves
  340. only on new evidence within ε.
  341. - **Stale data, stuck and biased signals.** A question whose data is older
  342. than its staleness opens an incident. An observed signal that used to change
  343. and stopped while its source kept delivering is called stuck when a run
  344. without a change that long has probability below 1e-7 under its own history.
  345. A smooth signal, whose changes keep their sign, is not judged by its run:
  346. after 3 intervals without a change it is stuck when its reading lies more
  347. than half the tolerance from the twin's own forecast made from its last
  348. change. A signal whose residual shifts by at least ε against every model of
  349. its property is called biased.
  350. - **The value of a suspect signal.** Its readings are kept on its record and
  351. not written to the object; the value in force is the twin's own estimate,
  352. marked as inferred: the forecast of a question of that property, from the
  353. newest version at which everything the model reads was observed. An
  354. estimate never rests on an estimate. A model that does not read the signal
  355. is forecast again at every check. One that reads it is forecast once, by the
  356. question with the longest horizon; past that horizon, or when the estimate
  357. is not finite or outside the signal's range, the value is unknown, and an
  358. answer or an action that needs it is refused with `fidelity.input_unknown`,
  359. naming the signal, until the signal is sound again. A model's own code says
  360. what its predictions read in `models.<id>.reads`; a code model that does not
  361. say is taken to read everything. Forecasts that started from a frozen, an
  362. inferred or an unknown value are not scored, so a failed sensor opens no
  363. breach and starts no correction of a model.
  364. - **Actions.** The manifest declares what may be done to the real system: an
  365. action type with typed parameters, the control it carries out with its
  366. bounds and autonomy (`advisory`, `supervised`, `autonomous`), and an HTTP
  367. call with its variables named, never stored. Every proposal is judged and
  368. recorded; an executed one is compared with what was observed after its
  369. settle time, and a mismatch is an incident.
  370. - **Interlock.** An autonomous action is executed while the twin is healthy,
  371. or correcting with no stale-data or mismatch incident open; a supervised one
  372. on a recorded approval, unless the twin failed. An autonomous action on a
  373. value that is unknown is blocked. A failed twin executes nothing.
  374. - **Control loops** run a policy function of the twin on an interval and
  375. propose what it returns. A loop is registered disabled; enabling it is a
  376. decision recorded with a name.
  377. - **Notifications** go out as CloudEvents to the address in the variable the
  378. contract names (or to the log): state changes, incidents, re-model requests,
  379. failed sync runs, pending approvals, blocked actions, mismatches.
  380. A twin that is blocked or failed leaves the real system in the state of its
  381. last executed command: handing a control back is done on the real system.
  382. ```sh
  383. dta fidelity status | checks | incidents | incident ID | errors QUESTION | signals
  384. dta fidelity retry ID --by NAME | resolve ID --by NAME | correct QUESTION --rung RUNG | clear UID --by NAME
  385. dta actions list | propose ACTION --object UID --param k=v --reason TEXT | records | approve UID --by NAME | reject UID --by NAME
  386. dta control list | enable LOOP --by NAME | disable LOOP --by NAME | run LOOP
  387. dta notifications
  388. ```
  389. ### When the twin fails, and manifest versions
  390. A twin whose fidelity state is `failed` executes nothing, so the real system
  391. keeps the last thing the twin set. Each control therefore carries the user's
  392. choice, `contract.controls.<control>.on_failure`, in the signed contract:
  393. | `mode` | What is sent, once per failure, to each object the twin commanded |
  394. |---|---|
  395. | `hold` | Nothing. |
  396. | `release` | A declared action that hands the object back to the real system's own control. |
  397. | `safe_value` | The control's own action with a constant inside the control's bounds. |
  398. It goes past the interlock and the control's autonomy and uses no model;
  399. `after_stale` sends it as well when the twin's data has been too old for that
  400. long. An autonomous control whose action calls the real system must make the
  401. choice. This is not a safety function: a twin that is stopped or cut off sends
  402. nothing, so protection against that belongs in the real system.
  403. While a breach incident waits for a managing agent to re-model, the twin keeps
  404. correcting on newer data: up to `fidelity.correction.retry_budget` cycles, each
  405. after a wait that doubles from `retry_backoff`. Only the request's time-out
  406. fails the twin.
  407. `manifest_version` states which version of the manifest's format a manifest is
  408. written in (now 2). A field newer than the stated version is refused, a
  409. manifest newer than the toolkit is refused (`manifest.too_new`), and before
  410. `dta build`, `dta up` and an import start anything the toolkit asks the runtime
  411. image what it reads and refuses an older one (`runtime.too_old`).
  412. ### Moving a twin, and the add-ons
  413. ```sh
  414. dta export [--out FILE|FOLDER] # one bundle: workspace, every version of the store, artifacts; no secrets
  415. dta import BUNDLE [--into DIR] # unpack, fill .env, run again: the store is restored and its heads verified
  416. ```
  417. - A bundle holds the manifest, the ontology, the twin's code, the record of the
  418. discussion, the reports, a backup of the versioned store and the artifact
  419. volume. `.env` is never in it. Export runs while the twin runs.
  420. - Import restores the store under the same commit ids: it compares the head of
  421. every branch with what the bundle names and refuses the import when one
  422. differs. Access, build, acceptance and handover are run again on the new
  423. host; the signature is carried.
  424. - A source that is a folder on the exporting machine is named in the bundle's
  425. warnings and is not carried.
  426. Three add-ons, each built only when the contract selects it and removed when
  427. it is deselected:
  428. | `contract.addons` | What is built |
  429. |---|---|
  430. | `sdk_python` | a Python package `twin_<id>` in `sdk/python/`: typed models from the ontology, a client generated from the twin's OpenAPI document, one accessor per type and one method per question; every call takes `at`; address and token from `DTA_BASE_URL` and `DTA_API_TOKEN` |
  431. | `mcp` | a service that offers every API operation as an MCP tool on `127.0.0.1:$DTA_MCP_PORT`; it holds no token and passes the caller's on |
  432. | `readonly_sql` | the store's SQL port on `127.0.0.1:$DTA_SQL_PORT` and a user that can only `SELECT`, for queries with `AS OF`, the commit log and diffs |
  433. ```sh
  434. dta generate sdk [--force]
  435. ```
  436. ## What a built twin serves
  437. One HTTP API under `/api/v1`, bound to the loopback address, with one bearer
  438. token. `GET /api/v1/openapi.json` describes every endpoint.
  439. | Group | For |
  440. |---|---|
  441. | `/meta` | identity, health, the manifest and the ontology at a version |
  442. | `/objects` | objects of a type: filter, sort, page, count; a search by uid across types; one object with the provenance of each value; its links and its history |
  443. | `/graph` | neighbours, shortest paths, subgraphs, the edge list, each over all links or only those that pass a filter on their properties |
  444. | `/history` | versions, the difference between two versions, branches |
  445. | `/block` | whole node and edge tables, and slices over a range of versions, as Arrow or Parquet |
  446. | `/sync` | a live twin: its sources and how far each has been read, its runs and the rows they set aside, starting a run |
  447. | `/questions` | the questions of the contract with what each may claim; answering one; measuring them |
  448. | `/functions` | the twin's functions with their arguments; running one |
  449. | `/models` | models with their backtests; predicting; evaluating the specifications again |
  450. | `/scenarios` | what-ifs: a branch from a version with assumed changes, filled with what the models predict; comparison with the live twin or another scenario |
  451. | `/derived` | writing values that were worked out, marked as such |
  452. | `/search` | objects of any type by uid or name |
  453. | `/fidelity` | a live twin: its state, checks, each question's error per origin, incidents with every step taken, suspect signals; retrying and resolving an incident, a correction by hand |
  454. | `/actions` | action types; proposing, approving and rejecting; the record of every proposal; reconciling executed actions with what was observed |
  455. | `/control` | control loops: enabling, disabling, a run now |
  456. | `/lifecycle` | background jobs, the twin's own structured logs, what its scheduler asks for and when, the notifications it sent |
  457. - Every read takes `at`: `head`, a version id, `#<n>` or a time. A response
  458. names the version it read, and a cursor keeps reading that version.
  459. - Tables come as JSON (paged), Arrow or Parquet.
  460. - Errors are RFC 9457 problem documents with a code and a trace id.
  461. - `GET /api/v1/meta/health` of a live twin tells its clock, the age of its
  462. data against each question's staleness, its last run and the run in
  463. progress; of any twin, whether the worker runs and how many questions are
  464. validated, provisional and failing.
  465. - Reading never changes the twin, nor does answering a question or
  466. predicting. Values that were assumed or simulated are marked in their
  467. provenance and live on a scenario's branch, never among the observations.
  468. - `POST /block/training-set` gives a frame for training outside the twin: a
  469. target with its own lagged values, other properties and aggregates over
  470. neighbours in the graph, on a regular grid, each value the one in force at
  471. its time.
  472. The browser UI (graph, spreadsheet, history, questions and scenarios tabs, a
  473. version slider, follow head, a branch selector) reads the twin only through
  474. this API; the token stays on its server. The questions tab shows what each
  475. question may claim and asks one on a click; the scenarios tab draws a
  476. scenario against the live twin or another scenario; choosing a scenario in
  477. the header shows its branch in every tab, marked as not being the live twin.
  478. While a live twin moves on, a session that is not following stays on the
  479. version it shows and counts the newer ones; "Jump to head" or following takes
  480. it there, and the open tab redraws in place. The header shows the twin's
  481. time, the age of its data and its last run; the history tab lists sources,
  482. runs and rows set aside, and can start a run. The UI is published on the
  483. loopback address and answers only requests addressed to it as `127.0.0.1` or
  484. `localhost`; a request that would change something is accepted only from its
  485. own pages. A live twin's header carries its fidelity state in a word, and a
  486. line while it is failed; the fidelity tab draws each question's error against
  487. its tolerance with the corrections made, and lists incidents with their
  488. steps, suspect signals, checks and notifications; the actions tab lists
  489. action types, every proposal's record and the loops. Both tabs only show.
  490. The operator skill is generated from the manifest: how to reach the twin, the
  491. questions it was built to answer with their tolerances and exact definitions,
  492. recipes for every kind of request, and what each error code means.
  493. ## Delegated tasks
  494. Parts of a build (profiling a source, designing a mapping, training a model
  495. candidate) can be handed to other agents working at the same time. The
  496. toolkit, not the agents, keeps them from interfering:
  497. - a task declares the manifest paths it reads and the manifest keys it owns;
  498. its only other write zone is `tasks/<id>/output/`;
  499. - tasks share a batch only when none writes what another reads or writes;
  500. - one holder per task, by a lease that expires unless renewed;
  501. - a task never changes shared state: it returns a patch for its own keys and
  502. proposals for anything else, and the patches are applied one at a time;
  503. - a result is refused, with every reason listed, if it touches anything
  504. outside its zone or if what it read has changed since the batch opened.
  505. ```sh
  506. dta tasks plan --open # open the next batch of the current stage
  507. dta tasks claim <id> --holder <name>
  508. dta tasks submit <id> --token <token>
  509. dta tasks merge --all
  510. dta tasks status --verify
  511. ```
  512. ## Development
  513. Requires [uv](https://docs.astral.sh/uv/) and Docker. Python 3.12 is pinned.
  514. ```sh
  515. uv sync # create the environment with every package installed
  516. uv run pytest # run the tests; those that need a database are skipped
  517. uv run ruff check packages && uv run ruff format --check packages
  518. uv run dta --help # the command line tool
  519. ```
  520. Tests of the store and of everything built on it run against a real Dolt
  521. server, and the SQL connector's tests against a real PostgreSQL server. Put
  522. their settings in a `.env` file at the repository root (never committed):
  523. ```sh
  524. DTA_STORE_HOST=127.0.0.1
  525. DTA_STORE_PORT=13307
  526. DTA_STORE_USER=root
  527. DTA_STORE_PASSWORD=<choose one>
  528. DTA_STORE_DATABASE=dta_test
  529. DTA_CONNECT_PG_HOST=127.0.0.1
  530. DTA_CONNECT_PG_PORT=15433
  531. DTA_CONNECT_PG_USER=connect_test
  532. DTA_CONNECT_PG_PASSWORD=<choose one>
  533. DTA_CONNECT_PG_DATABASE=connect_test
  534. ```
  535. ```sh
  536. docker compose -p dta-store-dev --env-file .env -f packages/core/tests/store/compose.yaml up -d
  537. docker compose -p dta-connect-dev --env-file .env -f packages/core/tests/connect/compose.yaml up -d --wait
  538. docker build -f docker/runtime.Dockerfile -t dta-runtime:0.1.0 . # the image the stack tests run
  539. uv run --env-file .env pytest # the whole suite
  540. uv run --env-file .env pytest -m scale # full-size fixtures with performance thresholds
  541. docker compose -p dta-connect-dev --env-file .env -f packages/core/tests/connect/compose.yaml down -v
  542. docker compose -p dta-store-dev --env-file .env -f packages/core/tests/store/compose.yaml down -v
  543. ```
  544. - With the `DTA_CONNECT_PG_*` variables set, the PostgreSQL server must be up:
  545. those tests fail rather than skip when it is not.
  546. - The toolkit's stack tests build small twins in their own compose projects
  547. (`dta-m3tk-<random>`) on loopback ports 18800 to 18899 and remove them.
  548. They are skipped when Docker or the image is missing. Rebuild the image
  549. after changing `packages/core` or `packages/runtime`.
  550. - The UI tests drive headless Chrome through Selenium, on loopback ports
  551. 18720 to 18739, and are skipped when Chrome is not installed.
  552. - The launcher's tests (`tests/plugin/launcher`) run the toolkit's image named
  553. by `DTA_TEST_TOOLKIT_IMAGE` (default `dta-toolkit:main`) and the runtime's by
  554. `DTA_TEST_RUNTIME_IMAGE` (default `dta-runtime:main`). Their first-use tests
  555. run copies of the plugin with versions of their own, build
  556. `dta-toolkit:0.0.0-firstuse-<random>` and `dta-runtime:0.0.0-firstuse-<random>`
  557. with docker's layer cache, and remove exactly those tags.
  558. - A few tests (the generated client package of two sample twins, one worker
  559. image, two checks of a sync plan, one connector test against a live system)
  560. use a simulated system that is not part of this repository. Without it they
  561. are skipped.
  562. ### Releasing a version
  563. The version is in five places that tests hold together: `bin/VERSION` (the
  564. image tags the launcher builds and runs), `.claude-plugin/plugin.json`, the
  565. three packages' `pyproject.toml`, and the default of `runtime.image` in the
  566. manifest's models. Change them together, and the version badge at the top of
  567. this file with them. A plugin installed from a
  568. marketplace stays on the copy it has until the version in `plugin.json`
  569. changes, so a change that users should get needs a new version.
  570. Secrets are read only from environment variables. A process that needs a
  571. variable that is missing stops at startup and names it.
  572. ## Known limits
  573. - Linux only (see Install).
  574. - A control's `on_failure` is not a safety function: a twin that is stopped or
  575. cut off sends nothing.
  576. - No continuous integration is set up: the whole suite needs Docker, a Dolt
  577. and a PostgreSQL server and about 70 minutes.
  578. ## License
  579. Apache License 2.0; see `LICENSE`.
  580. The repository contains no third-party code. The images built on your machine
  581. install the project's Python dependencies from the package index under their
  582. own licences: mostly permissive (MIT, BSD, Apache-2.0), some under the Mozilla
  583. Public License 2.0 (among them `certifi`, `orjson`, `tqdm`), and
  584. `psycopg2-binary` under the LGPL.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.