digitally-twin-anything
The digitally-twin-anything plugin: build, accept and hand over a digital twin of a real system with the dta command line tool.
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Check it deeper on your machine
Its scripts were only partly scanned.
As a Claude Code plugin, install it and then ask Claude Code:
/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/DOWingard/digitally-twin-anything with scanskillsafety
Or from a terminal, clone it and run it:
git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/DOWingard/digitally-twin-anything'
It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.
Badge for your README
Markdown
[](/listings/dowingard--digitally-twin-anything)HTML
<a href="/listings/dowingard--digitally-twin-anything"><img src="/api/listings/dowingard--digitally-twin-anything/badge.svg" alt="findsafeskills safety rating" /></a>Install
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
The repository's root is a Claude Code plugin, and a marketplace that lists it. The plugin gives the agent the `build-twin` skill and the `dta` command; `dta` runs the toolkit in a container, so no Python is needed on the machine.
Requirements:
- Linux. It is developed and tested on x86-64 Linux only. On macOS and Windows the launcher refuses to run: there Docker runs containers in a virtual machine, and the toolkit cannot reach a twin the way it does on Linux (`DTA_UNSUPPORTED_OS=1` tries it anyway, untested). - Docker: the engine, running, and its command line with the buildx plugin (Docker Desktop has both; on Linux, the `docker-buildx-plugin` package or your distribution's equivalent). The user who runs the agent must be allowed to use Docker's socket. Compose is not needed on the machine itself: the toolkit's image carries its own docker client with compose. - A POSIX shell (`sh`) and the usual command line tools. - For the first use, network access to Docker Hub (`registry-1.docker.io`, `auth.docker.io`, `production.cloudfront.docker.com`) and the Python package index (`pypi.org`, `files.pythonhosted.org`), and about 2 GB of free disk.
Add the plugin from a clone of this repository:
git clone <this repository> digitally-twin-anything # for one session claude --plugin-dir ./digitally-twin-anything # or for good, through the repository's marketplace claude plugin marketplace add ./digitally-twin-anything claude plugin install digitally-twin-anything@digitally-twin-anything
Inside a session, `/plugin marketplace add <path>` and `/plugin install digitally-twin-anything@digitally-twin-anything` do the same. A marketplace added from a local path loads the plugin in place, so a `git pull` in the clone takes effect at the next session. Once the repository is hosted on GitHub, `<owner>/<repo>` replaces the path.
What the plugin can do on your machine
Read this before installing. The `dta` launcher starts the toolkit's container with Docker's socket, the host's network and the working directory mounted, running as you. Access to Docker's socket is equivalent to root on the machine: the toolkit uses it to build images and to start and stop the twin's own containers, and anything that can run `dta` can do as much. Install the plugin only where you would let the agent use Docker.
- A twin's services listen on the loopback address only (`127.0.0.1`), behind one API token; nothing is published to the network. - Secrets (source credentials, the token, the store's password) live in the workspace's `.env`, which the toolkit never prints, never puts into an export and reads values of by name only. - A twin calls the real system only through actions you declared and signed, and only when the executors add-on is switched on in the contract.
First use
The images are built on your machine, not downloaded ready-made. The first `dta` command, whatever it is, builds `dta-toolkit:<version>` (the toolkit) and `dta-runtime:<version>` (what a twin's own services run from), the version being `bin/VERSION`, then runs. The agent says so and waits; you see docker's progress on standard error, and all of it goes to `~/.local/state/dta/build.log` (`$XDG_STATE_HOME/dta/build.log` when that is set). Commands started while it builds wait for it instead of building again.
Measured on one machine:
| What | Measured | |---|---| | Downloads of a first build | about 0.6 GB, counted from the registry's manifests and the lock file: two base images (110 MB), the Python packages of both images and the installer | | A first build | 49 s on a fast connection with the base images already present; expect several minutes on an ordinary one | | A build of unchanged sources with docker's layer cache | about 2 s for both | | Size | `dta-toolkit` 712 MB and `dta-runtime` 1.21 GB as docker lists them; 1.8 GB on disk together, since they share their 120 MB base |
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
.claude-plugin/marketplace.json · Manifest · 15 lines
- {
- "name": "digitally-twin-anything",
- "description": "The digitally-twin-anything plugin: build, accept and hand over a digital twin of a real system with the dta command line tool.",
- "owner": {
- "name": "Derek Wingard"
- },
- "plugins": [
- {
- "name": "digitally-twin-anything",
- "source": "./",
- "description": "Build, accept and hand over a digital twin of any system whose data lives in databases, APIs or files, with the dta command line tool: a typed graph with full history, served by an API, a browser UI and an operator skill.",
- "category": "development"
- }
- ]
- }
README.md · README · 690 lines
- <h1 align="center">DTA: Digitally Twin Anything</h1>
- <h3 align="center">One agent plugin for building a working digital twin of anything a dataset can describe.</h3>
- <p align="center"><b>Point an agent at your data. Get a twin that answers your questions within the error you accept, checks itself, and corrects itself.</b></p>
- <p align="center">
- <img alt="version 0.1.0" src="https://img.shields.io/badge/version-v0.1.0-cb3837?style=flat-square">
- <img alt="Claude Code plugin" src="https://img.shields.io/badge/Claude%20Code-plugin-5a4fcf?style=flat-square">
- <img alt="skill: build-twin" src="https://img.shields.io/badge/skill-build--twin-000000?style=flat-square">
- <img alt="requires Docker" src="https://img.shields.io/badge/Docker-required-2ea44f?style=flat-square&logo=docker&logoColor=white">
- <img alt="platform: Linux" src="https://img.shields.io/badge/platform-Linux-555555?style=flat-square&logo=linux&logoColor=white">
- <img alt="license: Apache-2.0" src="https://img.shields.io/badge/license-Apache--2.0-f5c518?style=flat-square">
- </p>
- ---
- An agent plugin that builds a working digital twin of anything that can be
- described by a large enough dataset, then keeps it accurate and manages it.
- A twin built by this system is:
- - a typed graph of the target (objects, links, properties) with its full
- version history, so every read can ask "as of" any past version;
- - synced from the target's data sources on a schedule (a live twin) or loaded
- once (a static twin);
- - able to answer a declared set of questions within declared error tolerances,
- and to correct itself or raise a failure when it cannot;
- - served through one HTTP API, a browser UI and a generated skill file that
- tells an agent how to operate it;
- - one container stack that can be exported and moved to another host.
- Status: version 0.1.0, a first release for Linux. A static twin (loaded once
- from recorded data) and a live twin (kept in step with its sources, one
- version per sync interval) can be built, accepted and handed over, with
- functions, models selected by backtest, questions whose accuracy the twin
- measures, and what-if scenarios. A live twin checks its own fidelity, corrects itself or says that
- it failed, and can act on the real system through typed actions and control
- loops. A twin can be exported into one file and imported on another host
- with the same history. The repository is itself the agent plugin: it carries
- the skill by which an agent builds a twin, and a launcher that runs the
- toolkit from its image.
- ## Install
- The repository's root is a Claude Code plugin, and a marketplace that lists
- it. The plugin gives the agent the `build-twin` skill and the `dta` command;
- `dta` runs the toolkit in a container, so no Python is needed on the machine.
- Requirements:
- - Linux. It is developed and tested on x86-64 Linux only. On macOS and
- Windows the launcher refuses to run: there Docker runs containers in a
- virtual machine, and the toolkit cannot reach a twin the way it does on
- Linux (`DTA_UNSUPPORTED_OS=1` tries it anyway, untested).
- - Docker: the engine, running, and its command line with the buildx plugin
- (Docker Desktop has both; on Linux, the `docker-buildx-plugin` package or
- your distribution's equivalent). The user who runs the agent must be allowed
- to use Docker's socket. Compose is not needed on the machine itself: the
- toolkit's image carries its own docker client with compose.
- - A POSIX shell (`sh`) and the usual command line tools.
- - For the first use, network access to Docker Hub (`registry-1.docker.io`,
- `auth.docker.io`, `production.cloudfront.docker.com`) and the Python package
- index (`pypi.org`, `files.pythonhosted.org`), and about 2 GB of free disk.
- Add the plugin from a clone of this repository:
- ```sh
- git clone <this repository> digitally-twin-anything
- # for one session
- claude --plugin-dir ./digitally-twin-anything
- # or for good, through the repository's marketplace
- claude plugin marketplace add ./digitally-twin-anything
- claude plugin install digitally-twin-anything@digitally-twin-anything
- ```
- Inside a session, `/plugin marketplace add <path>` and `/plugin install
- digitally-twin-anything@digitally-twin-anything` do the same. A marketplace
- added from a local path loads the plugin in place, so a `git pull` in the
- clone takes effect at the next session. Once the repository is hosted on
- GitHub, `<owner>/<repo>` replaces the path.
- ### What the plugin can do on your machine
- Read this before installing. The `dta` launcher starts the toolkit's container
- with Docker's socket, the host's network and the working directory mounted,
- running as you. Access to Docker's socket is equivalent to root on the
- machine: the toolkit uses it to build images and to start and stop the twin's
- own containers, and anything that can run `dta` can do as much. Install the
- plugin only where you would let the agent use Docker.
- - A twin's services listen on the loopback address only (`127.0.0.1`), behind
- one API token; nothing is published to the network.
- - Secrets (source credentials, the token, the store's password) live in the
- workspace's `.env`, which the toolkit never prints, never puts into an
- export and reads values of by name only.
- - A twin calls the real system only through actions you declared and signed,
- and only when the executors add-on is switched on in the contract.
- ### First use
- The images are built on your machine, not downloaded ready-made. The first
- `dta` command, whatever it is, builds `dta-toolkit:<version>` (the toolkit)
- and `dta-runtime:<version>` (what a twin's own services run from), the
- version being `bin/VERSION`, then runs. The agent says so and waits; you see
- docker's progress on standard error, and all of it goes to
- `~/.local/state/dta/build.log` (`$XDG_STATE_HOME/dta/build.log` when that is
- set). Commands started while it builds wait for it instead of building again.
- Measured on one machine:
- | What | Measured |
- |---|---|
- | Downloads of a first build | about 0.6 GB, counted from the registry's manifests and the lock file: two base images (110 MB), the Python packages of both images and the installer |
- | A first build | 49 s on a fast connection with the base images already present; expect several minutes on an ordinary one |
- | A build of unchanged sources with docker's layer cache | about 2 s for both |
- | Size | `dta-toolkit` 712 MB and `dta-runtime` 1.21 GB as docker lists them; 1.8 GB on disk together, since they share their 120 MB base |
- | Time the launcher adds to every later command | about 30 ms before the container starts, the check for the two images included |
- The builds need only the repository's committed files; local files of a clone
- (`.env`, environments, caches, notes, tests) never enter the build. Base
- images are pinned by digest, Python packages by the lock file's versions and
- hashes, the installer by version and hash, and the build backend of the
- project's own packages by version.
- To build both images again, after changing the sources or to start over:
- ```sh
- DTA_REBUILD=1 ./digitally-twin-anything/bin/dta status
- ```
- To remove them (a later `dta` command builds them again):
- ```sh
- docker image rm dta-toolkit:0.1.0 dta-runtime:0.1.0
- ```
- Docker keeps the layers it built in its build cache (`docker builder du`
- shows how much); `docker builder prune` empties that cache, for every project
- on the machine, not this one alone. Twins you built have their own containers,
- volumes and images: `dta down --destroy-data` in a twin's workspace removes
- them.
- `bin/dta` documents every variable it reads in its header; `DTA_TOOLKIT_IMAGE`
- runs an image of your choosing instead, which is then never built.
- ### Try it
- `examples/bike-share` holds a data generator and the manifest of a static twin
- of what it writes: 21 days of a fictional bike-share operator (80 stations,
- 40,275 trips, a status row per station every 15 minutes; 19 MB).
- With the agent, which is what the plugin is for: generate the data (step 1
- below), start Claude Code with the plugin in an empty folder and say what you
- want, for instance "Build a digital twin of the bike-share export in
- ~/bikes/export; its description is in description.md there." The agent asks
- for the purpose, the questions and the error you accept for each, shows a
- summary to sign, builds the twin and reports whether it passed acceptance.
- By hand, with the example's manifest in place of the agent's work:
- ```sh
- REPO=/path/to/digitally-twin-anything
- export PATH="$REPO/bin:$PATH"
- dta --help # the first command builds the two images
- # 1. the data
- mkdir -p ~/bikes/twin
- docker run --rm -u "$(id -u):$(id -g)" \
- -v "$REPO/examples/bike-share:/example:ro" -v "$HOME/bikes:/out" \
- --entrypoint python dta-runtime:0.1.0 /example/generate.py /out/export
- # 2. the workspace and the manifest
- cd ~/bikes/twin
- dta init
- cp "$REPO/examples/bike-share/manifest.yaml" manifest.yaml
- cp "$REPO/examples/bike-share/ontology/cykelbo.yaml" ontology/
- dta ontology check --write
- dta env template
- # in .env: CYKELBO_EXPORT_DIR=<absolute path of ~/bikes/export>, two free
- # ports as DTA_API_PORT and DTA_UI_PORT, DTA_BASE_URL=http://127.0.0.1:<API port>
- dta env check
- # 3. the stages
- dta source probe
- dta profile
- dta manifest validate
- dta manifest confirm --all-user
- dta manifest validate
- dta manifest sign --non-interactive --approved-by "<your name>"
- dta build
- dta accept
- # 4. the twin
- dta questions list
- dta questions answer q_forecast
- dta down # stop it; --destroy-data also removes its data
- ```
- Run this way from an export of the committed files, on a machine that was busy
- with other work: every command before the build took 1 to 4 s; `dta build`
- 8 min 25 s (loading the data 105 s, backtesting three candidate models for 80
- stations 364 s); `dta accept` 23 s, all 26 checks passing. The forecast one
- hour ahead was measured at 0.55 bikes (MAE) against the 2 the manifest accepts,
- and the UI answered on `DTA_UI_PORT`. `dta manifest sign` without the two
- options shows the summary and asks for your name instead.
- ## Layout
- | Path | What it is |
- |---|---|
- | `.claude-plugin` | The plugin's manifest and the marketplace that lists it |
- | `bin/dta` | The launcher: builds the two images when they are absent, then runs the toolkit's image on the working directory |
- | `skills/build-twin` | The skill by which an agent builds, accepts and hands over a twin |
- | `tests/plugin` | Tests of the launcher (against real docker) and of the skill's texts against the tool |
- | `packages/core` | Shared library: structured logging, environment configuration, the manifest schema and its checks, ontology loading (LinkML), the versioned store, the connector protocol |
- | `packages/toolkit` | The `dta` command line tool: manifest checks, the gate each build stage ends in, delegated tasks, the twin's container stack derived from its manifest, build, acceptance and the operator skill |
- | `packages/runtime` | What runs inside a twin's stack: installing and opening a twin, reading its sources, loading them into the graph and keeping it in step with them, the HTTP API, the scheduler, the worker that runs functions and models, training frames, the shipped model families, scenarios and the browser UI |
- | `docker/toolkit.Dockerfile` | Image of the toolkit, with a docker client and its compose and buildx plugins |
- | `docker/runtime.Dockerfile` | Image of the twin's services: one image, one command per service |
- | `examples/bike-share` | A data generator and the manifest of a static twin of what it writes |
- | `spikes/storage` | Benchmark that selected Dolt as the versioned store |
- ## The store
- A twin's graph lives in a [Dolt](https://github.com/dolthub/dolt) database, one
- table per object or link type, generated from the twin's ontology.
- - A version is one commit. Its time is data time (the latest data timestamp the
- twin had seen), never the wall clock, so history can be replayed exactly.
- - Every stored value carries its provenance: where it came from, whether it
- was observed, inferred, assumed or simulated, how confident, and when it
- was true.
- - A change set is validated against the ontology and written whole or not at
- all. A committed version never changes.
- - Reads take a selector (`head`, a version id, `#<n>`, a time, or a range)
- and return Arrow tables.
- - Branches hold scenarios; simulated facts cannot reach the live branch.
- ## The manifest
- A twin is fully specified by one file, `manifest.yaml`, in its workspace. Each
- required field has a status:
- | Status | Meaning |
- |---|---|
- | unknown | no value yet |
- | proposed | a value is set and waits for the user to confirm it |
- | derived | computed by the toolkit from measurements |
- | confirmed | accepted by the field's owner (the user or the building agent) |
- The build may start only when no required field is unknown or proposed.
- `dta manifest validate` reports what is inconsistent in the manifest as
- written; `dta manifest gaps` lists every required field still unknown or
- waiting for confirmation; `dta status` shows the stage the build stands at
- and the result of each stage's gate.
- ## Building a twin
- A build moves through stages; each ends in a gate, and `dta status` shows where
- the build stands. All commands act on a workspace (`--workspace`, default the
- current directory) and print one JSON document with `--json`.
- | Stage | Gate passes when | Command |
- |---|---|---|
- | 0 intake | the workspace and a manifest exist | `dta init` |
- | 1 contract | the user has confirmed the contract fields | `dta manifest confirm` |
- | 2 access | every source can be read from inside the twin's stack | `dta source probe` |
- | 3 profiling | every source has a profile in the manifest | `dta profile` |
- | 4 design | the manifest and its ontology are consistent | `dta manifest validate` |
- | 5 gaps | no required field is open, and the user has signed | `dta manifest sign` |
- | 6 build | the stack runs and serves this manifest | `dta build` |
- | 7 acceptance | every acceptance check passes | `dta accept` |
- | 8 handover | the operator skill is installed | `dta install-skill` |
- ```sh
- dta init
- # write manifest.yaml and the ontology file it names (LinkML)
- dta ontology check --write # confirm the ontology's types in the manifest
- dta env template # .env.example and .env; fill in what is not generated
- dta source probe # reach every source from inside the stack
- dta profile # measure the sources; profiles go into the manifest
- dta sync derive --write # a live twin: its sync interval, from the contract and the profiles
- dta manifest validate
- dta manifest confirm --all-user
- dta manifest validate # confirming changes the manifest; the design gate is of the result
- dta manifest sign
- dta build # image, store, install, load, API, contract, sync (live), code,
- # functions, models, questions, UI, skill
- dta accept # acceptance report, or a deficit report
- dta install-skill # the skill an agent operates the twin with
- ```
- - Sources are only ever read from inside the twin's stack, never from the host,
- so an address, its credentials and its network mean the same at build time
- and at run time. A source in another compose project is reached through
- `runtime.networks` in the manifest.
- - `dta build` can be run again at any time. Every step changes only what
- differs; the load is not repeated while the twin is where the last load left
- it. A failing step stops the build and leaves what was done in place.
- - `dta accept` runs every check and reports all failures together: the API
- against its own OpenAPI document (generated requests, fixed seed), the five
- data-readiness requirements, constraint violations in the load, coverage of
- every property the contract names, the UI, the skill and that the skill
- lists every endpoint the API serves; that every question is answered, that
- none is measured outside its tolerance, that every model specification has
- an accepted model, and that the worker runs.
- - `dta install-skill` refuses a twin that has not been accepted unless
- `--provisional` is given; the handover report then lists what has not passed.
- - `dta up`, `dta down` and `dta logs` manage the stack. `dta down
- --destroy-data` removes the twin's data and takes back the build and
- acceptance gates.
- - No value of `.env` is printed, logged or written to a report.
- ### A live twin
- A twin whose contract says `tier: live` keeps reading its sources after the
- load. The manifest's `sync.interval` (Δ) is how often; `dta sync derive`
- computes it as the longest of three: the sources' refresh period, their
- rate-limit period, and what the questions need, which is the shorter of the
- tightest staleness less the source's own delay and half the fastest change
- the profiles measured.
- - Time is the twin's clock, never the machine's: `runtime.clock` is `wall` or
- `source:<id>`, a source that tells the time. Intervals are counted from the
- Unix epoch; a run takes every interval that has closed and writes one
- version for each, so a twin that was stopped catches up interval by interval.
- - A scheduler service in the stack starts the runs through the API. One run
- at a time; a run asked for twice with the same key is started once.
- - Each run records itself in the twin (`TwinSyncRun`): what it read and wrote
- per source, the rows it set aside with the rule each broke, the rows that
- arrived for an interval already written. A source that fails does not stop
- the others; the run is then `partial`.
- - A dataset declared under `sources.<id>.incremental` is read on from where
- the last run stopped instead of whole.
- - Freshness is measured per question: the age of the newest data of the
- sources the question is answered from, against the question's staleness.
- - `dta build` loads history once, only while no run has written; it stops the
- scheduler before a load and its `sync` step starts the scheduler, makes one
- run and fails unless that run synced every source. `dta accept` adds: the
- scheduler runs, the latest run succeeded, no source is failing, every
- question is fresh.
- ```sh
- dta sync status # the clock, freshness per question, the sources
- dta sync run # one run now; waits and prints its report
- dta sync run --source scada # only these sources
- dta sync run --replay scada.readings # read one dataset again, whole
- ```
- ### Functions, models and questions
- The manifest declares what the twin computes; `dta build` registers it in the
- twin, where it is versioned like everything else.
- - A function answers a question from the graph. Eight are built in
- (`latest`, `below`, `above`, `integrate`, `paths`, `aggregate`, `series`,
- and `forecast`, which asks a question's model); others are Python functions
- of the twin's own, kept in the directory `runtime.code` names. Extra
- packages they need are listed in `runtime.worker_requirements`; the build
- makes a worker image with them.
- - A model specification names a question, a target property, candidate
- families and a backtest. Shipped families: `persistence`, `seasonal_naive`,
- `theta`, `ets`, `arima`, `mstl`, `ridge_lags`, `gbm_lags`; a twin can bring
- families of its own (a hydraulic simulation of a pipe network, say).
- The build backtests every candidate over the twin's history, on a rolling
- origin, and makes the best one that is within the question's tolerance and
- better than the baseline the question's active model.
- - A question is `validated` (measured within its tolerance), `provisional`
- (not measured, or too little history) or `failing`. Every answer says which.
- - Models and code of the twin's own run in the worker, a separate container
- with limits on memory and time that reads the twin only through the API;
- code of the twin's own runs there in a process of its own per call.
- - A build evaluates a model again only when its definition, its code or its
- question changed; `dta models evaluate` does it on request, for instance
- after new data.
- ```sh
- dta register # functions, models and questions of the manifest
- dta models evaluate [--spec ID]
- dta models list
- dta questions list
- dta questions measure
- dta questions answer ID [--arg name=value] [--at VERSION]
- dta scenarios list
- ```
- ### Fidelity, actions and control
- A live twin measures itself on its own clock. At every `fidelity.interval` it
- scores what its models predicted for each predictive question against what was
- then observed, and is in one state: `unknown` (no check yet), `healthy`,
- `correcting` or `failed`.
- - **Breach.** A question's error at an origin o is its metric over (o, o + h].
- Its window error E is the mean over the origins scored in the last
- `fidelity.window`; with at least 3 of them and E above the question's
- tolerance ε an incident opens.
- - **Correction, rung by rung, without a human.** Recalibrate the active model
- on recent data; reselect among the candidates the manifest declares; ask the
- managing agent to re-model (a notification), and fail when nobody did within
- `correction.remodel_timeout`. A correction is accepted only if it passes the
- specification's acceptance on the recent history, and an incident resolves
- only on new evidence within ε.
- - **Stale data, stuck and biased signals.** A question whose data is older
- than its staleness opens an incident. An observed signal that used to change
- and stopped while its source kept delivering is called stuck when a run
- without a change that long has probability below 1e-7 under its own history.
- A smooth signal, whose changes keep their sign, is not judged by its run:
- after 3 intervals without a change it is stuck when its reading lies more
- than half the tolerance from the twin's own forecast made from its last
- change. A signal whose residual shifts by at least ε against every model of
- its property is called biased.
- - **The value of a suspect signal.** Its readings are kept on its record and
- not written to the object; the value in force is the twin's own estimate,
- marked as inferred: the forecast of a question of that property, from the
- newest version at which everything the model reads was observed. An
- estimate never rests on an estimate. A model that does not read the signal
- is forecast again at every check. One that reads it is forecast once, by the
- question with the longest horizon; past that horizon, or when the estimate
- is not finite or outside the signal's range, the value is unknown, and an
- answer or an action that needs it is refused with `fidelity.input_unknown`,
- naming the signal, until the signal is sound again. A model's own code says
- what its predictions read in `models.<id>.reads`; a code model that does not
- say is taken to read everything. Forecasts that started from a frozen, an
- inferred or an unknown value are not scored, so a failed sensor opens no
- breach and starts no correction of a model.
- - **Actions.** The manifest declares what may be done to the real system: an
- action type with typed parameters, the control it carries out with its
- bounds and autonomy (`advisory`, `supervised`, `autonomous`), and an HTTP
- call with its variables named, never stored. Every proposal is judged and
- recorded; an executed one is compared with what was observed after its
- settle time, and a mismatch is an incident.
- - **Interlock.** An autonomous action is executed while the twin is healthy,
- or correcting with no stale-data or mismatch incident open; a supervised one
- on a recorded approval, unless the twin failed. An autonomous action on a
- value that is unknown is blocked. A failed twin executes nothing.
- - **Control loops** run a policy function of the twin on an interval and
- propose what it returns. A loop is registered disabled; enabling it is a
- decision recorded with a name.
- - **Notifications** go out as CloudEvents to the address in the variable the
- contract names (or to the log): state changes, incidents, re-model requests,
- failed sync runs, pending approvals, blocked actions, mismatches.
- A twin that is blocked or failed leaves the real system in the state of its
- last executed command: handing a control back is done on the real system.
- ```sh
- dta fidelity status | checks | incidents | incident ID | errors QUESTION | signals
- dta fidelity retry ID --by NAME | resolve ID --by NAME | correct QUESTION --rung RUNG | clear UID --by NAME
- dta actions list | propose ACTION --object UID --param k=v --reason TEXT | records | approve UID --by NAME | reject UID --by NAME
- dta control list | enable LOOP --by NAME | disable LOOP --by NAME | run LOOP
- dta notifications
- ```
- ### When the twin fails, and manifest versions
- A twin whose fidelity state is `failed` executes nothing, so the real system
- keeps the last thing the twin set. Each control therefore carries the user's
- choice, `contract.controls.<control>.on_failure`, in the signed contract:
- | `mode` | What is sent, once per failure, to each object the twin commanded |
- |---|---|
- | `hold` | Nothing. |
- | `release` | A declared action that hands the object back to the real system's own control. |
- | `safe_value` | The control's own action with a constant inside the control's bounds. |
- It goes past the interlock and the control's autonomy and uses no model;
- `after_stale` sends it as well when the twin's data has been too old for that
- long. An autonomous control whose action calls the real system must make the
- choice. This is not a safety function: a twin that is stopped or cut off sends
- nothing, so protection against that belongs in the real system.
- While a breach incident waits for a managing agent to re-model, the twin keeps
- correcting on newer data: up to `fidelity.correction.retry_budget` cycles, each
- after a wait that doubles from `retry_backoff`. Only the request's time-out
- fails the twin.
- `manifest_version` states which version of the manifest's format a manifest is
- written in (now 2). A field newer than the stated version is refused, a
- manifest newer than the toolkit is refused (`manifest.too_new`), and before
- `dta build`, `dta up` and an import start anything the toolkit asks the runtime
- image what it reads and refuses an older one (`runtime.too_old`).
- ### Moving a twin, and the add-ons
- ```sh
- dta export [--out FILE|FOLDER] # one bundle: workspace, every version of the store, artifacts; no secrets
- dta import BUNDLE [--into DIR] # unpack, fill .env, run again: the store is restored and its heads verified
- ```
- - A bundle holds the manifest, the ontology, the twin's code, the record of the
- discussion, the reports, a backup of the versioned store and the artifact
- volume. `.env` is never in it. Export runs while the twin runs.
- - Import restores the store under the same commit ids: it compares the head of
- every branch with what the bundle names and refuses the import when one
- differs. Access, build, acceptance and handover are run again on the new
- host; the signature is carried.
- - A source that is a folder on the exporting machine is named in the bundle's
- warnings and is not carried.
- Three add-ons, each built only when the contract selects it and removed when
- it is deselected:
- | `contract.addons` | What is built |
- |---|---|
- | `sdk_python` | a Python package `twin_<id>` in `sdk/python/`: typed models from the ontology, a client generated from the twin's OpenAPI document, one accessor per type and one method per question; every call takes `at`; address and token from `DTA_BASE_URL` and `DTA_API_TOKEN` |
- | `mcp` | a service that offers every API operation as an MCP tool on `127.0.0.1:$DTA_MCP_PORT`; it holds no token and passes the caller's on |
- | `readonly_sql` | the store's SQL port on `127.0.0.1:$DTA_SQL_PORT` and a user that can only `SELECT`, for queries with `AS OF`, the commit log and diffs |
- ```sh
- dta generate sdk [--force]
- ```
- ## What a built twin serves
- One HTTP API under `/api/v1`, bound to the loopback address, with one bearer
- token. `GET /api/v1/openapi.json` describes every endpoint.
- | Group | For |
- |---|---|
- | `/meta` | identity, health, the manifest and the ontology at a version |
- | `/objects` | objects of a type: filter, sort, page, count; a search by uid across types; one object with the provenance of each value; its links and its history |
- | `/graph` | neighbours, shortest paths, subgraphs, the edge list, each over all links or only those that pass a filter on their properties |
- | `/history` | versions, the difference between two versions, branches |
- | `/block` | whole node and edge tables, and slices over a range of versions, as Arrow or Parquet |
- | `/sync` | a live twin: its sources and how far each has been read, its runs and the rows they set aside, starting a run |
- | `/questions` | the questions of the contract with what each may claim; answering one; measuring them |
- | `/functions` | the twin's functions with their arguments; running one |
- | `/models` | models with their backtests; predicting; evaluating the specifications again |
- | `/scenarios` | what-ifs: a branch from a version with assumed changes, filled with what the models predict; comparison with the live twin or another scenario |
- | `/derived` | writing values that were worked out, marked as such |
- | `/search` | objects of any type by uid or name |
- | `/fidelity` | a live twin: its state, checks, each question's error per origin, incidents with every step taken, suspect signals; retrying and resolving an incident, a correction by hand |
- | `/actions` | action types; proposing, approving and rejecting; the record of every proposal; reconciling executed actions with what was observed |
- | `/control` | control loops: enabling, disabling, a run now |
- | `/lifecycle` | background jobs, the twin's own structured logs, what its scheduler asks for and when, the notifications it sent |
- - Every read takes `at`: `head`, a version id, `#<n>` or a time. A response
- names the version it read, and a cursor keeps reading that version.
- - Tables come as JSON (paged), Arrow or Parquet.
- - Errors are RFC 9457 problem documents with a code and a trace id.
- - `GET /api/v1/meta/health` of a live twin tells its clock, the age of its
- data against each question's staleness, its last run and the run in
- progress; of any twin, whether the worker runs and how many questions are
- validated, provisional and failing.
- - Reading never changes the twin, nor does answering a question or
- predicting. Values that were assumed or simulated are marked in their
- provenance and live on a scenario's branch, never among the observations.
- - `POST /block/training-set` gives a frame for training outside the twin: a
- target with its own lagged values, other properties and aggregates over
- neighbours in the graph, on a regular grid, each value the one in force at
- its time.
- The browser UI (graph, spreadsheet, history, questions and scenarios tabs, a
- version slider, follow head, a branch selector) reads the twin only through
- this API; the token stays on its server. The questions tab shows what each
- question may claim and asks one on a click; the scenarios tab draws a
- scenario against the live twin or another scenario; choosing a scenario in
- the header shows its branch in every tab, marked as not being the live twin.
- While a live twin moves on, a session that is not following stays on the
- version it shows and counts the newer ones; "Jump to head" or following takes
- it there, and the open tab redraws in place. The header shows the twin's
- time, the age of its data and its last run; the history tab lists sources,
- runs and rows set aside, and can start a run. The UI is published on the
- loopback address and answers only requests addressed to it as `127.0.0.1` or
- `localhost`; a request that would change something is accepted only from its
- own pages. A live twin's header carries its fidelity state in a word, and a
- line while it is failed; the fidelity tab draws each question's error against
- its tolerance with the corrections made, and lists incidents with their
- steps, suspect signals, checks and notifications; the actions tab lists
- action types, every proposal's record and the loops. Both tabs only show.
- The operator skill is generated from the manifest: how to reach the twin, the
- questions it was built to answer with their tolerances and exact definitions,
- recipes for every kind of request, and what each error code means.
- ## Delegated tasks
- Parts of a build (profiling a source, designing a mapping, training a model
- candidate) can be handed to other agents working at the same time. The
- toolkit, not the agents, keeps them from interfering:
- - a task declares the manifest paths it reads and the manifest keys it owns;
- its only other write zone is `tasks/<id>/output/`;
- - tasks share a batch only when none writes what another reads or writes;
- - one holder per task, by a lease that expires unless renewed;
- - a task never changes shared state: it returns a patch for its own keys and
- proposals for anything else, and the patches are applied one at a time;
- - a result is refused, with every reason listed, if it touches anything
- outside its zone or if what it read has changed since the batch opened.
- ```sh
- dta tasks plan --open # open the next batch of the current stage
- dta tasks claim <id> --holder <name>
- dta tasks submit <id> --token <token>
- dta tasks merge --all
- dta tasks status --verify
- ```
- ## Development
- Requires [uv](https://docs.astral.sh/uv/) and Docker. Python 3.12 is pinned.
- ```sh
- uv sync # create the environment with every package installed
- uv run pytest # run the tests; those that need a database are skipped
- uv run ruff check packages && uv run ruff format --check packages
- uv run dta --help # the command line tool
- ```
- Tests of the store and of everything built on it run against a real Dolt
- server, and the SQL connector's tests against a real PostgreSQL server. Put
- their settings in a `.env` file at the repository root (never committed):
- ```sh
- DTA_STORE_HOST=127.0.0.1
- DTA_STORE_PORT=13307
- DTA_STORE_USER=root
- DTA_STORE_PASSWORD=<choose one>
- DTA_STORE_DATABASE=dta_test
- DTA_CONNECT_PG_HOST=127.0.0.1
- DTA_CONNECT_PG_PORT=15433
- DTA_CONNECT_PG_USER=connect_test
- DTA_CONNECT_PG_PASSWORD=<choose one>
- DTA_CONNECT_PG_DATABASE=connect_test
- ```
- ```sh
- docker compose -p dta-store-dev --env-file .env -f packages/core/tests/store/compose.yaml up -d
- docker compose -p dta-connect-dev --env-file .env -f packages/core/tests/connect/compose.yaml up -d --wait
- docker build -f docker/runtime.Dockerfile -t dta-runtime:0.1.0 . # the image the stack tests run
- uv run --env-file .env pytest # the whole suite
- uv run --env-file .env pytest -m scale # full-size fixtures with performance thresholds
- docker compose -p dta-connect-dev --env-file .env -f packages/core/tests/connect/compose.yaml down -v
- docker compose -p dta-store-dev --env-file .env -f packages/core/tests/store/compose.yaml down -v
- ```
- - With the `DTA_CONNECT_PG_*` variables set, the PostgreSQL server must be up:
- those tests fail rather than skip when it is not.
- - The toolkit's stack tests build small twins in their own compose projects
- (`dta-m3tk-<random>`) on loopback ports 18800 to 18899 and remove them.
- They are skipped when Docker or the image is missing. Rebuild the image
- after changing `packages/core` or `packages/runtime`.
- - The UI tests drive headless Chrome through Selenium, on loopback ports
- 18720 to 18739, and are skipped when Chrome is not installed.
- - The launcher's tests (`tests/plugin/launcher`) run the toolkit's image named
- by `DTA_TEST_TOOLKIT_IMAGE` (default `dta-toolkit:main`) and the runtime's by
- `DTA_TEST_RUNTIME_IMAGE` (default `dta-runtime:main`). Their first-use tests
- run copies of the plugin with versions of their own, build
- `dta-toolkit:0.0.0-firstuse-<random>` and `dta-runtime:0.0.0-firstuse-<random>`
- with docker's layer cache, and remove exactly those tags.
- - A few tests (the generated client package of two sample twins, one worker
- image, two checks of a sync plan, one connector test against a live system)
- use a simulated system that is not part of this repository. Without it they
- are skipped.
- ### Releasing a version
- The version is in five places that tests hold together: `bin/VERSION` (the
- image tags the launcher builds and runs), `.claude-plugin/plugin.json`, the
- three packages' `pyproject.toml`, and the default of `runtime.image` in the
- manifest's models. Change them together, and the version badge at the top of
- this file with them. A plugin installed from a
- marketplace stays on the copy it has until the version in `plugin.json`
- changes, so a change that users should get needs a new version.
- Secrets are read only from environment variables. A process that needs a
- variable that is missing stops at startup and names it.
- ## Known limits
- - Linux only (see Install).
- - A control's `on_failure` is not a safety function: a twin that is stopped or
- cut off sends nothing.
- - No continuous integration is set up: the whole suite needs Docker, a Dolt
- and a PostgreSQL server and about 70 minutes.
- ## License
- Apache License 2.0; see `LICENSE`.
- The repository contains no third-party code. The images built on your machine
- install the project's Python dependencies from the package index under their
- own licences: mostly permissive (MIT, BSD, Apache-2.0), some under the Mozilla
- Public License 2.0 (among them `certifi`, `orjson`, `tqdm`), and
- `psycopg2-binary` under the LGPL.
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.