Skip to content
Back to search

apache-hamilton

MarketplaceWorks with Claude CodeGitHub

Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026. Its scripts were only partly scanned.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/apache/hamilton with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/apache/hamilton'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/apache--hamilton/badge.svg)](/listings/apache--hamilton)

HTML

<a href="/listings/apache--hamilton"><img src="/api/listings/apache--hamilton/badge.svg" alt="findsafeskills safety rating" /></a>

Install

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Apache Hamilton supports Python 3.8+. We include the optional `visualization` dependency to display our Apache Hamilton DAG. For visualizations, [Graphviz](https://graphviz.org/download/) needs to be installed on your system separately.

pip install "apache-hamilton[visualization]"

To use the Apache Hamilton UI, install the `ui` and `sdk` dependencies.

pip install "apache-hamilton[ui,sdk]"

To try Apache Hamilton in the browser, visit [www.tryhamilton.dev](https://www.tryhamilton.dev/?utm_source=README)

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/marketplace.json · Manifest · 46 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. {
  2. "name": "apache-hamilton",
  3. "owner": {
  4. "name": "Hamilton Team",
  5. "email": "dev@hamilton.apache.org"
  6. },
  7. "metadata": {
  8. "description": "Official Apache Hamilton plugins for Claude Code - create DAGs, apply decorators, debug dataflows, and optimize pipelines",
  9. "version": "2.0.0"
  10. },
  11. "plugins": [
  12. {
  13. "name": "hamilton",
  14. "source": {
  15. "source": "git-subdir",
  16. "url": "https://github.com/apache/hamilton.git",
  17. "path": ".claude-plugin"
  18. },
  19. "description": "Expert AI assistant for Apache Hamilton framework development - create DAGs, apply decorators, debug dataflows, and optimize pipelines",
  20. "version": "2.0.0",
  21. "author": {
  22. "name": "Hamilton Team",
  23. "email": "dev@hamilton.apache.org"
  24. },
  25. "homepage": "https://github.com/apache/hamilton",
  26. "repository": "https://github.com/apache/hamilton",
  27. "license": "Apache-2.0",
  28. "keywords": [
  29. "hamilton",
  30. "dag",
  31. "dataflow",
  32. "workflow",
  33. "pipeline",
  34. "data-engineering",
  35. "ml-ops",
  36. "feature-engineering",
  37. "llm",
  38. "rag",
  39. "async",
  40. "spark",
  41. "observability"
  42. ],
  43. "category": "data-engineering"
  44. }
  45. ]
  46. }
README.md · README · 324 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. <!--
  2. Licensed to the Apache Software Foundation (ASF) under one
  3. or more contributor license agreements. See the NOTICE file
  4. distributed with this work for additional information
  5. regarding copyright ownership. The ASF licenses this file
  6. to you under the Apache License, Version 2.0 (the
  7. "License"); you may not use this file except in compliance
  8. with the License. You may obtain a copy of the License at
  9. http://www.apache.org/licenses/LICENSE-2.0
  10. Unless required by applicable law or agreed to in writing,
  11. software distributed under the License is distributed on an
  12. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
  13. KIND, either express or implied. See the License for the
  14. specific language governing permissions and limitations
  15. under the License.
  16. -->
  17. <div align="center">
  18. <h1><img src="https://github.com/apache/hamilton/assets/2328071/feb6abaa-b6d5-4271-a320-0ae4a18d8aa7" width="50"/> Apache Hamilton — portable & expressive <br> data transformation DAGs</h1>
  19. <a href='https://hamilton.apache.org/?badge=latest'>
  20. <img src='https://readthedocs.org/projects/hamilton/badge/?version=latest' alt='Documentation Status' />
  21. </a><a href="https://www.python.org/downloads/" target="_blank">
  22. <img src="https://img.shields.io/badge/python-3.11%20|%203.12%20|%203.13%20|%203.14%20|%203.15-blue.svg" alt="Python supported"/>
  23. </a>
  24. <a href="https://pypi.org/project/apache-hamilton/" target="_blank">
  25. <img src="https://badge.fury.io/py/apache-hamilton.svg" alt="PyPi Version"/>
  26. </a>
  27. <a href="https://pepy.tech/project/apache-hamilton" target="_blank">
  28. <img src="https://pepy.tech/badge/apache-hamilton" alt="Total Downloads"/>
  29. </a>
  30. <a href="https://pepy.tech/project/apache-hamilton" target="_blank">
  31. <img src="https://static.pepy.tech/badge/apache-hamilton/month" alt="Total Monthly Downloads"/>
  32. </a>
  33. <br/>
  34. <a href="https://join.slack.com/t/hamilton-opensource/shared_invite/zt-2niepkra8-DGKGf_tTYhXuJWBTXtIs4g" target="_blank">
  35. <img src="https://img.shields.io/badge/Apache Hamilton-Join-purple.svg?logo=slack" alt="Apache Hamilton Slack"/>
  36. </a>
  37. <a href="https://twitter.com/hamilton_os" target="_blank">
  38. <img src="https://img.shields.io/badge/HamiltonOS-Follow-purple.svg?logo=X"/>
  39. </a>
  40. </div>
  41. <br></br>
  42. > **Disclaimer**
  43. >
  44. > Apache Hamilton is an effort undergoing incubation at the Apache Software Foundation (ASF), sponsored by the Apache Incubator PMC.
  45. >
  46. > Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects.
  47. >
  48. > While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
  49. Apache Hamilton (incubating) is a lightweight Python library for directed acyclic graphs (DAGs) of data transformations. Your DAG is **portable**; it runs anywhere Python runs, whether it's a script, notebook, Airflow pipeline, FastAPI server, etc. Your DAG is **expressive**; Apache Hamilton has extensive features to define and modify the execution of a DAG (e.g., data validation, experiment tracking, remote execution).
  50. To create a DAG, write regular Python functions that specify their dependencies with their parameters. As shown below, it results in readable code that can always be visualized. Apache Hamilton loads that definition and automatically builds the DAG for you!
  51. <div align="center">
  52. <img src="./docs/_static/abc_highlight.png" alt="Create a project" width="65%"/>
  53. </div>
  54. <div align="center">
  55. Functions <code>B()</code> and <code>C()</code> refer to function <code>A</code> via their parameters
  56. </div>
  57. <br>
  58. Apache Hamilton brings modularity and structure to any Python application moving data: ETL pipelines, ML workflows, LLM applications, RAG systems, BI dashboards, and the [Apache Hamilton UI](https://hamilton.apache.org/concepts/ui) allows you to automatically visualize, catalog, and monitor execution.
  59. > Apache Hamilton is great for DAGs, but if you need loops or conditional logic to create an LLM agent or a simulation, take a look at our sister library [Burr](https://github.com/apache/burr) 🤖 .
  60. # Installation
  61. Apache Hamilton supports Python 3.11+. We include the optional `visualization` dependency to display our Apache Hamilton DAG. For visualizations, [Graphviz](https://graphviz.org/download/) needs to be installed on your system separately.
  62. ```bash
  63. pip install "apache-hamilton[visualization]"
  64. ```
  65. To use the Apache Hamilton UI, install the `ui` and `sdk` dependencies.
  66. ```bash
  67. pip install "apache-hamilton[ui,sdk]"
  68. ```
  69. To try Apache Hamilton in the browser, visit [www.tryhamilton.dev](https://www.tryhamilton.dev/?utm_source=README)
  70. # Why use Apache Hamilton?
  71. Data teams write code to deliver business value, but few have the resources to standardize practices and provide quality assurance. Moving from proof-of-concept to production and cross-function collaboration (e.g., data science, engineering, ops) remain challenging for teams, big or small. Apache Hamilton is designed to help throughout a project's lifecycle:
  72. - **Separation of concerns**. Apache Hamilton separates the DAG "definition" and "execution" which lets data scientists focus on solving problems and engineers manage production pipelines.
  73. - **Effective collaboration**. The [Apache Hamilton UI provides a shared interface](https://hamilton.apache.org/hamilton-ui/ui/) for teams to inspect results and debug failures throughout the development cycle.
  74. - **Low-friction dev to prod**. Use `@config.when()` to modify your DAG between execution environments instead of error-prone `if/else` feature flags. The notebook extension prevents the pain of migrating code from a notebook to a Python module.
  75. - **Portable transformations**. Your DAG is [independent of infrastructure or orchestration](https://blog.dagworks.io/publish/posts/detail/145543927?referrer=%2Fpublish%2Fposts), meaning you can develop and debug locally and reuse code across contexts (local, Airflow, FastAPI, etc.).
  76. - **Maintainable DAG definition**. Apache Hamilton [automatically builds the DAG from a single line of code whether it has 10 or 1000 nodes](https://hamilton.apache.org/concepts/driver/). It can also assemble multiple Python modules into a pipeline, encouraging modularity.
  77. - **Expressive DAGs**. [Function modifiers](https://hamilton.apache.org/concepts/function-modifiers/) are a unique feature to keep your code [DRY](https://en.wikipedia.org/wiki/Don't_repeat_yourself) and reduce the complexity of maintaining large DAGs. Other frameworks inevitably lead to code redundancy or bloated functions.
  78. - **Built-in coding style**. The Apache Hamilton DAG is [defined using Python functions](https://hamilton.apache.org/concepts/node/), encouraging modular, easy-to-read, self-documenting, and unit testable code.
  79. - **Data and schema validation**. Decorate functions with `@check_output` to validate output properties, and raise warnings or exceptions. Add the `SchemaValidator()` adapter to automatically inspect dataframe-like objects (pandas, polars, Ibis, etc.) to track and validate their schema.
  80. - **Built for plugins**. Apache Hamilton is designed to play nice with all tools and provides the right abstractions to create custom integrations with your stack. Our lively community will help you build what you need!
  81. # Apache Hamilton UI
  82. You can track the execution of your Apache Hamilton DAG in the [Apache Hamilton UI](https://hamilton.apache.org/hamilton-ui/ui/). It automatically populates a data catalog with lineage / tracing and provides execution observability to inspect results and debug errors. You can run it as a [local server](https://hamilton.apache.org/hamilton-ui/ui/#local-mode) or a [self-hosted application using Docker](https://hamilton.apache.org/hamilton-ui/ui/#docker-deployed-mode).
  83. <p align="center">
  84. <img src="./docs/_static/hamilton_1.jpeg" alt="Description1" width="30%" style="margin-right: 20px;"/>
  85. <img src="./docs/_static/hamilton_2.jpeg" alt="Description2" width="30%" style="margin-right: 20px;"/>
  86. <img src="./docs/_static/hamilton_3.jpeg" alt="Description3" width="30%"/>
  87. </p>
  88. <p align="center">
  89. <em>DAG catalog, automatic dataset profiling, and execution tracking</em>
  90. </p>
  91. ## Get started with the Apache Hamilton UI
  92. 1. To use the Apache Hamilton UI, install the dependencies (see `Installation` section) and start the server with
  93. ```bash
  94. hamilton ui
  95. ```
  96. 2. On the first connection, create a `username` and a new project (the `project_id` should be `1`).
  97. <div align="center">
  98. <img src="./docs/_static/new_project.png" alt="Create a project" width="70%"/>
  99. </div>
  100. <br>
  101. 3. Track your Apache Hamilton DAG by creating a `HamiltonTracker` object with your `username` and `project_id` and adding it to your `Builder`. Now, your DAG will appear in the UI's catalog and all executions will be tracked!
  102. ```python
  103. from hamilton import driver
  104. from hamilton_sdk.adapters import HamiltonTracker
  105. import my_dag
  106. # use your `username` and `project_id`
  107. tracker = HamiltonTracker(
  108. username="my_username",
  109. project_id=1,
  110. dag_name="hello_world",
  111. )
  112. # adding the tracker to the `Builder` will add the DAG to the catalog
  113. dr = (
  114. driver.Builder()
  115. .with_modules(my_dag)
  116. .with_adapters(tracker) # add your tracker here
  117. .build()
  118. )
  119. # executing the `Driver` will track results
  120. dr.execute(["C"])
  121. ```
  122. # Documentation & learning resources
  123. * 📚 See the [official documentation](https://hamilton.apache.org/) to learn about the core concepts of Apache Hamilton.
  124. * 👨‍🏫 Consult the [examples on GitHub](https://github.com/apache/hamilton/tree/main/examples) to learn about specific features or integrations with other frameworks.
  125. * 📰 The [DAGWorks blog](https://blog.dagworks.io/) includes guides about how to build a data platform and narrative tutorials.
  126. * 📺 Find video tutorials on the [DAGWorks YouTube channel](https://www.youtube.com/@DAGWorks-Inc)
  127. * 📣 Reach out via the [Apache Hamilton Slack community](https://join.slack.com/t/hamilton-opensource/shared_invite/zt-2niepkra8-DGKGf_tTYhXuJWBTXtIs4g) for help and troubleshooting
  128. # How does Apache Hamilton compare to X?
  129. Apache Hamilton is not an orchestrator ([you might not need one](https://blog.dagworks.io/p/lean-data-automation-a-principal)), nor a feature store ([but you can use it to build one!](https://blog.dagworks.io/p/featurization-integrating-hamilton)). Its purpose is to help you structure and manage data transformations. If you know dbt, Apache Hamilton does for Python what dbt does for SQL.
  130. Another way to frame it is to think about the different layers of a data stack. Apache Hamilton is at the **asset layer**. It helps you organize data transformations code (the **expression layer**), manage changes, and validate & test data.
  131. <div align="center" style="width: 100%">
  132. <table>
  133. <colgroup>
  134. <col style="width: 20%">
  135. <col style="width: 40%">
  136. <col style="width: 40%">
  137. </colgroup>
  138. <thead>
  139. <tr>
  140. <th>Layer</th>
  141. <th>Purpose</th>
  142. <th>Example Tools</th>
  143. </tr>
  144. </thead>
  145. <tbody>
  146. <tr>
  147. <td>Orchestration</td>
  148. <td>Operational system for the creation of assets</td>
  149. <td>Airflow, Metaflow, Prefect, Dagster</td>
  150. </tr>
  151. <tr>
  152. <td>Asset</td>
  153. <td>Organize expressions into meaningful units <br> (e.g., dataset, ML model, table)</td>
  154. <td><b>Apache Hamilton</b>, dbt, dlt, SQLMesh, <a href="https://github.com/apache/burr">Burr</a></td>
  155. </tr>
  156. <tr>
  157. <td>Expression</td>
  158. <td>Language to write data transformations</td>
  159. <td>pandas, SQL, polars, Ibis, LangChain</td>
  160. </tr>
  161. <tr>
  162. <td>Execution</td>
  163. <td>Perform data transformations</td>
  164. <td>Spark, Snowflake, DuckDB, RAPIDS</td>
  165. </tr>
  166. <tr>
  167. <td>Data</td>
  168. <td>Physical representation of data, inputs and outputs</td>
  169. <td>S3, Postgres, file system, Snowflake</td>
  170. </tr>
  171. </tbody>
  172. </table>
  173. </div>
  174. See our page on [Why use Apache Hamilton?](https://hamilton.apache.org/get-started/why-hamilton/) and framework [code comparisons](https://hamilton.apache.org/code-comparisons/) for more information.
  175. # 📑 License
  176. Apache Hamilton is released under the Apache 2.0 License. See [LICENSE](https://github.com/apache/hamilton/blob/main/LICENSE.md) for details.
  177. # 🌎 Community
  178. ## 👨‍💻 Contributing
  179. We're very supportive of changes by new contributors, big or small! Make sure to discuss potential changes by creating an issue or commenting on an existing one before opening a pull request. Good first contributions include creating an example or an integration with your favorite Python library!
  180. To contribute, checkout our [contributing guidelines](https://github.com/apache/hamilton/blob/main/CONTRIBUTING.md), our [developer setup guide](https://github.com/apache/hamilton/blob/main/developer_setup.md), and our [Code of Conduct](https://www.apache.org/foundation/policies/conduct.html).
  181. ## 😎 Used by
  182. Apache Hamilton was started at Stitch Fix before the original creators founded DAGWorks Inc! The library is battle-tested and has been supporting production use cases since 2019.
  183. >Read more about the [origin story](https://multithreaded.stitchfix.com/blog/2021/10/14/functions-dags-hamilton/).
  184. * [Stitch Fix](https://www.stitchfix.com/) — Time series forecasting
  185. * [UK Government Digital Services](https://github.com/alphagov/govuk-feedback-analysis) — National feedback pipeline (processing & analysis)
  186. * [IBM](https://www.ibm.com/) — Internal search and ML pipelines
  187. * [Opendoor](https://www.opendoor.com/) — Manage PySpark pipelines
  188. * [Lexis Nexis](https://www.lexisnexis.com/en-us/home.page) — Feature processing and lineage
  189. * [Adobe](https://www.adobe.com/) — Prompt engineering research
  190. * [WrenAI](https://github.com/Canner/WrenAI) — async text-to-SQL workflows
  191. * [British Cycling](https://www.britishcycling.org.uk/) — Telemetry analysis
  192. * [Oak Ridge & PNNL](https://pnnl.gov/) — Naturf project
  193. * [ORNL](https://www.ornl.gov/)
  194. * [Federal Reserve Board](https://www.federalreserve.gov/)
  195. * [Joby Aviation](https://www.jobyaviation.com/) — Flight data processing
  196. * [Two](https://www.two.inc/)
  197. * [Transfix](https://transfix.io/) — Online featurization and prediction
  198. * [Railofy](https://www.railofy.com) — Orchestrate pandas code
  199. * [Habitat Energy](https://www.habitat.energy/) — Time-series feature engineering
  200. * [KI-Insurance](https://www.ki-insurance.com/) — Feature engineering
  201. * [Ascena Retail](https://www.ascena.com/) — Feature engineering
  202. * [NaroHQ](https://www.narohq.com/)
  203. * [EquipmentShare](https://www.equipmentshare.com/)
  204. * [Everstream.ai](https://www.everstream.ai/)
  205. * [Flectere](https://flectere.net/)
  206. * [F33.ai](https://f33.ai/)
  207. * [Kora Money](https://www.koramoney.com)
  208. * [Capitec Bank](https://www.capitecbank.co.za/)
  209. * [Best Egg](https://bestegg.com/)
  210. * [RTV Euro AGD](https://www.euro.com.pl/)
  211. * [Wealth.com](https://www.wealth.com/)
  212. * [wren.ai](https://wren.ai/)
  213. ## 🤝 Code Contributors
  214. [![Contributors](https://contrib.rocks/image?repo=apache/hamilton)](https://github.com/apache/hamilton/graphs/contributors)
  215. ## 🙌 Special Mentions & 🦟 Bug Hunters
  216. Thanks to our awesome community and their active involvement in the Apache Hamilton library.
  217. [Nils Olsson](https://github.com/nilsso), [Michał Siedlaczek](https://github.com/elshize), [Alaa Abedrabbo](https://github.com/AAbedrabbo), [Shreya Datar](https://github.com/datarshreya), [Baldo Faieta](https://github.com/baldofaieta), [Anwar Brini](https://github.com/AnwarBrini), [Gourav Kumar](https://github.com/gms101), [Amos Aikman](https://github.com/amosaikman), [Ankush Kundaliya](https://github.com/akundaliya), [David Weselowski](https://github.com/j7zAhU), [Peter Robinson](https://github.com/Peter4137), [Seth Stokes](https://github.com/sT0v), [Louis Maddox](https://github.com/lmmx), [Stephen Bias](https://github.com/s-ducks), [Anup Joseph](https://github.com/AnupJoseph), [Jan Hurst](https://github.com/janhurst), [Flavia Santos](https://github.com/flaviassantos), [Nicolas Huray](https://github.com/nhuray), [Manabu Niseki](https://github.com/ninoseki), [Kyle Pounder](https://github.com/kpounder), [Alex Bustos](https://github.com/bustosalex1), [Andy Day](https://github.com/adayNU), [Alexander Cai](https://github.com/adzcai), [Nils Müller-Wendt](https://github.com/ChronoJon), [Paul Larsen](https://github.com/munichpavel), [
  218. Kemal Eren](https://github.com/kemaleren), [Jernej Frank](https://github.com/jernejfrank), [Noah Ridge](https://github.com/noahridge)
  219. # 🎓 Citations
  220. We'd appreciate citing Apache Hamilton by referencing one of the following:
  221. ```bibtex
  222. @inproceedings{DBLP:conf/vldb/KrawczykI22,
  223. title = {Hamilton: a modular open source declarative paradigm for high level
  224. modeling of dataflows},
  225. author = {Stefan Krawczyk and Elijah ben Izzy},
  226. editor = {Satyanarayana R. Valluri and Mohamed Za{\"{\i}}t},
  227. booktitle = {1st International Workshop on Composable Data Management Systems,
  228. CDMS@VLDB 2022, Sydney, Australia, September 9, 2022},
  229. year = {2022},
  230. url = {https://cdmsworkshop.github.io/2022/Proceedings/ShortPapers/Paper6\_StefanKrawczyk.pdf},
  231. timestamp = {Wed, 19 Oct 2022 16:20:48 +0200},
  232. biburl = {https://dblp.org/rec/conf/vldb/KrawczykI22.bib},
  233. bibsource = {dblp computer science bibliography, https://dblp.org}
  234. }
  235. ```
  236. ```bibtex
  237. @inproceedings{CEURWS:conf/vldb/KrawczykIQ22,
  238. title = {Hamilton: enabling software engineering best practices for data transformations via generalized dataflow graphs},
  239. author = {Stefan Krawczyk and Elijah ben Izzy and Danielle Quinn},
  240. editor = {Cinzia Cappiello and Sandra Geisler and Maria-Esther Vidal},
  241. booktitle = {1st International Workshop on Data Ecosystems co-located with 48th International Conference on Very Large Databases (VLDB 2022)},
  242. pages = {41--50},
  243. url = {https://ceur-ws.org/Vol-3306/paper5.pdf},
  244. year = {2022}
  245. }
  246. ```
  247. # 📚 Libraries built on / for Apache Hamilton
  248. * [Hypster](https://github.com/gilad-rubin/hypster) - hyperparameter management
  249. * [DSP Decision Engine](https://github.com/capitec/dsp-decision-engine) - decision trees
  250. * [NaturF](https://github.com/IMMM-SFA/naturf/) - library for data transformation for weather forecasting
  251. * [WrenAI](https://github.com/Canner/WrenAI) - RAG
  252. * [FlowerPower](https://github.com/legout/flowerpower/) - Scheduler for Apache Hamilton

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.