Skip to content
Back to search

alchemylake-databricks

Governed, provenance-sealed creative rendering for Databricks — a Streamlit App + Unity Catalog ai_render() function that turns any governed source (Sample Lakehouse, UC, Genie, or BYO data) into infographics, reports, decks, video briefings, music, and podcasts.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/zorost--alchemylake-databricks/badge.svg)](/listings/zorost--alchemylake-databricks)

HTML

<a href="/listings/zorost--alchemylake-databricks"><img src="/api/listings/zorost--alchemylake-databricks/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 384 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. <p align="center">
  2. <img src=".github/sol-invictus-lapis.png" alt="AlchemyLake — Sol Invictus engraving on the lapis plate" width="640">
  3. </p>
  4. # AlchemyLake for Databricks
  5. [![License](https://img.shields.io/badge/license-Apache--2.0-blue)](./LICENSE)
  6. [![Databricks CLI](https://img.shields.io/badge/Databricks%20CLI-%E2%89%A5%200.230-FF3621)](https://docs.databricks.com/dev-tools/cli/)
  7. [![MCP](https://img.shields.io/badge/MCP-13%20tools-172478)](https://app.alchemylake.com/docs#developers)
  8. Install a **governed creative** surface directly inside your Databricks
  9. workspace. AlchemyLake turns the data your lakehouse already trusts into
  10. finished deliverables **Genie doesn't make**: board-ready **PowerPoint decks
  11. with a read-aloud script and Q&A prep under every slide**, enterprise **PDF
  12. dossiers with a statistical appendix and an Excel evidence workbook**,
  13. **deep-research dossiers** (a planned multi-step investigation — Genie sources
  14. get governed SQL follow-ups in one continuing conversation), designed
  15. **infographics**, animated **video briefings with narration** (six formats),
  16. two-host **data podcasts** (five formats), and **sonified scores** (six
  17. genres) whose tempo follows your growth.
  18. ## Get started free — no card required
  19. 1. **[Sign up](https://app.alchemylake.com/sign-up)** — 50 free credits land in
  20. your account immediately, nothing to enter but an email.
  21. 2. **Grab a developer key** — sign in at [app.alchemylake.com](https://app.alchemylake.com) →
  22. **Studio → Developer · MCP & keys → Forge a new key** (starts with `alk_…`).
  23. 3. **Clone this repo and deploy it** into your Databricks workspace:
  24. ```bash
  25. git clone https://github.com/zorost/alchemylake-databricks.git
  26. cd alchemylake-databricks
  27. # edit the workspace host in databricks.yml (see Path 1 below), then:
  28. databricks bundle deploy -t prod
  29. databricks bundle run alchemylake_app -t prod
  30. ```
  31. Tracking `main` gets you the latest build; `git checkout v0.4.0` first pins
  32. the deploy to a tagged release instead, if you'd rather not move with `main`.
  33. 4. Open the App URL the CLI prints, paste your key, then **Load governed
  34. sources → pick one → Transmute.**
  35. That's a governed PDF, deck, infographic, or video briefing — sealed to your
  36. own data, running inside your own workspace — for free, in about five minutes.
  37. No Databricks Apps on your workspace? Skip straight to
  38. [Path 2](#path-2--register-alchemylake-as-an-mcp-server-genie--agent-bricks) —
  39. register one URL and Genie, Claude, or Cursor get the same 50 free credits to
  40. work with, no bundle required.
  41. **Every lane binds to a governed source**, and the numbers discipline holds
  42. throughout: the platform computes an analyst-grade statistical dossier from the
  43. bound rows by code — trend with fit quality, outliers, correlations, segment
  44. shares, concentration, pivots — the model never authors a figure, a **verifier
  45. checks every claim after generation**, and the result is sealed (source · row
  46. count · data sha256 · verification score) with the seal embedded in the file
  47. itself. Genie answers bind as sources too, and one-click **recipes** ship a
  48. board deck or a whole campaign pack from one table. Every credit accounted.
  49. **Where Genie stops, AlchemyLake starts.** Genie answers questions about your
  50. data inside the workspace — tables and charts. AlchemyLake takes the same
  51. governed rows the last mile: *"Ask Genie for Q2 revenue by region, then render
  52. an 8-slide deck titled Q2 Momentum"* leaves the workspace as a .pptx the CFO
  53. can present cold — every figure verified against the rows, provenance sealed
  54. into the file.
  55. This repository **is** a self-contained **Databricks Asset Bundle**. One command
  56. deploys a Databricks App (SSO-authenticated, running on your workspace) that
  57. calls the AlchemyLake platform over MCP. Nothing bypasses governance: the same
  58. credit ledger, provenance seals, and role checks that protect the web app
  59. protect every call made from inside Databricks. Your data never leaves the
  60. lakehouse except as the exact rows you choose to bind.
  61. ```
  62. .
  63. ├── databricks.yml # the Asset Bundle (App resource + targets)
  64. ├── app/
  65. │ ├── app.py # the Streamlit App (thin MCP client)
  66. │ ├── app.yaml # App runtime config (command + env)
  67. │ └── requirements.txt
  68. └── sql/
  69. └── ai_render.sql # optional UC function: call AlchemyLake from SQL/Genie
  70. ```
  71. ---
  72. ## Three ways to use AlchemyLake with Databricks
  73. | Path | What you get | Setup |
  74. |---|---|---|
  75. | **1. The App (this bundle)** | A governed render UI inside your workspace, SSO’d, next to your data | `databricks bundle deploy` |
  76. | **2. MCP for Genie / Agent Bricks** | Every agent gains **13 governed tools** (`list_governed_sources`, `upload_source`, `get_wallet`, `render_governed_chat`, `render_deep_research`, `render_infographic`, `render_report`, `render_presentation`, `render_video_briefing`, `render_music`, `render_podcast`, `list_recipes`, `run_recipe`) — all render tools accept `source_id` for data-bound, verified output, and chat returns a `thread_id` so a Genie conversation continues across agent turns | Register one URL |
  77. | **3. `ai_render()` in SQL** | Sealed narrative from a query or Genie space | Run `sql/ai_render.sql` |
  78. **What agents can ship from a table** (things a Genie answer alone cannot):
  79. - `render_deep_research` — a planned multi-step investigation: the brief is
  80. decomposed into sub-questions, each answered with real evidence (Genie
  81. sources get governed SQL follow-ups in one continuing conversation), then
  82. synthesized into a sealed PDF dossier + Excel evidence workbook.
  83. - `render_presentation` — a .pptx board deck, 5–20 slides, speaker script + Q&A
  84. in every notes pane, AI cover art, real charts from the rows.
  85. - `render_report` — an enterprise PDF dossier (KPI band, chart sections,
  86. statistical appendix, citations, methodology) **plus** an Excel evidence
  87. workbook: raw rows, facts, statistics, pivot, correlations.
  88. - `render_infographic` — a designed KPI poster with the exact figures rendered
  89. in-image and a branded provenance strip.
  90. - `render_video_briefing` — an animated video briefing: platform-drawn charts,
  91. spoken narration, motion and crossfades. Six formats (`style` param):
  92. consultant walkthrough, newsroom segment, executive stand-up, documentary
  93. deep-dive, field report, social recap.
  94. - `render_podcast` — a two-host audio briefing with a sealed transcript. Five
  95. formats: two-host interview, skeptic's debate, executive stand-up, narrative
  96. deep-dive, plain-language walkthrough.
  97. - `render_music` — a sonified score (tempo ↔ momentum, mode ↔ trend) plus the
  98. literal data-motif WAV of the rows. Six genres: cinematic score, corporate
  99. uplift, ambient data fields, electronic pulse, orchestral arc, lo-fi data
  100. study.
  101. Prefer plain REST? The same keys work on the public API:
  102. `https://app.alchemylake.com/api/public/v1` (OpenAPI at `/api/public/v1/openapi.json`).
  103. Prefer a terminal? `npx alchemylake` is a zero-dependency CLI over that API:
  104. ```bash
  105. export ALCHEMYLAKE_API_KEY=alk_YOUR_KEY
  106. npx alchemylake upload ./q3-actuals.xlsx # CSV/Excel/PDF/Word/text → governed source
  107. npx alchemylake render report --prompt "Board brief" --source up.1a2b3c
  108. npx alchemylake render deep_research --prompt "Why did Q3 dip?" --source genie:dbx1
  109. ```
  110. ---
  111. ## How it fits together
  112. AlchemyLake never runs *inside* this bundle — the App is a thin, SSO'd client that
  113. calls the governed platform over HTTPS/MCP. Your tables stay in your lakehouse; only
  114. the exact rows you bind for a given render ever leave the workspace, and only to
  115. produce that one sealed deliverable.
  116. ```mermaid
  117. flowchart LR
  118. subgraph WS["Your Databricks workspace"]
  119. UC[("Unity Catalog<br/>tables & volumes")]
  120. Genie["Genie space"]
  121. CSV["Uploaded CSV / Excel"]
  122. App["AlchemyLake App<br/>(this bundle · Streamlit · SSO'd)"]
  123. Agents["Genie / Agent Bricks<br/>Claude · Cursor"]
  124. end
  125. AL(["AlchemyLake<br/>governed render platform"])
  126. UC -- "bind rows" --> App
  127. Genie -- "bind an answer<br/>(conversation continues)" --> App
  128. CSV -- "bind (BYO-data)" --> App
  129. App <-- "MCP: render, verify, seal" --> AL
  130. Agents <-- "13 MCP tools, same contract" --> AL
  131. AL -. "no-egress text tier" .-> FM["Your Databricks<br/>Foundation Model endpoint"]
  132. AL --> Out(["Sealed deliverable<br/>PDF · PPTX · image · video · audio"])
  133. ```
  134. Every render — from the App, from an agent, or from `ai_render()` in SQL — goes
  135. through the same discipline before it comes back:
  136. ```mermaid
  137. flowchart LR
  138. A["Bind<br/><sub>pick a governed source</sub>"] --> B["Analyze<br/><sub>compute the facts</sub>"]
  139. B --> C["Design<br/><sub>plan the deliverable</sub>"]
  140. C --> D["Transmute<br/><sub>generate</sub>"]
  141. D --> E["Verify<br/><sub>check every claim</sub>"]
  142. E --> F["Seal<br/><sub>embed provenance</sub>"]
  143. ```
  144. The model is never the source of a number — every figure is computed from your rows
  145. first, and whatever the render says is checked against those figures afterward. The
  146. seal (source · row count · data hash · verification score) rides embedded in the file
  147. itself, in every format, so it survives being forwarded, downloaded, or printed.
  148. ### Beyond a Genie answer
  149. | | Genie alone | Genie / Agent Bricks + AlchemyLake |
  150. |---|:---:|:---:|
  151. | Answers a question inside the workspace | ✅ | ✅ |
  152. | Ships a board-ready **.pptx** with a read-aloud script + Q&A prep per slide | | ✅ |
  153. | Ships an enterprise **PDF dossier + Excel evidence workbook** | | ✅ |
  154. | Runs a **deep-research investigation** (planned Genie SQL follow-ups → sealed dossier) | | ✅ |
  155. | Ships a designed **infographic** with figures rendered in-image | | ✅ |
  156. | Ships a narrated **video briefing** or two-host **podcast** | | ✅ |
  157. | Every number checked against the source *after* generation and scored | | ✅ |
  158. | Provenance (source, rows, hash, score — plus Genie reasoning, SQL, and trusted-asset status) embedded in the output | | ✅ |
  159. | Scores your Genie space's curation health and certifies it with live probes | | ✅ |
  160. | Metered by an auditable, per-render credit ledger | | ✅ |
  161. ### What people actually ask it for
  162. Every lane binds to a governed source, so the figures in the output are the figures
  163. in your table — not a paraphrase. A sample of real workflows across the eight lanes:
  164. | Lane | Ships | Example asks |
  165. |---|---|---|
  166. | **Analyst** (chat) | A sealed, sourced answer — follow-ups stay in context | *"Draft the board narrative from the certified loan book."* · *"Now break that down by region."* |
  167. | **Deep Research** | A sealed investigation dossier (PDF + Excel) | *"Why did Q3 dip, and what should we do about it?"* · *"Investigate concentration risk across our segments."* |
  168. | **Infographic** | A KPI poster, figures rendered in-image | *"This week's sales milestone, ready for social."* · *"'45 days without a recordable incident' for the plant floor."* |
  169. | **Report** | PDF dossier + Excel evidence workbook | *"Quarterly board-packet narrative, sealed."* · *"Compliance rollup with a hash of the source data baked in."* |
  170. | **Presentation** | .pptx, speaker script + Q&A per slide | *"QBR deck with the CRM's exact numbers."* · *"Steering-committee deck anyone on the team can present cold."* |
  171. | **Video briefing** | A narrated, animated data video | *"All-hands opener with the real ARR on screen."* · *"30-second earnings-day teaser, safe to post at the bell."* |
  172. | **Music** | A score that follows the actual trend | *"A launch-video bed whose tempo rises with the real growth curve."* |
  173. | **Podcast** | A two-host audio briefing + transcript | *"A 3-minute briefing leadership can play on the commute."* · *"An audio version of the same sealed report, for accessibility."* |
  174. Free to try: [sign up](https://app.alchemylake.com/sign-up) for 50 credits, no card
  175. required, and point this bundle at your own key.
  176. ---
  177. ## Prerequisites
  178. 1. **Databricks CLI ≥ 0.230** and authentication to your workspace:
  179. ```bash
  180. databricks auth login --host https://<your-workspace>.cloud.databricks.com
  181. ```
  182. 2. **Databricks Apps enabled** on the workspace (Premium/Enterprise).
  183. 3. An **AlchemyLake developer key** (`alk_…`): [sign up free](https://app.alchemylake.com/sign-up)
  184. (50 credits, no card required) or sign in at <https://app.alchemylake.com> →
  185. **Studio → Developer · MCP & keys → Forge a new key**.
  186. ---
  187. ## Path 1 — Deploy the App (one command)
  188. First set your workspace host in `databricks.yml` — replace the
  189. `your-workspace…` placeholder under **both** `targets.dev` and `targets.prod`
  190. (run `databricks auth profiles` if you need to find it). Then:
  191. ```bash
  192. databricks bundle deploy -t prod # upload source + create the App
  193. databricks bundle run alchemylake_app -t prod # start it; prints the App URL
  194. ```
  195. > **CLI note (Terraform pin).** Some Databricks CLI builds download a Terraform
  196. > whose signing key has expired, which breaks `bundle deploy`. If you hit that,
  197. > point the CLI at a local Terraform binary and pin the version to match it:
  198. > ```bash
  199. > export DATABRICKS_TF_EXEC_PATH="$(command -v terraform)"
  200. > export DATABRICKS_TF_VERSION="1.9.8" # set to your local `terraform version`
  201. > ```
  202. > This is the exact, verified path AlchemyLake's own reference App is deployed
  203. > with — it runs live in-workspace as a Streamlit App under SSO.
  204. Open the App URL that the CLI prints. Paste your `alk_…` key in the sidebar
  205. (or bind it from a secret — see below), then **Load governed sources → pick one
  206. → Transmute**. Every result carries its provenance seal.
  207. The pasted key is remembered in that browser (client-side only, until cleared
  208. or replaced) so it survives a refresh — a workspace-bound secret still always
  209. takes priority. The App also carries a self-contained **Docs** tab and an
  210. expanded **About / Install** tab (how to use it, why the verification model
  211. matters, where the render compute actually runs, and the fastest path to a
  212. key) so it's useful even where the public docs site isn't reachable.
  213. ### Bind the key from a secret (recommended for teams)
  214. So analysts never paste a key:
  215. ```bash
  216. databricks secrets create-scope alchemylake
  217. databricks secrets put-secret alchemylake api_key # paste alk_...
  218. ```
  219. Then uncomment the `ALCHEMYLAKE_API_KEY` / `valueFrom` block in
  220. `app/app.yaml` and `databricks.yml`, and redeploy.
  221. ---
  222. ## Path 2 — Register AlchemyLake as an MCP server (Genie / Agent Bricks)
  223. No bundle required. In your agent/MCP client configuration add:
  224. ```json
  225. {
  226. "mcpServers": {
  227. "alchemylake": {
  228. "url": "https://app.alchemylake.com/api/mcp",
  229. "headers": { "Authorization": "Bearer alk_YOUR_KEY" }
  230. }
  231. }
  232. }
  233. ```
  234. Genie / Agent Bricks, Claude, and Cursor will discover the thirteen tools
  235. automatically. Ask, for example: *“Using AlchemyLake, write a sealed 3-bullet
  236. executive summary of `gold_ridership_national_monthly`”* — or go bigger:
  237. *“Render an 8-slide presentation of that table titled Ridership Momentum”*
  238. (a .pptx with a presenter script under every slide), *“Run deep research on
  239. why ridership dipped this quarter”* (`render_deep_research` returns a sealed
  240. dossier + evidence workbook), or *“Run the campaign-pack recipe”*
  241. (`run_recipe` renders the poster, the report, and the copy in one call, all
  242. verified and sealed).
  243. ---
  244. ## Path 3 — `ai_render()` in SQL / Genie
  245. Edit `sql/ai_render.sql` (set your catalog/schema) and run it in a SQL editor
  246. on **serverless compute with external network access enabled**. Then:
  247. ```sql
  248. SELECT ai_render(
  249. 'One-sentence executive read of this quarter''s ridership',
  250. 'ntd_demo.ntd.gold_ridership_national_monthly'
  251. ) AS narrative;
  252. ```
  253. See the header of `sql/ai_render.sql` for the network-egress and secret
  254. prerequisites, and the MCP fallback if egress cannot be enabled.
  255. ---
  256. ## Validate before deploying
  257. ```bash
  258. databricks bundle validate -t prod
  259. ```
  260. ---
  261. ## FAQ
  262. **Does my data leave the lakehouse?** Only the exact rows you bind for a given
  263. render are sent to produce that render — nothing else is read, and bound data is
  264. never used to train models. A no-egress **text tier** can route even that narrative
  265. step to your own Databricks Foundation Model endpoint instead; see
  266. [the residency docs](https://app.alchemylake.com/docs#residency).
  267. **Do I need Databricks Apps enabled for this?** Only for Path 1 (the App). Path 2
  268. (MCP) and Path 3 (`ai_render()`) need nothing beyond CLI/SQL access and a developer
  269. key — no App, no extra compute resource.
  270. **Can agents use this without a human in the loop?** Yes — that's Path 2. Register
  271. `/api/mcp` once and Genie, Agent Bricks, Claude, and Cursor all gain the same thirteen
  272. governed tools, metered and sealed exactly like the UI.
  273. **What does a Genie question cost?** Genie compute bills inside *your* Databricks
  274. workspace (serverless DBUs — roughly 1.5 DBU ≈ $0.11 per question; Databricks
  275. gives every user 150 free Genie DBUs a month). AlchemyLake credits meter only the
  276. render itself, and the Studio shows the DBU estimate next to any Genie source.
  277. **I don't have Databricks — can I still use everything?** Yes. Upload CSV, Excel,
  278. PDF, Word, text or Markdown (in the web Studio, this App's Sources tab, the API,
  279. the CLI, or the `upload_source` MCP tool) and every lane — including Deep
  280. Research — runs with the same verification and seals.
  281. **Is AlchemyLake a Databricks product?** No. AlchemyLake is an independent platform
  282. by [Zorost Intelligence](mailto:info@zorost.com) that plugs into Databricks; this
  283. repository is Apache-2.0 licensed and is how it plugs in. The hosted platform it
  284. talks to is a separate, metered service (50 free credits to start, no card
  285. required).
  286. **What does this bundle actually contain?** Only a Streamlit client (`app/app.py`),
  287. its App runtime config, and an optional SQL function — everything under
  288. [`sql/`](./sql) and [`app/`](./app). It holds no model weights, no proprietary
  289. rendering code, and no secrets; it is a thin, auditable client over the same MCP/REST
  290. contract documented at [app.alchemylake.com/docs](https://app.alchemylake.com/docs).
  291. **Something's broken.** Open an issue in this repository, or email
  292. info@zorost.com — see [`SECURITY.md`](./SECURITY.md) for vulnerability reports
  293. specifically.
  294. ---
  295. ## Documentation
  296. - [`docs/DEVELOPER_GUIDE.md`](./docs/DEVELOPER_GUIDE.md) — **CLI, REST API & MCP**: get a key, every command, VS Code/Cursor/Claude setup, use cases, Python/Node examples, GitHub Actions.
  297. - [`docs/GENIE_GUIDE.md`](./docs/GENIE_GUIDE.md) — Genie integration: connect a space, health score, certification, all eight lanes, sharing model.
  298. - Platform docs: [app.alchemylake.com/docs](https://app.alchemylake.com/docs) (Studio walkthrough, credits, security).
  299. ---
  300. ## Uninstall
  301. ```bash
  302. databricks bundle destroy -t prod
  303. ```
  304. ---
  305. *AlchemyLake is a product of **Zorost Intelligence**. Platform docs:
  306. <https://app.alchemylake.com/docs>. Support: info@zorost.com.*

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.