Skip to content
Back to search

mcp-apache-spark-history-server

MCP Server and CLI for Apache Spark History Server. Debug Spark applications from AI agents, scripts, or the terminal.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/kubeflow/mcp-apache-spark-history-server with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/kubeflow/mcp-apache-spark-history-server'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/kubeflow--mcp-apache-spark-history-server/badge.svg)](/listings/kubeflow--mcp-apache-spark-history-server)

HTML

<a href="/listings/kubeflow--mcp-apache-spark-history-server"><img src="/api/listings/kubeflow--mcp-apache-spark-history-server/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

# Auto-detect latest version, OS, and architecture
VERSION=$(curl -s https://api.github.com/repos/kubeflow/mcp-apache-spark-history-server/releases | grep -m1 '"tag_name": "cli/' | cut -d'"' -f4 | sed 's|cli/||')
OS=$(uname -s | tr '[:upper:]' '[:lower:]')
ARCH=$(uname -m)
[ "$ARCH" = "x86_64" ] && ARCH="amd64"
[ "$ARCH" = "aarch64" ] && ARCH="arm64"

curl -sSL "https://github.com/kubeflow/mcp-apache-spark-history-server/releases/download/cli%2F${VERSION}/shs-${VERSION}-${OS}-${ARCH}.tar.gz" | tar xz
sudo mv shs /usr/local/bin/

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 519 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. # Kubeflow Spark AI Toolkit
  2. [![CI](https://github.com/kubeflow/mcp-apache-spark-history-server/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/kubeflow/mcp-apache-spark-history-server/actions)
  3. [![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)
  4. [![MCP](https://img.shields.io/badge/MCP-Compatible-green.svg)](https://modelcontextprotocol.io/)
  5. [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
  6. [![Kubeflow](https://img.shields.io/badge/Kubeflow-Official-orange.svg)](https://github.com/kubeflow)
  7. [![Slack](https://img.shields.io/badge/Slack-CNCF%20%23mcp--apache--spark--history--server-purple.svg)](https://cloud-native.slack.com/archives/C09FRRM6QM7)
  8. > **Connect AI agents and engineers to Apache Spark History Server for intelligent job analysis, performance monitoring, and investigation**
  9. ---
  10. > [!IMPORTANT]
  11. > ### ✨ NEW — Spark History Server CLI is now available
  12. > [![SHS CLI](https://img.shields.io/badge/NEW-SHS%20CLI%20%60shs%60-brightgreen?style=for-the-badge&logo=go&logoColor=white)](skills/cli/README.md)
  13. >
  14. > A standalone Go binary that queries Spark History Server **directly from your terminal** — no MCP, no AI framework, no daemon process. Inspect jobs, compare runs, investigate failures, and script against the Spark REST API.
  15. >
  16. > **[Get started with the SHS CLI →](skills/cli/README.md)**
  17. ---
  18. This project provides **two interfaces** to your Spark History Server data:
  19. | | 🛠️ [SHS CLI (`shs`)](#️-shs-cli-shs--for-engineers--scripts) | ⚡ [MCP Server](#-mcp-server--for-ai-agents) |
  20. |---|---|---|
  21. | **For** | Engineers, shell scripts, CI/CD, coding agents | AI agents and MCP-compatible clients |
  22. | **Mental model** | "I know the command I want to run" | "Agent, investigate this Spark app" |
  23. | **Install** | Single static binary — no dependencies | Python 3.12+, uv |
  24. | **Get started** | [CLI docs →](skills/cli/README.md) | [MCP docs →](#-mcp-server--for-ai-agents) |
  25. 📺 **See it in action:**
  26. [![Watch the demo video](https://img.shields.io/badge/YouTube-Watch%20Demo-red?style=for-the-badge&logo=youtube)](https://www.youtube.com/watch?v=e3P_2_RiUHw)
  27. ---
  28. ## 🛠️ SHS CLI (`shs`) — For Engineers & Scripts
  29. A standalone Go binary. Query your Spark History Server directly from the terminal, shell scripts, or CI/CD pipelines. Also works as a **skill** for coding agents like Claude Code and Kiro.
  30. ### Install
  31. ```bash
  32. # Auto-detect latest version, OS, and architecture
  33. VERSION=$(curl -s https://api.github.com/repos/kubeflow/mcp-apache-spark-history-server/releases | grep -m1 '"tag_name": "cli/' | cut -d'"' -f4 | sed 's|cli/||')
  34. OS=$(uname -s | tr '[:upper:]' '[:lower:]')
  35. ARCH=$(uname -m)
  36. [ "$ARCH" = "x86_64" ] && ARCH="amd64"
  37. [ "$ARCH" = "aarch64" ] && ARCH="arm64"
  38. curl -sSL "https://github.com/kubeflow/mcp-apache-spark-history-server/releases/download/cli%2F${VERSION}/shs-${VERSION}-${OS}-${ARCH}.tar.gz" | tar xz
  39. sudo mv shs /usr/local/bin/
  40. ```
  41. ### Quick Start
  42. ```bash
  43. # Generate a config file
  44. shs setup config > config.yaml # then set your Spark History Server URL
  45. # Explore applications
  46. shs apps
  47. shs jobs -a APP_ID --status failed
  48. shs stages -a APP_ID --sort duration
  49. shs compare apps --app-a APP1 --app-b APP2
  50. # Use as a skill with Claude Code or Kiro
  51. shs setup skill > ~/.claude/skills/spark-history.md
  52. ```
  53. **[CLI documentation](skills/cli/README.md)** for full usage, or check out a [real-world example](skills/cli/examples/compare/README.md) of Claude Code comparing two **TPC-DS 3TB benchmark** runs.
  54. ---
  55. ## ⚡ MCP Server — For AI Agents
  56. An [MCP (Model Context Protocol)](https://modelcontextprotocol.io/) server that exposes Spark History Server data as tools for AI agents. Agents query your Spark infrastructure using natural language — the server handles tool selection, multi-server routing, and structured data retrieval.
  57. **Use the MCP server when** you want an AI agent to conduct multi-step investigations, synthesize findings across tools, or answer natural-language questions about your Spark applications.
  58. The server supports MCP protocol versions `2026-07-28` and `2025-11-25` (and earlier) at the same time. It negotiates the version with each client, so no configuration is needed.
  59. ### Install
  60. ```bash
  61. # Run directly with uvx (no install needed)
  62. uvx --from mcp-apache-spark-history-server spark-mcp
  63. # Or install with pip
  64. uv tool install mcp-apache-spark-history-server
  65. spark-mcp
  66. ```
  67. The package is published to [PyPI](https://pypi.org/project/mcp-apache-spark-history-server/).
  68. ### Coding Agent Integration
  69. Register the server with a single command. Both examples run it over **stdio** via `uvx`.
  70. With no config file present, the server defaults to a Spark History Server at `http://localhost:18080`;
  71. point it elsewhere with a [config file](#config-file-location) or `SHS_SERVERS__LOCAL__URL`.
  72. **Claude Code** (`claude mcp add`):
  73. ```bash
  74. claude mcp add --env SHS_MCP__TRANSPORT=stdio --env SHS_SERVERS__LOCAL__URL=http://localhost:18080\
  75. --transport stdio spark-history \
  76. -- uvx --from mcp-apache-spark-history-server spark-mcp
  77. ```
  78. **Kiro CLI** (`kiro-cli mcp add`):
  79. ```bash
  80. kiro-cli mcp add --name spark-history --command uvx \
  81. --args --from --args mcp-apache-spark-history-server --args spark-mcp \
  82. --env SHS_MCP__TRANSPORT=stdio --env SHS_SERVERS__LOCAL__URL=http://localhost:18080
  83. ```
  84. Verify in either client with `claude mcp list` / `kiro-cli mcp list`, then ask the agent to *"list the available Spark applications."*
  85. The server also ships **prompts** — guided, multi-step workflows you run as a command. In Claude Code: `/mcp__spark-history__investigate_failure <app_id>`. In Kiro CLI: `/prompts investigate_failure` (or `@investigate_failure`). See [Prompts](#prompts-2) for the full list and arguments.
  86. #### Passing server flags and environment
  87. The commands above have two layers: the **client's** own options and the arguments/environment forwarded to **`spark-mcp`**. `spark-mcp` itself takes a single flag, `--config` / `-c`; everything else is set through `SHS_*` [environment variables](#configure).
  88. | To pass to `spark-mcp`… | Claude Code | Kiro CLI |
  89. |---|---|---|
  90. | A flag (e.g. `--config`) | append after `--`: `… spark-mcp --config /path/config.yaml` | add `--args` pairs: `--args --config --args /path/config.yaml` |
  91. | An environment variable | `--env KEY=value` (before `--transport`) | `--env KEY=value` |
  92. For example, to point at a remote Spark History Server with an explicit config file:
  93. ```bash
  94. # Claude Code
  95. claude mcp add --env SHS_MCP__TRANSPORT=stdio --transport stdio spark-history \
  96. -- uvx --from mcp-apache-spark-history-server spark-mcp --config ~/.config/spark-mcp/config.yaml
  97. # Kiro CLI
  98. kiro-cli mcp add --name spark-history --command uvx \
  99. --args --from --args mcp-apache-spark-history-server --args spark-mcp \
  100. --args --config --args ~/.config/spark-mcp/config.yaml \
  101. --env SHS_MCP__TRANSPORT=stdio
  102. ```
  103. ### Configure
  104. Basic configuration below. Create a file named `config.yaml`:
  105. ```yaml
  106. servers:
  107. local:
  108. default: true
  109. url: "http://your-spark-history-server:18080"
  110. auth: # optional
  111. username: "user"
  112. password: "pass"
  113. include_plan_description: false # include SQL plans by default (default: false)
  114. mcp:
  115. transport: "streamable-http" # or: stdio
  116. port: "18888"
  117. debug: false
  118. ```
  119. #### Config file location
  120. The server looks for its config file in the following order and uses the first one it finds:
  121. 1. The `--config` / `-c` flag (e.g. `spark-mcp --config /path/to/config.yaml`)
  122. 2. The `SHS_MCP_CONFIG` environment variable
  123. 3. `./config.yaml` in the current working directory
  124. 4. `~/.config/spark-mcp/config.yaml` (honors `$XDG_CONFIG_HOME` when set)
  125. If none exist, the server starts with built-in defaults that can be overridden by `SHS_*` environment variables. When a path is given explicitly via the flag or `SHS_MCP_CONFIG` but the file is missing, the server fails fast instead of falling back.
  126. > **Tip for MCP clients:** when the server is launched by an MCP client (Claude Desktop, Kiro, etc.), the working directory is not guaranteed, so a `./config.yaml` may not be found. Prefer `--config` / `SHS_MCP_CONFIG`, or place the file at `~/.config/spark-mcp/config.yaml`.
  127. Configurations can be overridden with environment variables. Nesting levels are
  128. separated by a **double underscore** (`__`), so field names and server names may
  129. themselves contain single underscores (e.g. `SHS_SERVERS__MY_SERVER__URL` maps
  130. to `servers.my_server.url`).
  131. ```
  132. SHS_MCP__PORT Port for MCP server (default: 18888)
  133. SHS_MCP__TRANSPORT Transport mode: streamable-http or stdio
  134. SHS_MCP__DEBUG Enable debug mode (default: false)
  135. SHS_MCP__ADDRESS Bind address (default: localhost)
  136. SHS_SERVERS__*__URL URL for a specific server
  137. SHS_SERVERS__*__AUTH__USERNAME
  138. SHS_SERVERS__*__AUTH__PASSWORD
  139. SHS_SERVERS__*__AUTH__TOKEN
  140. SHS_SERVERS__*__VERIFY_SSL
  141. SHS_SERVERS__*__SSL_CA_CERT Path to a PEM CA bundle for a specific server
  142. SHS_SERVERS__*__TIMEOUT
  143. SHS_SERVERS__*__EMR_CLUSTER_ARN
  144. SHS_SERVERS__*__INCLUDE_PLAN_DESCRIPTION
  145. ```
  146. ### Multi-Server Setup
  147. Configure multiple Spark History Servers and route queries to specific ones:
  148. ```yaml
  149. servers:
  150. production:
  151. default: true
  152. url: "http://prod-spark-history:18080"
  153. auth:
  154. username: "user"
  155. password: "pass"
  156. staging:
  157. url: "http://staging-spark-history:18080"
  158. ```
  159. Agents can target a specific server per query:
  160. > *"Get application `<app_id>` from the production server"*
  161. ### Running Applications (Live Spark UI)
  162. The driver of a running application serves the same REST API (`/api/v1`) as
  163. the Spark History Server, on its Spark UI port (`spark.ui.port`, `4040` by
  164. default). Add the driver UI as another server to query an application while it
  165. runs:
  166. ```yaml
  167. servers:
  168. history:
  169. default: true
  170. url: "http://spark-history-server:18080"
  171. live:
  172. url: "http://spark-driver:4040"
  173. ```
  174. If the Spark UI is reached through a reverse proxy under a path prefix, put the
  175. prefix in `url` (e.g. `https://sparkui.example.com/<namespace>/<pod>`); the
  176. server appends `/api/v1` to it.
  177. How a driver UI differs from a History Server:
  178. - A driver only serves its own application, so `list_applications` on it
  179. returns that single application, and the driver UI goes away when the
  180. application ends. Each server entry is a fixed URL, so to cover every running
  181. application, put a proxy in front that routes each application to its driver
  182. UI while it runs and to the History Server afterwards.
  183. - `get_executor_thread_dump` only works against a driver UI. The History Server
  184. responds with `503` for every application, including running ones.
  185. - When a tool call does not name a server, the first configured server that has
  186. the application is used. With event logging enabled, a running application is
  187. also listed by the History Server, so name the server explicitly:
  188. > *"Get the driver thread dump of `<app_id>` from the live server"*
  189. ### Custom CA bundle
  190. For a private CA or TLS-inspecting proxy, keep `verify_ssl: true` and point
  191. `ssl_ca_cert` to a mounted PEM bundle:
  192. ```yaml
  193. servers:
  194. local:
  195. ssl_ca_cert: "/etc/ssl/custom-ca/ca-bundle.pem"
  196. ```
  197. Set the same path with `SHS_SERVERS__LOCAL__SSL_CA_CERT`. The CA certificates
  198. in the bundle are added to the container's default trust roots, so include the
  199. issuing CA and intermediates.
  200. ## 🏗️ Architecture
  201. ```mermaid
  202. graph TB
  203. subgraph Clients
  204. A[🤖 AI Agent / LLM]
  205. B[👩‍💻 Engineer / Script / CI]
  206. C[🔧 Coding Agent - Claude Code / Kiro]
  207. end
  208. subgraph "Kubeflow Spark AI Toolkit"
  209. D[⚡ MCP Server]
  210. E[🛠️ CLI - shs]
  211. end
  212. subgraph "Spark History Servers"
  213. F[🔥 Production]
  214. G[🔥 Staging / Dev]
  215. end
  216. A -->|MCP Protocol| D
  217. B -->|Terminal commands| E
  218. C -->|shs skill file| E
  219. D -->|REST API| F
  220. D -->|REST API| G
  221. E -->|REST API| F
  222. E -->|REST API| G
  223. ```
  224. ### Connect an AI Agent
  225. | Agent | Transport | Guide |
  226. |-------|-----------|-------|
  227. | **Claude Desktop** | stdio | [Setup →](examples/integrations/claude-desktop/) |
  228. | **Claude Code** | stdio or streamable-http | [Setup →](examples/integrations/claude-desktop/) |
  229. | **Kiro** | stdio or streamable-http | [Setup →](examples/integrations/kiro/) |
  230. | **LangGraph** | streamable-http | [Setup →](examples/integrations/langgraph/) |
  231. | **Strands Agents** | streamable-http | [Setup →](examples/integrations/strands-agents/) |
  232. | **Local / Inspector** | streamable-http | [Setup →](TESTING.md) |
  233. ### Available Tools (19)
  234. <details>
  235. <summary>Available Tools</summary>
  236. #### Application Information
  237. | Tool | Description |
  238. |------|-------------|
  239. | `list_applications` | List applications with optional status, date, and limit filters; pass `app_id` for a single application's detail (status, resources, duration, attempts). Returned applications always include their attempts. |
  240. #### Job Analysis
  241. | Tool | Description |
  242. |------|-------------|
  243. | `list_jobs` | List jobs with status/job-id filtering and sorting (e.g. slowest by duration) |
  244. #### Stage Analysis
  245. | Tool | Description |
  246. |------|-------------|
  247. | `list_stages` | List stages with status filtering and sorting (e.g. slowest by duration) |
  248. | `get_stage` | Stage detail with attempt and task metric distributions |
  249. | `list_stage_task_failures` | Failed tasks of a stage with their error messages (exception/stack trace) |
  250. #### Executor & Resource Analysis
  251. | Tool | Description |
  252. |------|-------------|
  253. | `list_executors` | List executors with executor-id filtering and sorting (failed-tasks/duration/gc/id) |
  254. | `get_executor_summary` | Aggregate metrics across all executors |
  255. | `get_resource_usage_timeline` | Chronological executor add/remove with resource totals |
  256. | `get_executor_thread_dump` | JVM thread dump for a driver/executor, with state/name/blocked filters ([live Spark UI](#running-applications-live-spark-ui) only) |
  257. #### Configuration & Environment
  258. | Tool | Description |
  259. |------|-------------|
  260. | `get_environment` | Spark config, JVM info, system properties, classpath; optional `section` filter to return a single part |
  261. #### SQL & Query Analysis
  262. | Tool | Description |
  263. |------|-------------|
  264. | `list_sql_executions` | List SQL executions as curated summaries, with status/description filters, sorting, and a default limit |
  265. | `get_sql_execution` | SQL execution header by default; opt-in plan, node metrics, job summaries, aggregated stage metrics, and stage list |
  266. | `compare_sql_executions` | Compare aggregated performance metrics (stages, tasks, shuffle, spill, GC) between two SQL executions; opt-in plan-structure diff |
  267. #### Performance & Bottleneck Analysis
  268. | Tool | Description |
  269. |------|-------------|
  270. | `get_job_bottlenecks` | Identify bottlenecks across stages, tasks, and executors |
  271. #### Comparative Analysis
  272. | Tool | Description |
  273. |------|-------------|
  274. | `compare_job_environments` | Diff Spark configs between two applications |
  275. | `compare_job_performance` | Diff performance metrics between two applications |
  276. | `compare_stages` | Compare two stages (optionally across applications): stage metrics and per-task p25/p50/p75/max quantiles |
  277. #### AWS Spark Troubleshooting (opt-in)
  278. | Tool | Description |
  279. |------|-------------|
  280. | `aws_analyze_spark_workload` | One-shot root cause analysis of failed/slow Spark workloads |
  281. | `aws_spark_code_recommendation` | Code fix recommendations for identified Spark issues |
  282. > Automatically available when AWS credentials and region are configured. See [IAM setup guide](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/spark-troubleshooting-agent-iam-setup.html).
  283. </details>
  284. #### Example Agent Queries
  285. - *"Why is my ETL job running slower than yesterday?"* → `get_job_bottlenecks` + `list_stages` + `compare_job_performance`
  286. - *"What caused job 42 to fail?"* → `list_jobs` + `get_stage`
  287. - *"Compare today's batch with yesterday's run"* → `compare_job_performance` + `compare_job_environments`
  288. - *"Find my slowest SQL queries and explain why"* → `list_sql_executions` + `get_sql_execution` + `compare_sql_executions`
  289. ### Prompts (2)
  290. Beyond tools, the server exposes **MCP prompts** — reusable, multi-step investigation workflows that your agent runs as a single command. Each prompt expands into a guided sequence of tool calls (which the agent still executes one at a time), so you get a consistent, evidence-driven analysis without having to remember the right tool order.
  291. | Prompt | Arguments | Description |
  292. |--------|-----------|-------------|
  293. | `investigate_failure` | `app_id` (required), `server` (optional) | Guided root-cause investigation for a failed Spark application — walks from the failed app down to the individual task exceptions. |
  294. | `compare_applications` | `app_a` (required), `app_b` (required), `server` (optional), `context` (optional) | Layered, descriptive comparison of two applications (configuration → app metrics → SQL/jobs → stages). |
  295. > `server` is optional: when omitted, the prompt searches every configured server for the application(s). Pass `server="<name>"` only to target a specific server or disambiguate an id that exists on more than one.
  296. #### Using prompts in Claude Code
  297. MCP prompts surface as slash commands with the format `/mcp__<server>__<prompt>` (note the **double underscores**), where `<server>` is the name you registered the server under (`spark-history` in the [examples above](#coding-agent-integration)). Type `/` to discover them, then pass arguments space-separated after the command:
  298. ```text
  299. # Investigate a failed application
  300. /mcp__spark-history__investigate_failure spark-cc4d615a5e6b4500b8eb1e9deb48cb4e
  301. # Target a specific configured server
  302. /mcp__spark-history__investigate_failure spark-cc4d615a5e6b4500b8eb1e9deb48cb4e production
  303. # Compare two applications
  304. /mcp__spark-history__compare_applications spark-app-A spark-app-B
  305. ```
  306. #### Using prompts in Kiro CLI
  307. List and run prompts with the `/prompts` command, or type `@` then Tab to autocomplete. Run a prompt by name and supply its arguments:
  308. ```text
  309. # Open the prompt picker (shows prompts from all MCP servers)
  310. /prompts
  311. # Run a prompt directly
  312. /prompts investigate_failure
  313. # Or use the @ shortcut
  314. @investigate_failure
  315. ```
  316. Kiro CLI prompts you for arguments interactively when the prompt declares them, so you can run `/prompts investigate_failure` and provide `app_id` (and optionally `server`) when asked.
  317. ---
  318. ## 📸 Screenshots
  319. ### 🔍 Get Spark Application
  320. ![Get Application](screenshots/get-application.png)
  321. ### ⚡ Job Performance Comparison
  322. ![Job Comparison](screenshots/job-compare.png)
  323. ---
  324. ## 🚀 Kubernetes Deployment
  325. Deploy the MCP server using Helm:
  326. ```bash
  327. helm install spark-history-mcp ./deploy/kubernetes/helm/mcp-apache-spark-history-server/
  328. # Production configuration
  329. helm install spark-history-mcp ./deploy/kubernetes/helm/mcp-apache-spark-history-server/ \
  330. --set replicaCount=3 \
  331. --set autoscaling.enabled=true
  332. ```
  333. See [`deploy/kubernetes/helm/`](deploy/kubernetes/helm/) for full configuration options.
  334. When deployed in Kubernetes, connect Claude Desktop via `mcp-remote`:
  335. ```bash
  336. kubectl port-forward svc/spark-history-mcp 18888:18888
  337. ```
  338. ---
  339. ## 📔 AWS Integration
  340. - **[AWS Glue](examples/aws/glue/README.md)** — Connect to Glue Spark History Server
  341. - **[Amazon EMR](examples/aws/emr/README.md)** — Use EMR Persistent UI for Spark analysis
  342. - **AWS Spark Troubleshooting** — One-shot root cause analysis and code fix recommendations for failed Spark workloads (EMR EC2, EMR Serverless). Automatically available when AWS credentials and region are configured. See [IAM setup guide](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/spark-troubleshooting-agent-iam-setup.html) for required permissions.
  343. ---
  344. ## 🔧 Development Setup
  345. ```bash
  346. git clone https://github.com/kubeflow/mcp-apache-spark-history-server.git
  347. cd mcp-apache-spark-history-server
  348. # Install Task runner
  349. brew install go-task # macOS; see https://taskfile.dev/installation/ for others
  350. # MCP Server
  351. task install # install Python dependencies
  352. task start-spark-bg # start Spark History Server with sample data
  353. task start-mcp-bg # start MCP server
  354. task start-inspector-bg # open MCP Inspector at http://localhost:6274
  355. task stop-all
  356. # CLI
  357. cd skills/cli
  358. task build # build ./bin/shs
  359. task test # unit tests
  360. task test-e2e # e2e tests (starts/stops Docker SHS automatically)
  361. task start-shs # start SHS with CLI e2e sample data
  362. ```
  363. ---
  364. ## 🌍 Adopters
  365. Using this project? Add your organization to [ADOPTERS.md](ADOPTERS.md) and help grow the community.
  366. ## 🤝 Contributing
  367. See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
  368. ## 📄 License
  369. Apache License 2.0 — see [LICENSE](LICENSE).
  370. ## 📝 Trademark Notice
  371. *Built for use with Apache Spark™ History Server. Not affiliated with or endorsed by the Apache Software Foundation.*
  372. ---
  373. <div align="center">
  374. **Connect your Spark infrastructure to AI agents and engineers**
  375. [🛠️ SHS CLI](skills/cli/README.md) · [⚡ MCP Server](#-mcp-server--for-ai-agents) · [🧪 Test](TESTING.md) · [🤝 Contribute](#-contributing)
  376. *Built by the community, for the community* 💙
  377. </div>

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.