Skip to content
Back to search

CrowdSentinels-AI-MCP

AI-powered threat hunting and incident response MCP server for Elasticsearch/OpenSearch

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/thomasxm--crowdsentinels-ai-mcp/badge.svg)](/listings/thomasxm--crowdsentinels-ai-mcp)

HTML

<a href="/listings/thomasxm--crowdsentinels-ai-mcp"><img src="/api/listings/thomasxm--crowdsentinels-ai-mcp/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Install from PyPI (recommended)

# Install with pip
pip install crowdsentinel-mcp-server

# Or install with uv
uv pip install crowdsentinel-mcp-server

# Download detection rules, Chainsaw, and Sigma rules (one-time)
crowdsentinel setup

# Validate connector configuration and readiness (offline, no external calls)
crowdsentinel doctor

**Optional extras:**

# Velociraptor live endpoint forensics (adds 25 MCP tools)
pip install 'crowdsentinel-mcp-server[velociraptor]'

# Long-tail deobfuscation coverage via chepy (XOR, exotic encodings).
# The stdlib decoder core works without this — chepy only extends it.
pip install 'crowdsentinel-mcp-server[deobf]'

Detection rules (6,060 Lucene + EQL + ES|QL) are **bundled with the package** — no download needed. The `setup` command downloads additional tools: - Chainsaw binary for EVTX analysis - 3,000+ Sigma rules for Chainsaw

Downloaded tools are stored in `~/.crowdsentinel/` and persist across package upgrades.

**System dependency for PCAP analysis:**

# Required for network traffic analysis and cross-tool IoC correlation
sudo apt install tshark    # Debian/Ubuntu/Kali
sudo dnf install wireshark-cli  # Fedora/RHEL
brew install wireshark     # macOS

Run directly with uvx (no install needed)

# Elasticsearch 8.x (default)
uvx crowdsentinel-mcp-server

# Other backends
uvx crowdsentinel-mcp-server-es7   # Elasticsearch 7.x
uvx crowdsentinel-mcp-server-es9   # Elasticsearch 9.x
uvx opensearch-mcp-server          # OpenSearch 1.x/2.x/3.x

Install from source

git clone https://github.com/thomasxm/CrowdSentinels-AI-MCP.git
cd CrowdSentinels-AI-MCP
chmod +x setup.sh && ./setup.sh

# Or install everything, including all optional extras (deobf/chepy, velociraptor)
./setup.sh --full

The setup script will: - Install dependencies (pipx, uv, Claude Code CLI if needed) - Bundle 6,060 detection rules and download Chainsaw binary - Prompt for Elasticsearch credentials (never hardcoded) - Configure the MCP server with Claude Code - Validate your connection

With `--full`, every optional extra is installed too, so all 139 MCP tools have their dependencies available — the recommended mode for a production-ready server.

Installed Size

CrowdSentinel bundles 6,060 detection rules and integrates with external analysis tools. Below is the full disk space breakdown so you can plan accordingly.

**Core package (via `pip` or `uvx`):**

| Component | Size | Notes | |---|---|---| | **CrowdSentinel package** | **49 MB** | The server itself | | — Bundled Sigma rules (`src/rules/`) | 30 MB | 6,060 pre-converted detection rules | | — Elastic TOML rules (`src/detection-rules/`) | 17 MB | Original TOML format rules + hunting queries | | — Python code (clients, tools, etc.) | 2 MB | Actual application code | | **Dependencies** | **64 MB** | All transitive deps | | — `cryptography` | 14 MB | Largest dependency (TLS) | | — `elasticsearch` | 8.3 MB | ES Python client | | — `pygments` | 5.2 MB | Syntax highlighting | | — `pydantic_core` | 5 MB | Validation engine | | — `opensearchpy` | 3.6 MB | OpenSearch client | | — Others (27 packages) | ~28 MB | mcp, fastmcp, httpx, anthropic, etc. | | **Core total** | **113 MB** | `pip install crowdsentinel-mcp-server` |

**Additional tools (via `crowdsentinel setup`):**

| Component | Download | Installed | Notes | |---|---|---|---| | Chainsaw binary (v2.13.1) | ~3 MB | ~15 MB | EVTX log analysis engine | | Sigma rules (SigmaHQ) | ~3 MB | ~30 MB | 3,000+ Sigma rules for Chainsaw | | Chainsaw mappings | — | <1 MB | Event log source mappings | | **Setup total** | **~6 MB** | **~46 MB** | Stored in `~/.crowdsentinel/` |

**System dependency (via package manager):**

| Component | Installed | Install Command | Notes | |---|---|---|---| | tshark + Wireshark libs | ~132 MB | `sudo apt install tshark` | PCAP network analysis — required for cross-tool IoC correlation |

**Full installation summary:**

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 1,322 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. <!-- mcp-name: io.github.thomasxm/crowdsentinel-mcp-server -->
  2. <div align="center">
  3. <img src="https://raw.githubusercontent.com/thomasxm/crowdsentinel-assets/main/logo.png" alt="CrowdSentinel Logo" width="400"/>
  4. # CrowdSentinel MCP Server
  5. ### AI-Powered Threat Hunting & Incident Response Framework
  6. [![PyPI](https://img.shields.io/pypi/v/crowdsentinel-mcp-server.svg)](https://pypi.org/project/crowdsentinel-mcp-server/)
  7. [![PyPI Downloads](https://img.shields.io/pypi/dm/crowdsentinel-mcp-server.svg)](https://pypi.org/project/crowdsentinel-mcp-server/)
  8. [![Python](https://img.shields.io/pypi/pyversions/crowdsentinel-mcp-server?cacheSeconds=3600)](https://pypi.org/project/crowdsentinel-mcp-server/)
  9. [![License](https://img.shields.io/badge/License-GPL%20v3-blue.svg)](LICENSE)
  10. [![MCP](https://img.shields.io/badge/MCP-Compatible-purple.svg)](https://modelcontextprotocol.io/)
  11. [![Tools](https://img.shields.io/badge/MCP%20Tools-139-brightgreen.svg)](https://github.com/thomasxm/CrowdSentinels-AI-MCP)
  12. [![Rules](https://img.shields.io/badge/Detection%20Rules-6%2C060-blue.svg)](https://github.com/thomasxm/CrowdSentinels-AI-MCP)
  13. [![Install from PyPI](https://img.shields.io/badge/Install-pip%20install%20crowdsentinel--mcp--server-3775A9?style=for-the-badge&logo=pypi&logoColor=white)](https://pypi.org/project/crowdsentinel-mcp-server/)
  14. [MCP Official Registry](https://registry.modelcontextprotocol.io/v0/servers?search=io.github.thomasxm/crowdsentinel-mcp-server) · [PyPI Package](https://pypi.org/project/crowdsentinel-mcp-server/)
  15. **Open-source threat hunting orchestrator connecting LLMs to enterprise security data via Model Context Protocol (MCP)**
  16. [Quick Start](#quick-start) · [Installation](#installation) · [CLI Usage](#cli-usage) · [Features](#key-features) · [Architecture](#architecture) · [Documentation](#documentation) · [Examples](#usage-examples)
  17. </div>
  18. > **Warning**
  19. > This project is in active development and intended for **security testing, research, and educational purposes only**. It is not production-ready. Do not deploy in production environments. APIs, tool interfaces, and data formats may change without notice. Use at your own risk.
  20. ---
  21. ## Demo
  22. https://github.com/user-attachments/assets/0d0381f0-5b68-43b2-8630-19ec130885b2
  23. ---
  24. ## What is CrowdSentinel?
  25. **CrowdSentinel** transforms traditional SIEM querying into intelligent, framework-driven investigations using natural language. It serves as a unified security intelligence layer that connects large language models to enterprise security data sources, enabling:
  26. - **Natural Language Threat Hunting** — Query Elasticsearch using plain English
  27. - **AI-Guided Investigation Workflows** — Built-in prompts guide agents through proper IR methodology
  28. - **Persistent Investigation State** — Memory-managed IoC tracking, forensic timelines, and cross-query correlation that survives across sessions (8GB FIFO storage)
  29. - **Cross-Tool IoC Correlation** — IoCs discovered in one tool are automatically available to all others
  30. - **Multi-Source Analysis** — Elasticsearch, EVTX logs (Chainsaw), PCAP files (Wireshark), live endpoint forensics (Velociraptor), local host forensics (osquery)
  31. - **Velociraptor Endpoint Forensics** — 25 MCP tools for live artefact collection (processes, network, persistence, execution evidence, NTFS MFT, SRUM, remote EVTX) with automatic IoC extraction
  32. - **Encoded-Command Deobfuscation** — Automatically peels base64/UTF-16LE/gzip/hex layers off encoded commands (e.g. PowerShell `-EncodedCommand`) so C2 addresses hidden inside them surface as IoCs
  33. - **Adaptive Schema Intelligence** — Data-driven schema registry (ECS, Sysmon, Windows Security, auditd, Zeek, Packetbeat, AWS CloudTrail) adapts hunts to each index's field conventions
  34. - **DFIR Knowledge Resources** — 9 MCP resources exposing investigation playbooks, Pyramid of Pain reference, and cross-correlation guidance directly to connected AI agents
  35. - **Standalone CLI** — Full threat hunting from the terminal without an MCP client
  36. ---
  37. <a name="installation"></a>
  38. ## Installation
  39. ### Install from PyPI (recommended)
  40. ```bash
  41. # Install with pip
  42. pip install crowdsentinel-mcp-server
  43. # Or install with uv
  44. uv pip install crowdsentinel-mcp-server
  45. # Download detection rules, Chainsaw, and Sigma rules (one-time)
  46. crowdsentinel setup
  47. # Validate connector configuration and readiness (offline, no external calls)
  48. crowdsentinel doctor
  49. ```
  50. **Optional extras:**
  51. ```bash
  52. # Velociraptor live endpoint forensics (adds 25 MCP tools)
  53. pip install 'crowdsentinel-mcp-server[velociraptor]'
  54. # Long-tail deobfuscation coverage via chepy (XOR, exotic encodings).
  55. # The stdlib decoder core works without this — chepy only extends it.
  56. pip install 'crowdsentinel-mcp-server[deobf]'
  57. ```
  58. Detection rules (6,060 Lucene + EQL + ES|QL) are **bundled with the package** — no download needed. The `setup` command downloads additional tools:
  59. - Chainsaw binary for EVTX analysis
  60. - 3,000+ Sigma rules for Chainsaw
  61. Downloaded tools are stored in `~/.crowdsentinel/` and persist across package upgrades.
  62. **System dependency for PCAP analysis:**
  63. ```bash
  64. # Required for network traffic analysis and cross-tool IoC correlation
  65. sudo apt install tshark # Debian/Ubuntu/Kali
  66. sudo dnf install wireshark-cli # Fedora/RHEL
  67. brew install wireshark # macOS
  68. ```
  69. ### Run directly with uvx (no install needed)
  70. ```bash
  71. # Elasticsearch 8.x (default)
  72. uvx crowdsentinel-mcp-server
  73. # Other backends
  74. uvx crowdsentinel-mcp-server-es7 # Elasticsearch 7.x
  75. uvx crowdsentinel-mcp-server-es9 # Elasticsearch 9.x
  76. uvx opensearch-mcp-server # OpenSearch 1.x/2.x/3.x
  77. ```
  78. ### Install from source
  79. ```bash
  80. git clone https://github.com/thomasxm/CrowdSentinels-AI-MCP.git
  81. cd CrowdSentinels-AI-MCP
  82. chmod +x setup.sh && ./setup.sh
  83. # Or install everything, including all optional extras (deobf/chepy, velociraptor)
  84. ./setup.sh --full
  85. ```
  86. The setup script will:
  87. - Install dependencies (pipx, uv, Claude Code CLI if needed)
  88. - Bundle 6,060 detection rules and download Chainsaw binary
  89. - Prompt for Elasticsearch credentials (never hardcoded)
  90. - Configure the MCP server with Claude Code
  91. - Validate your connection
  92. With `--full`, every optional extra is installed too, so all 139 MCP tools have their dependencies available — the recommended mode for a production-ready server.
  93. ### Installed Size
  94. CrowdSentinel bundles 6,060 detection rules and integrates with external analysis tools. Below is the full disk space breakdown so you can plan accordingly.
  95. **Core package (via `pip` or `uvx`):**
  96. | Component | Size | Notes |
  97. |---|---|---|
  98. | **CrowdSentinel package** | **49 MB** | The server itself |
  99. | — Bundled Sigma rules (`src/rules/`) | 30 MB | 6,060 pre-converted detection rules |
  100. | — Elastic TOML rules (`src/detection-rules/`) | 17 MB | Original TOML format rules + hunting queries |
  101. | — Python code (clients, tools, etc.) | 2 MB | Actual application code |
  102. | **Dependencies** | **64 MB** | All transitive deps |
  103. | — `cryptography` | 14 MB | Largest dependency (TLS) |
  104. | — `elasticsearch` | 8.3 MB | ES Python client |
  105. | — `pygments` | 5.2 MB | Syntax highlighting |
  106. | — `pydantic_core` | 5 MB | Validation engine |
  107. | — `opensearchpy` | 3.6 MB | OpenSearch client |
  108. | — Others (27 packages) | ~28 MB | mcp, fastmcp, httpx, anthropic, etc. |
  109. | **Core total** | **113 MB** | `pip install crowdsentinel-mcp-server` |
  110. **Additional tools (via `crowdsentinel setup`):**
  111. | Component | Download | Installed | Notes |
  112. |---|---|---|---|
  113. | Chainsaw binary (v2.13.1) | ~3 MB | ~15 MB | EVTX log analysis engine |
  114. | Sigma rules (SigmaHQ) | ~3 MB | ~30 MB | 3,000+ Sigma rules for Chainsaw |
  115. | Chainsaw mappings | — | <1 MB | Event log source mappings |
  116. | **Setup total** | **~6 MB** | **~46 MB** | Stored in `~/.crowdsentinel/` |
  117. **System dependency (via package manager):**
  118. | Component | Installed | Install Command | Notes |
  119. |---|---|---|---|
  120. | tshark + Wireshark libs | ~132 MB | `sudo apt install tshark` | PCAP network analysis — required for cross-tool IoC correlation |
  121. **Full installation summary:**
  122. | Scenario | Total Disk Space |
  123. |---|---|
  124. | Core only (`pip install`) | ~113 MB |
  125. | Core + setup (`crowdsentinel setup`) | ~159 MB |
  126. | Full platform (+ tshark) | **~291 MB** |
  127. > **Note:** PyPI download size is only **8.9 MB** (wheel) thanks to compression of the bundled detection rules.
  128. ---
  129. <a name="quick-start"></a>
  130. ## Quick Start
  131. ### 1. Set environment variables
  132. ```bash
  133. export ELASTICSEARCH_HOSTS="https://localhost:9200"
  134. export ELASTICSEARCH_API_KEY="your_api_key"
  135. # Or use username/password:
  136. # export ELASTICSEARCH_USERNAME="elastic"
  137. # export ELASTICSEARCH_PASSWORD="your_password"
  138. export VERIFY_CERTS="false"
  139. ```
  140. ### 2. Connect to an MCP Client
  141. CrowdSentinel works with any MCP-compatible AI agent. Choose your client below:
  142. <details>
  143. <summary><b>Claude Code (CLI)</b></summary>
  144. ```bash
  145. claude mcp add crowdsentinel \
  146. -e ELASTICSEARCH_HOSTS="https://localhost:9200" \
  147. -e ELASTICSEARCH_API_KEY="your_api_key" \
  148. -e VERIFY_CERTS="false" \
  149. -- uvx crowdsentinel-mcp-server
  150. ```
  151. </details>
  152. <details>
  153. <summary><b>Claude Desktop</b></summary>
  154. Edit `~/.config/Claude/claude_desktop_config.json` (Linux) or `~/Library/Application Support/Claude/claude_desktop_config.json` (macOS):
  155. ```json
  156. {
  157. "mcpServers": {
  158. "crowdsentinel": {
  159. "command": "uvx",
  160. "args": ["crowdsentinel-mcp-server"],
  161. "env": {
  162. "ELASTICSEARCH_HOSTS": "https://localhost:9200",
  163. "ELASTICSEARCH_API_KEY": "your_api_key",
  164. "VERIFY_CERTS": "false"
  165. }
  166. }
  167. }
  168. }
  169. ```
  170. </details>
  171. <details>
  172. <summary><b>VS Code Copilot</b></summary>
  173. Create `.vscode/mcp.json` in your workspace:
  174. ```json
  175. {
  176. "servers": {
  177. "crowdsentinel": {
  178. "command": "uvx",
  179. "args": ["crowdsentinel-mcp-server"],
  180. "env": {
  181. "ELASTICSEARCH_HOSTS": "https://localhost:9200",
  182. "ELASTICSEARCH_API_KEY": "your_api_key",
  183. "VERIFY_CERTS": "false"
  184. }
  185. }
  186. }
  187. }
  188. ```
  189. Then enable MCP in VS Code settings: `"chat.mcp.enabled": true`
  190. </details>
  191. <details>
  192. <summary><b>Cursor</b></summary>
  193. Create or edit `~/.cursor/mcp.json`:
  194. ```json
  195. {
  196. "mcpServers": {
  197. "crowdsentinel": {
  198. "command": "uvx",
  199. "args": ["crowdsentinel-mcp-server"],
  200. "env": {
  201. "ELASTICSEARCH_HOSTS": "https://localhost:9200",
  202. "ELASTICSEARCH_API_KEY": "your_api_key",
  203. "VERIFY_CERTS": "false"
  204. }
  205. }
  206. }
  207. }
  208. ```
  209. </details>
  210. <details>
  211. <summary><b>Roo Code (VS Code Extension)</b></summary>
  212. Create `.roo/mcp.json` in your workspace:
  213. ```json
  214. {
  215. "mcpServers": {
  216. "crowdsentinel": {
  217. "command": "uvx",
  218. "args": ["crowdsentinel-mcp-server"],
  219. "env": {
  220. "ELASTICSEARCH_HOSTS": "https://localhost:9200",
  221. "ELASTICSEARCH_API_KEY": "your_api_key",
  222. "VERIFY_CERTS": "false"
  223. }
  224. }
  225. }
  226. }
  227. ```
  228. Or configure via Roo Code settings panel: Settings > MCP Servers > Add Server.
  229. </details>
  230. <details>
  231. <summary><b>5ire</b></summary>
  232. In 5ire settings (v0.15.0+), add an MCP server with:
  233. - **Command**: `uvx`
  234. - **Arguments**: `crowdsentinel-mcp-server`
  235. - **Environment Variables**:
  236. - `ELASTICSEARCH_HOSTS` = `https://localhost:9200`
  237. - `ELASTICSEARCH_API_KEY` = `your_api_key`
  238. - `VERIFY_CERTS` = `false`
  239. > **Note:** 5ire v0.14.0 has known MCP compatibility issues. Use v0.15.0+ for reliable operation.
  240. </details>
  241. <details>
  242. <summary><b>Any MCP Client (Generic)</b></summary>
  243. **stdio transport** (default — works with most clients):
  244. ```json
  245. {
  246. "mcpServers": {
  247. "crowdsentinel": {
  248. "command": "uvx",
  249. "args": ["crowdsentinel-mcp-server"],
  250. "env": {
  251. "ELASTICSEARCH_HOSTS": "https://localhost:9200",
  252. "ELASTICSEARCH_API_KEY": "your_api_key",
  253. "VERIFY_CERTS": "false"
  254. }
  255. }
  256. }
  257. }
  258. ```
  259. **SSE transport** (for web-based clients):
  260. ```bash
  261. crowdsentinel-mcp-server --transport sse --port 8001
  262. # Connect to: http://localhost:8001/sse/
  263. ```
  264. **HTTP transport** (for REST API clients):
  265. ```bash
  266. crowdsentinel-mcp-server --transport streamable-http --port 8001
  267. # Connect to: http://localhost:8001/mcp/
  268. ```
  269. </details>
  270. ### 3. Or use the CLI directly
  271. ```bash
  272. # Download rules and tools (one-time)
  273. crowdsentinel setup
  274. # Check cluster health
  275. crowdsentinel health
  276. # Hunt for threats
  277. crowdsentinel hunt "powershell encoded" -i winlogbeat-*
  278. # Run detection rules
  279. crowdsentinel rules -p windows --tactic credential_access
  280. crowdsentinel detect windows_builtin_win_alert_mimikatz_keywords_lucene -i winlogbeat-*
  281. # Analyse PCAP files
  282. crowdsentinel pcap overview capture.pcap
  283. crowdsentinel pcap beaconing capture.pcap
  284. # Hunt EVTX logs with Chainsaw
  285. crowdsentinel chainsaw hunt /path/to/evtx/ --sigma-rules /path/to/sigma/
  286. ```
  287. ---
  288. <a name="cli-usage"></a>
  289. ## CLI Usage
  290. CrowdSentinel provides a full CLI for threat hunting from the terminal:
  291. ```bash
  292. pip install crowdsentinel-mcp-server
  293. crowdsentinel setup # Download rules, Chainsaw, Sigma (one-time)
  294. crowdsentinel --help
  295. ```
  296. ### Available Commands
  297. | Command | Description | Example |
  298. |:--------|:------------|:--------|
  299. | `setup` | Download detection rules, Chainsaw, and Sigma rules | `crowdsentinel setup` |
  300. | `health` | Show cluster health | `crowdsentinel health` |
  301. | `indices` | List all indices | `crowdsentinel indices` |
  302. | `hunt` | IR-focused threat hunt with IoC extraction | `crowdsentinel hunt "powershell" -i winlogbeat-*` |
  303. | `eql` | Execute an EQL query | `crowdsentinel eql "process where process.name == 'cmd.exe'" -i winlogbeat-*` |
  304. | `esql` | Execute an ES\|QL query | `crowdsentinel esql "FROM logs-* \| LIMIT 10"` |
  305. | `detect` | Execute a detection rule by ID | `crowdsentinel detect win_susp_logon -i winlogbeat-*` |
  306. | `rules` | List available detection rules | `crowdsentinel rules -p windows --tactic credential_access --type eql` |
  307. | `schema` | Detect schema for an index pattern | `crowdsentinel schema -i winlogbeat-*` |
  308. | `ioc` | Hunt for a specific Indicator of Compromise | `crowdsentinel ioc 203.0.113.42 --type ip -i winlogbeat-*` |
  309. | `analyse` | Analyse search results from stdin (JSON) | `cat results.json \| crowdsentinel analyse -c "context"` |
  310. | `analyse --mcp` | AI agent analysis using all 139 MCP tools | `crowdsentinel hunt "query" \| crowdsentinel analyse --mcp -c "context"` |
  311. | `auth` | Manage LLM authentication for agent mode | `crowdsentinel auth login` |
  312. | `pcap` | Analyse PCAP files (overview, beaconing, lateral movement) | `crowdsentinel pcap beaconing capture.pcap` |
  313. | `chainsaw` | Hunt EVTX logs with Chainsaw and Sigma rules | `crowdsentinel chainsaw hunt /path/to/evtx/` |
  314. | `doctor` | Validate connector configuration and readiness (offline — no external calls) | `crowdsentinel doctor` |
  315. ### Output Formats
  316. All commands support `--output/-o` with three formats:
  317. ```bash
  318. crowdsentinel hunt "failed login" -i winlogbeat-* -o json # Structured JSON (default)
  319. crowdsentinel hunt "failed login" -i winlogbeat-* -o table # Human-readable table
  320. crowdsentinel hunt "failed login" -i winlogbeat-* -o summary # Condensed summary
  321. ```
  322. ### Agent Mode (`--mcp`)
  323. The `analyse --mcp` flag replaces deterministic analysis with an AI agent that autonomously uses all 139 MCP tools to investigate. The agent follows the 4-phase IR methodology: hunt, analyse, correlate, report.
  324. **Authentication:**
  325. ```bash
  326. # Option 1: Browser sign-in (ChatGPT subscription — no API billing)
  327. crowdsentinel auth login
  328. # Option 2: Anthropic (setup-token or API key)
  329. crowdsentinel auth login --provider anthropic
  330. # Option 3: Environment variable
  331. export ANTHROPIC_API_KEY="sk-ant-..." # or OPENAI_API_KEY
  332. # Option 4: Local models (Ollama, vLLM — free)
  333. crowdsentinel analyse --mcp --model-url http://localhost:11434/v1 --model llama3.1
  334. # Check auth status
  335. crowdsentinel auth status
  336. ```
  337. **Agent flags:**
  338. | Flag | Default | Description |
  339. |:-----|:--------|:------------|
  340. | `--mcp` | off | Enable AI agent with MCP tools |
  341. | `--mcp-server NAME:CMD` | none | Add external MCP server (e.g., VirusTotal) |
  342. | `--model` | auto-detect | LLM model to use |
  343. | `--model-url` | none | OpenAI-compatible API endpoint |
  344. | `--max-steps` | 30 | Maximum tool calls |
  345. | `--timeout` | 300 | Maximum seconds |
  346. ### Pipeline Examples
  347. **Deterministic analysis (no API key needed):**
  348. ```bash
  349. # Hunt then analyse
  350. crowdsentinel hunt "powershell encoded" -i winlogbeat-* -o json | \
  351. crowdsentinel analyse -c "Encoded PowerShell commands" -o summary
  352. # Investigate failed authentication attempts
  353. crowdsentinel hunt "event.code:4625" -i winlogbeat-* -o json | \
  354. crowdsentinel analyse -c "Failed login brute force investigation" -o summary
  355. # Triage process execution and privilege escalation
  356. crowdsentinel hunt "event.code:4688 OR event.code:4672 OR event.code:1" -i winlogbeat-* -o json | \
  357. crowdsentinel analyse -c "Process execution and privilege escalation" -o summary
  358. ```
  359. **AI agent investigation (requires auth):**
  360. ```bash
  361. # Credential dumping investigation — agent hunts, analyses kill chain, checks adjacent stages
  362. crowdsentinel hunt "mimikatz OR lsass OR procdump" -i winlogbeat-* -o json | \
  363. crowdsentinel analyse --mcp -c "Credential dumping tools investigation" --max-steps 15 -o summary
  364. # Encoded PowerShell — full IR workflow with kill chain and adjacent stage hunting
  365. crowdsentinel hunt "powershell -enc OR FromBase64String" -i winlogbeat-* -o json | \
  366. crowdsentinel analyse --mcp -c "Full IR workflow: encoded PowerShell" --max-steps 30 -o table
  367. # Process execution with detection rules
  368. crowdsentinel hunt "event.code:4688" -i winlogbeat-* -o json | \
  369. crowdsentinel analyse --mcp -c "Execute detection rules against process creation" --max-steps 20 -o summary
  370. # PCAP beaconing — agent generates IoCs and maps to kill chain
  371. crowdsentinel pcap beaconing capture.pcap -o json | \
  372. crowdsentinel analyse --mcp -c "Investigate beaconing for C2 infrastructure" --max-steps 10 -o summary
  373. # Anti-forensics investigation
  374. crowdsentinel hunt "event.code:1102" -i winlogbeat-* -o json | \
  375. crowdsentinel analyse --mcp -c "Security log cleared - anti-forensics" --max-steps 10 -o summary
  376. # With external MCP server (e.g., VirusTotal)
  377. crowdsentinel hunt "powershell" -i winlogbeat-* -o json | \
  378. crowdsentinel analyse --mcp --mcp-server "vt:uvx virustotal-mcp-server" \
  379. -c "Check IoCs against VirusTotal" -o summary
  380. ```
  381. ---
  382. <a name="key-features"></a>
  383. ## Key Features
  384. <table>
  385. <tr>
  386. <td width="50%">
  387. ### 139 MCP Tools
  388. Threat hunting, detection rules, forensics, endpoint collection, network analysis, cross-correlation, command deobfuscation, and IoC enrichment — all accessible via natural language
  389. ### 6,060 Detection Rules
  390. Pre-built Lucene, EQL & ES|QL rules with automatic MITRE ATT&CK mapping
  391. ### Investigation State
  392. Persistent IoC tracking across tools and sessions with cross-source correlation and FIFO storage
  393. </td>
  394. <td width="50%">
  395. ### 4 Security Frameworks
  396. - Cyber Kill Chain (7 stages)
  397. - Pyramid of Pain (6 levels)
  398. - Diamond Model (4 vertices)
  399. - MITRE ATT&CK (automatic mapping)
  400. ### 5 Data Sources + Threat Intel
  401. - Elasticsearch / OpenSearch (SIEM)
  402. - Velociraptor (live endpoint forensics)
  403. - osquery (local host forensics)
  404. - EVTX logs (Chainsaw + Sigma)
  405. - PCAP files (Wireshark/TShark)
  406. - IoC enrichment (Shodan, VirusTotal, AbuseIPDB, ThreatFox)
  407. - MISP (export events, search attributes)
  408. </td>
  409. </tr>
  410. </table>
  411. ---
  412. <a name="architecture"></a>
  413. ## Architecture
  414. ```
  415. ┌─────────────────────────────────────────────────────────────────┐
  416. │ LLM Client / Claude Code CLI │
  417. └─────────────────────────────┬───────────────────────────────────┘
  418. │ MCP Protocol (stdio/SSE/HTTP)
  419. │ HTTP: Origin validation (anti DNS-rebinding)
  420. │ + transport-level audit logging
  421. ▼
  422. ┌─────────────────────────────────────────────────────────────────┐
  423. │ CrowdSentinel MCP Server │
  424. │ ┌───────────────┐ ┌───────────────┐ ┌───────────────────────┐ │
  425. │ │ 139 Tools │ │ 6,060 Rules │ │ Security Frameworks │ │
  426. │ │ - Hunting │ │ - Lucene │ │ - Cyber Kill Chain │ │
  427. │ │ - Detection │ │ - EQL │ │ - Pyramid of Pain │ │
  428. │ │ - Forensics │ │ - ES|QL │ │ - Diamond Model │ │
  429. │ │ - Endpoint │ │ │ │ - MITRE ATT&CK │ │
  430. │ │ - Network │ ├───────────────┤ │ │ │
  431. │ │ - Enrichment │ │ Schema │ │ 9 MCP Resources │ │
  432. │ │ - Correlation │ │ Registry │ │ (DFIR knowledge base) │ │
  433. │ │ - Deobfuscate │ │ (data-driven) │ │ │ │
  434. │ └───────────────┘ └───────────────┘ └───────────────────────┘ │
  435. │ ┌─────────────────────────────────────────────────────────────┐│
  436. │ │ Investigation State (Persistent) ││
  437. │ │ Cross-source IoC sharing, auto-capture, STIX 2.1 export ││
  438. │ └─────────────────────────────────────────────────────────────┘│
  439. └─────┬──────────┬──────────┬──────────────┬─────────────┬────────┘
  440. │ │ │ │ │
  441. ▼ ▼ ▼ ▼ ▼
  442. ┌───────────┐ ┌──────────┐ ┌────────┐ ┌───────────┐ ┌───────────┐
  443. │Elastic- │ │Velocirap-│ │osquery │ │ Chainsaw │ │ Wireshark │
  444. │search/ │ │tor │ │(local │ │(EVTX/ │ │ (PCAP) │
  445. │OpenSearch │ │(EDR/DFIR)│ │ host) │ │ Sigma) │ │ (Network) │
  446. │ (SIEM) │ │(Endpoint)│ │ │ │(Offline) │ │ │
  447. └───────────┘ └──────────┘ └────────┘ └───────────┘ └───────────┘
  448. │
  449. ┌───────────────────────┼───────────────────────┐
  450. ▼ ▼ ▼
  451. ┌───────────────┐ ┌───────────────┐ ┌───────────────────────────┐
  452. │ Shodan │ │ VirusTotal │ │ AbuseIPDB + ThreatFox │
  453. │ InternetDB │ │ (v3) │ │ + MISP (export/search) │
  454. │ (free/no key)│ │ (free tier) │ │ (free tier) │
  455. └───────────────┘ └───────────────┘ └───────────────────────────┘
  456. │
  457. ▼ (Roadmap)
  458. ┌───────────────┐ ┌───────────────┐ ┌───────────────┐
  459. │ Splunk │ │ Wazuh │ │ Zeek │
  460. │ │ │ (EDR/XDR) │ │ (NSM/IDS) │
  461. └───────────────┘ └───────────────┘ └───────────────┘
  462. ```
  463. ---
  464. ## What's Included
  465. ### Tool Categories (139 Tools)
  466. | Category | Tools | Description |
  467. |:---------|:-----:|:------------|
  468. | **Elasticsearch Core** | 19 | Index, document, cluster, alias, data stream, raw API operations |
  469. | **Threat Hunting** | 8 | Attack pattern detection, kill-chain stage hunting, timeframe hunts, anomaly detection |
  470. | **EQL & ES\|QL Hunting** | 11 | Event-sequence queries, ES\|QL pipes, saved hunt history, capability detection |
  471. | **Asset Discovery & Schema Intelligence** | 12 | Index discovery, data-driven schema detection (ECS, Sysmon, auditd, Zeek, CloudTrail…), smart search |
  472. | **IoC Analysis & Deobfuscation** | 6 | IoC extraction, encoded-command decoding, kill-chain mapping, investigation reports |
  473. | **Detection Rules** | 9 | 6,060 rule library — list, execute, validate, suggest, batch execution |
  474. | **Investigation Prompts** | 5 | Fast triage spine — 10 critical IR questions |
  475. | **Chainsaw (EVTX)** | 5 | Sigma rule hunting, iterative IoC discovery |
  476. | **Wireshark (PCAP)** | 11 | Network forensics, beaconing, lateral movement detection, traffic decoding |
  477. | **Threat Intelligence** | 5 | IoC enrichment (Shodan, VirusTotal, AbuseIPDB, ThreatFox) + MISP export/search |
  478. | **Investigation State** | 10 | Persistent IoCs, cross-tool sharing, STIX 2.1 export, reporting |
  479. | **Velociraptor (Endpoint)** | 25 | Live forensic artefact collection — processes, network, persistence, execution evidence, MFT, SRUM |
  480. | **osquery (Local Host)** | 4 | SQL-based local host forensics — processes, sockets, users, ad-hoc queries |
  481. | **Cross-Source Correlation** | 5 | SIEM ↔ endpoint pivots, IoC-driven endpoint hunts, unified timelines, time correlation |
  482. | **Workflow Guidance & Status** | 4 | Next-step guidance, progressive disclosure, system status, storage cleanup |
  483. > Velociraptor tools register when `VELOCIRAPTOR_API_CONFIG` is set; osquery tools register when a local `osqueryi` binary is present; the 10 high-risk write tools are hidden when `DISABLE_HIGH_RISK_OPERATIONS=true`.
  484. ### Security Frameworks
  485. | Framework | Purpose |
  486. |:----------|:--------|
  487. | **Cyber Kill Chain** | Hunt by attack stage (7 stages), predict adversary's next move |
  488. | **Pyramid of Pain** | Prioritise IoCs by difficulty for attackers to change (6 levels) |
  489. | **Diamond Model** | Map adversary, capability, infrastructure, victim relationships |
  490. | **MITRE ATT&CK** | Automatic technique mapping for all detections |
  491. ### Detection Rules (6,060 Rules)
  492. | Type | Count | Source | Description |
  493. |:-----|------:|:-------|:------------|
  494. | **Lucene** | 1,966 | Sigma-converted | Fast pattern matching queries |
  495. | **EQL** | 3,963 | Sigma-converted + Elastic | Event sequences and correlations |
  496. | **ES\|QL** | 131 | Elastic TOML rules | Pipe-based query language (ES 8.11+) |
  497. **Platforms:** Windows, Linux, macOS, Cloud (AWS/Azure/GCP), Network, Identity
  498. **Log Sources:** PowerShell, Sysmon, Security Events, Process Creation, Audit logs
  499. ### Encoded-Command Deobfuscation
  500. Real intrusions hide C2 addresses and URLs inside encodings the SIEM stores verbatim — most commonly PowerShell `-EncodedCommand` (base64 of UTF-16LE), but also `FromBase64String`, base64→gzip, bare base64, and hex blobs. CrowdSentinel decodes these layers automatically during IoC extraction and via the dedicated `decode_command` tool, so the inner indicators surface instead of staying buried:
  501. - **Zero extra dependencies** — the decoder core uses only the Python standard library (`base64`/`zlib`/`codecs`/`binascii`)
  502. - **Optional long tail** — install the `deobf` extra (`pip install 'crowdsentinel-mcp-server[deobf]'`) to add [chepy](https://github.com/securisec/chepy) for XOR and exotic encodings; its absence degrades gracefully
  503. - **Provenance** — every decode records its recipe (e.g. `base64->utf-16le`) for the IR report
  504. - **Safe by design** — decode-only (nothing is executed), with depth and output-size bounds so a decompression bomb cannot hang the server
  505. ---
  506. ## Configuration
  507. ### Environment Variables
  508. ```bash
  509. # Connection (required — choose one)
  510. ELASTICSEARCH_HOSTS="https://localhost:9200" # Self-hosted
  511. # OR
  512. ELASTICSEARCH_CLOUD_ID="deployment:base64..." # Elastic Cloud
  513. # Authentication — choose one (in priority order):
  514. ELASTICSEARCH_BEARER_TOKEN="service_token_here" # Service/bearer token
  515. ELASTICSEARCH_API_KEY="your_api_key" # API key (recommended)
  516. ELASTICSEARCH_USERNAME="elastic" # Basic auth
  517. ELASTICSEARCH_PASSWORD="your_password"
  518. # TLS / Certificate verification
  519. VERIFY_CERTS="true" # Verify against system CA bundle
  520. # VERIFY_CERTS="/path/to/ca.crt" # Verify against custom CA certificate
  521. # ELASTICSEARCH_CA_CERT="/path/to/ca.crt" # Explicit CA certificate path
  522. # ELASTICSEARCH_CLIENT_CERT="/path/to/client.crt" # Client certificate (mTLS)
  523. # ELASTICSEARCH_CLIENT_KEY="/path/to/client.key" # Client private key (mTLS)
  524. # Options
  525. REQUEST_TIMEOUT="30" # Request timeout in seconds
  526. DISABLE_HIGH_RISK_OPERATIONS="true" # Block all write operations
  527. # HTTP/SSE transport security (stdio unaffected)
  528. # MCP_ALLOWED_ORIGINS="https://app.example.com" # Extra allowed Origins (comma-separated); loopback always allowed
  529. # Velociraptor endpoint forensics (optional — enables 25 extra tools)
  530. # VELOCIRAPTOR_API_CONFIG="/path/to/api_client.yaml"
  531. # Threat Intelligence (optional — Shodan InternetDB works without any key)
  532. VIRUSTOTAL_API_KEY="your_vt_key" # Free: 500 lookups/day
  533. ABUSEIPDB_API_KEY="your_abuse_key" # Free: 1,000 lookups/day
  534. THREATFOX_API_KEY="your_tf_key" # Free: unlimited
  535. # MISP Integration (optional — offline JSON export works without a server)
  536. MISP_URL="https://misp.example.org" # MISP instance URL
  537. MISP_API_KEY="your_misp_key" # MISP API key (40-char hex)
  538. # MISP_SSL_VERIFY="true" # Set "false" for self-signed (dev only)
  539. ```
  540. > **Security Warning:** Never use `VERIFY_CERTS="false"` or plain-text passwords in production. Use API keys or service tokens with TLS certificate verification enabled. For self-signed certificates, set `ELASTICSEARCH_CA_CERT` to your CA certificate path.
  541. ### Production Configuration Examples
  542. <details>
  543. <summary><b>Elastic Cloud</b></summary>
  544. ```bash
  545. ELASTICSEARCH_CLOUD_ID="my-deployment:dXMtY2VudHJhbC0x..."
  546. ELASTICSEARCH_API_KEY="your_cloud_api_key"
  547. VERIFY_CERTS="true"
  548. ```
  549. </details>
  550. <details>
  551. <summary><b>Self-Hosted with Custom CA</b></summary>
  552. ```bash
  553. ELASTICSEARCH_HOSTS="https://es-cluster.internal:9200"
  554. ELASTICSEARCH_API_KEY="your_api_key"
  555. ELASTICSEARCH_CA_CERT="/etc/elasticsearch/certs/ca.crt"
  556. VERIFY_CERTS="true"
  557. ```
  558. </details>
  559. <details>
  560. <summary><b>Mutual TLS (mTLS)</b></summary>
  561. ```bash
  562. ELASTICSEARCH_HOSTS="https://es-cluster.internal:9200"
  563. ELASTICSEARCH_CA_CERT="/etc/elasticsearch/certs/ca.crt"
  564. ELASTICSEARCH_CLIENT_CERT="/etc/elasticsearch/certs/client.crt"
  565. ELASTICSEARCH_CLIENT_KEY="/etc/elasticsearch/certs/client.key"
  566. VERIFY_CERTS="true"
  567. ```
  568. </details>
  569. <details>
  570. <summary><b>Development / Testing (insecure)</b></summary>
  571. ```bash
  572. ELASTICSEARCH_HOSTS="http://localhost:9200"
  573. ELASTICSEARCH_USERNAME="elastic"
  574. ELASTICSEARCH_PASSWORD="test123"
  575. VERIFY_CERTS="false"
  576. ```
  577. > This configuration is **not suitable for production**. Use API keys with TLS in production environments.
  578. </details>
  579. ### Transport Modes
  580. | Mode | Use Case | Command |
  581. |:-----|:---------|:--------|
  582. | **stdio** (default) | MCP clients (Claude Code, etc.) | Auto-configured by setup.sh |
  583. | **SSE** | Web/streaming clients | `--transport sse --port 8000` |
  584. | **HTTP** | REST API access | `--transport streamable-http` |
  585. The HTTP transports are hardened at the transport layer:
  586. - **Origin validation (anti DNS-rebinding)** — browser requests are rejected unless their `Origin` is loopback or listed in `MCP_ALLOWED_ORIGINS` (comma-separated; `*` alone disables the check). Requests without an `Origin` header — every non-browser MCP client — are unaffected.
  587. - **Audit logging** — every HTTP exchange is logged at the transport boundary (JSON-RPC method, tool name, body size, path), including requests the tool layer never sees, such as unknown tools or protocol errors. Logs are metadata-only, memory-bounded, and sanitised against log forgery.
  588. ### Disabling Write Operations
  589. For read-only access (recommended for production):
  590. ```bash
  591. DISABLE_HIGH_RISK_OPERATIONS="true"
  592. ```
  593. This hides destructive tools: `create_index`, `delete_index`, `index_document`, `delete_document`, `delete_by_query`, `create_data_stream`, `delete_data_stream`, `put_alias`, `delete_alias`, `general_api_request`
  594. ---
  595. <a name="usage-examples"></a>
  596. ## Usage Examples
  597. ### Example 1: Hunt for Brute Force Attacks
  598. ```
  599. You: Hunt for brute force attacks in the last hour
  600. CrowdSentinel: Searching for failed authentication attempts...
  601. [Calls hunt_by_timeframe with attack_types=["brute_force"]]
  602. Found 47 failed login attempts from 3 source IPs targeting the "admin" account.
  603. Top sources:
  604. - 192.168.1.105: 28 attempts
  605. - 10.0.0.42: 15 attempts
  606. - 172.16.0.8: 4 attempts
  607. ```
  608. ### Example 2: Investigate a Suspicious Host
  609. ```
  610. You: Show me all activity on host DESKTOP-ABC123 in the last 24 hours
  611. CrowdSentinel: Generating a forensic timeline for that host...
  612. [Calls get_host_activity_timeline]
  613. Timeline shows:
  614. - 09:15: User "jsmith" logged in (Event 4624)
  615. - 09:17: PowerShell executed encoded command
  616. - 09:18: New scheduled task created "WindowsUpdate"
  617. - 09:22: Outbound connection to 203.0.113.42:443
  618. - 09:25: LSASS memory access detected
  619. ```
  620. ### Example 3: Kill Chain Analysis
  621. ```
  622. You: We found C2 beaconing. What should we hunt for next?
  623. CrowdSentinel: Hunting adjacent kill chain stages...
  624. [Calls hunt_adjacent_stages with current_stage="COMMAND_AND_CONTROL"]
  625. Looking backwards (Installation stage):
  626. - Found scheduled task persistence on 3 hosts
  627. - Registry Run key modifications detected
  628. Looking forwards (Actions on Objectives):
  629. - Lateral movement to 2 additional hosts via SMB
  630. - Credential dumping attempts detected
  631. ```
  632. ### Example 4: Network Traffic Analysis
  633. ```
  634. You: Analyse this PCAP for C2 beaconing
  635. CrowdSentinel: Checking for regular connection intervals...
  636. [Calls detect_beaconing on the PCAP file]
  637. Detected beaconing pattern:
  638. - Host: 192.168.1.50 → 203.0.113.100:443
  639. - Interval: 60 seconds (±2s jitter)
  640. - Duration: 4 hours
  641. - Confidence: HIGH
  642. Recommend blocking 203.0.113.100 and investigating 192.168.1.50.
  643. ```
  644. ### Example 5: CLI Threat Hunt
  645. ```bash
  646. # Hunt for encoded PowerShell
  647. crowdsentinel hunt "powershell -enc" -i winlogbeat-* --timeframe 1440 -o json
  648. # Pipe results to analysis
  649. crowdsentinel hunt "event.code:4625" -i winlogbeat-* -o json | \
  650. crowdsentinel analyse -c "Failed authentication investigation"
  651. # Search detection rules for lateral movement
  652. crowdsentinel rules --tactic lateral_movement -p windows
  653. ```
  654. ---
  655. ## Compatibility
  656. | Package | Backend | Install |
  657. |:--------|:--------|:--------|
  658. | `crowdsentinel-mcp-server` | Elasticsearch 8.x (default) | `pip install crowdsentinel-mcp-server` |
  659. | `crowdsentinel-mcp-server-es7` | Elasticsearch 7.x | `pip install crowdsentinel-mcp-server-es7` |
  660. | `crowdsentinel-mcp-server-es9` | Elasticsearch 9.x | `pip install crowdsentinel-mcp-server-es9` |
  661. | `opensearch-mcp-server` | OpenSearch 1.x, 2.x, 3.x | `pip install opensearch-mcp-server` |
  662. ---
  663. ## For Developers
  664. <details>
  665. <summary><b>Project Structure</b></summary>
  666. ```
  667. crowdsentinel-mcp-server/
  668. ├── src/
  669. │ ├── server.py # MCP server entry point
  670. │ ├── version.py # Version constant
  671. │ ├── risk_config.py # Write operation controls
  672. │ ├── transport_security.py # Origin validation for HTTP transports (anti DNS-rebinding)
  673. │ ├── transport_audit.py # Transport-level audit logging (every HTTP exchange)
  674. │ │
  675. │ ├── cli/ # Standalone CLI
  676. │ │ └── main.py # CLI entry point (argparse) — 15 commands incl. doctor
  677. │ │
  678. │ ├── agent/ # `analyse --mcp` AI agent mode (multi-provider LLM auth)
  679. │ ├── config/ # Connector configuration
  680. │ │
  681. │ ├── clients/ # Backend logic layer
  682. │ │ ├── base.py # Base client, authentication
  683. │ │ ├── exceptions.py # Exception handling decorators
  684. │ │ └── common/
  685. │ │ ├── client.py # Unified SearchClient (multiple inheritance)
  686. │ │ ├── threat_hunting.py # Threat hunting queries
  687. │ │ ├── ioc_analysis.py # IoC extraction & analysis
  688. │ │ ├── deobfuscate.py # Encoded-command decoding (stdlib core + optional chepy)
  689. │ │ ├── cyber_kill_chain.py # Kill chain logic
  690. │ │ ├── rule_loader.py # Detection rule loading
  691. │ │ ├── chainsaw_client.py # EVTX/Sigma integration
  692. │ │ ├── osquery_client.py # Local osquery execution (sqlglot-validated SQL)
  693. │ │ ├── misp_client.py # MISP export/search
  694. │ │ ├── threat_intel.py # Shodan/VirusTotal/AbuseIPDB/ThreatFox enrichment
  695. │ │ └── schemas/ # Data-driven schema registry
  696. │ │ ├── ecs.py, sysmon.py, windows_security.py, auditd.py
  697. │ │ └── defs/ # JSON schema defs (zeek, packetbeat, aws_cloudtrail, …)
  698. │ │
  699. │ ├── tools/ # MCP tool interfaces (25 modules, 139 tools)
  700. │ │ ├── register.py # Dynamic tool registration
  701. │ │ ├── threat_hunting.py # Hunting tool definitions
  702. │ │ ├── rule_management.py # Rule management tools
  703. │ │ ├── velociraptor_tools.py # Live endpoint forensics (25 tools)
  704. │ │ ├── osquery_tools.py # Local host forensics (4 tools)
  705. │ │ ├── cross_correlation.py # SIEM ↔ endpoint correlation (5 tools)
  706. │ │ ├── chainsaw_hunting.py # Chainsaw tools
  707. │ │ ├── wireshark_tools.py # Network analysis tools
  708. │ │ ├── threat_intel.py # Enrichment + MISP tools
  709. │ │ └── investigation_state_tools.py # State management tools
  710. │ │
  711. │ ├── storage/ # Persistent investigation state
  712. │ │ ├── investigation_state.py # Core state management
  713. │ │ ├── storage_manager.py # File system storage (8GB FIFO)
  714. │ │ └── models.py # Pydantic models (IoC, Investigation)
  715. │ │
  716. │ ├── utils/ # Shared utilities
  717. │ │
  718. │ └── wireshark/ # Network traffic analysis
  719. │ ├── core/ # TShark execution, PCAP parsing
  720. │ ├── hunting/ # Beaconing, lateral movement, IoC hunting
  721. │ ├── baseline/ # Traffic baseline creation
  722. │ ├── extraction/ # File carving from traffic
  723. │ └── reporting/ # NCSC-style reports, timelines
  724. │
  725. ├── rules/ # 6,060 detection rules (EQL + Lucene)
  726. ├── chainsaw/ # Chainsaw binary + 3,000+ Sigma rules
  727. ├── skills/ # Claude Code agent skills
  728. ├── Makefile # Test pyramid + release automation
  729. └── tests/ # Tiered test suite (unit / integration / api / e2e / system)
  730. ```
  731. </details>
  732. <details>
  733. <summary><b>Design Patterns</b></summary>
  734. | Pattern | Usage |
  735. |:--------|:------|
  736. | **Multiple Inheritance** | `SearchClient` composes all specialised clients |
  737. | **Decorator** | Exception handling via `@handle_exceptions` |
  738. | **Factory** | `create_search_client()` creates appropriate client |
  739. | **Plugin Architecture** | Tools registered dynamically via `ToolsRegister` |
  740. | **Auto-Capture** | Tool results automatically analysed for IoCs |
  741. </details>
  742. <details>
  743. <summary><b>Adding a New Tool</b></summary>
  744. 1. **Create client method** in `src/clients/common/your_module.py`:
  745. ```python
  746. class YourClient(SearchClientBase):
  747. def your_method(self, param: str) -> dict:
  748. # Implementation
  749. return results
  750. ```
  751. 2. **Add to SearchClient** in `src/clients/common/client.py`:
  752. ```python
  753. class SearchClient(YourClient, OtherClients, ...):
  754. pass
  755. ```
  756. 3. **Create tool wrapper** in `src/tools/your_tools.py`:
  757. ```python
  758. class YourTools:
  759. def __init__(self, client, mcp):
  760. self.client = client
  761. self.mcp = mcp
  762. def register_tools(self):
  763. @self.mcp.tool()
  764. def your_tool(param: str) -> str:
  765. """Tool description for LLM."""
  766. result = self.client.your_method(param)
  767. return json.dumps(result)
  768. ```
  769. 4. **Register in server** in `src/server.py`:
  770. ```python
  771. from src.tools.your_tools import YourTools
  772. def _register_tools(self):
  773. # ... existing tools ...
  774. YourTools(self.client, self.mcp).register_tools()
  775. ```
  776. </details>
  777. <details>
  778. <summary><b>Running Tests</b></summary>
  779. The test suite is a bottom-up pyramid (see `tests/README.md`): hermetic unit tests, then live tiers that spin up a real Elasticsearch (via [testcontainers](https://testcontainers-python.readthedocs.io/)) and the real MCP server.
  780. ```bash
  781. make test-unit # fast hermetic loop (default developer command)
  782. make coverage # unit tier + coverage policy gate
  783. make check-tests # policy: every new function vs origin/main has a test
  784. make test-integration # real ES via testcontainers (Docker required)
  785. make test-api # + real MCP server subprocess over streamable-http
  786. make test-e2e # + real MCP client journey over the wire
  787. make test-suite # the whole pyramid, bottom-up, with a summary table
  788. ```
  789. | Tier | What it proves | Backing services |
  790. |:-----|:---------------|:-----------------|
  791. | Unit | Function-level behaviour, fail-fast validation | None (hermetic) |
  792. | Integration | App ↔ Elasticsearch trust boundary (auth, TLS, seeded data) | Real ES via testcontainers |
  793. | API | MCP-over-HTTP contract (schemas, errors, Origin security, audit logs) | Real ES + real server subprocess |
  794. | E2E | Full analyst journey via a real MCP client | Real ES + real server + `fastmcp.Client` |
  795. | System | Full pipeline against your own range cluster | BYO Elasticsearch |
  796. Live tiers are opt-in (`--run-integration` / `--run-api` / `--run-e2e` / `--run-system`), so plain `pytest tests/` stays fast and hermetic.
  797. </details>
  798. <details>
  799. <summary><b>Local Testing Environment</b></summary>
  800. ```bash
  801. # Start Elasticsearch
  802. docker-compose -f docker-compose-elasticsearch.yml up -d
  803. # Start OpenSearch
  804. docker-compose -f docker-compose-opensearch.yml up -d
  805. ```
  806. **Default credentials (testing only):**
  807. - Elasticsearch: `elastic` / `test123`
  808. - OpenSearch: `admin` / `admin`
  809. </details>
  810. ---
  811. ## Roadmap
  812. | Feature | Status | Description |
  813. |:--------|:------:|:------------|
  814. | **Velociraptor Integration** | **Done** | 25 live endpoint forensics tools + 5 cross-correlation tools via the Velociraptor API |
  815. | **osquery Integration** | **Done** | SQL-based local host forensics (processes, sockets, users, ad-hoc queries) |
  816. | **Encoded-Command Deobfuscation** | **Done** | Recover IoCs hidden in base64/UTF-16LE/gzip/hex-encoded commands (optional chepy for the long tail) |
  817. | **Threat Intel Enrichment** | **Done** | IoC enrichment via Shodan InternetDB, VirusTotal, AbuseIPDB, ThreatFox + STIX 2.1 export |
  818. | **MISP Integration** | **Done** | Export investigation IoCs as MISP events, search MISP attributes (offline JSON export without a server) |
  819. | **Zeek Integration** | Partial | Zeek logs indexed into ES are hunted via the data-driven schema registry; native NSM integration planned |
  820. | **Wazuh Integration** | Planned | Open-source EDR/XDR — its indexer is OpenSearch-based, so CrowdSentinel's existing OpenSearch client can hunt Wazuh alerts and agent telemetry |
  821. | **Splunk Integration** | Planned | Add Splunk as a data source alongside Elasticsearch |
  822. | **Sigma Rule Converter** | Planned | Convert Sigma rules to native ES/Splunk queries |
  823. | **Case Management** | Planned | Export investigations to TheHive, JIRA |
  824. | **Custom Rule Builder** | Planned | Create detection rules via natural language |
  825. See [CHANGELOG.md](CHANGELOG.md) for detailed version history.
  826. ---
  827. ## Velociraptor Integration (Live Endpoint Forensics)
  828. CrowdSentinel integrates with [Velociraptor](https://docs.velociraptor.app/) for live endpoint forensic artefact collection. When configured, 25 additional MCP tools and 5 cross-correlation tools become available.
  829. ### Setup
  830. ```bash
  831. # Install optional dependencies
  832. pip install crowdsentinel-mcp-server[velociraptor]
  833. # Set the Velociraptor API config path
  834. export VELOCIRAPTOR_API_CONFIG="/path/to/api_client.yaml"
  835. ```
  836. The API client config is generated from the Velociraptor server:
  837. ```bash
  838. velociraptor config api_client --name crowdsentinel --role administrator,api \
  839. --config server.config.yaml /path/to/api_client.yaml
  840. ```
  841. ### Available Tools
  842. | Category | Tools | Use Case |
  843. |:---------|:------|:---------|
  844. | **Discovery** | `velociraptor_client_info`, `velociraptor_list_clients`, `velociraptor_list_artifacts` | Resolve hostnames, enumerate enrolled endpoints, discover available artefacts |
  845. | **Generic Collection** | `velociraptor_collect_artifact`, `velociraptor_get_collection_results` | Collect any Velociraptor artefact and fetch its results |
  846. | **Live State** | `velociraptor_pslist`, `velociraptor_netstat`, `velociraptor_users`, `velociraptor_groups`, `velociraptor_mounts` | Running processes, active network connections, accounts, groups, mounted volumes |
  847. | **Execution Evidence** | `velociraptor_prefetch`, `velociraptor_amcache`, `velociraptor_shimcache`, `velociraptor_userassist`, `velociraptor_bam`, `velociraptor_srum` | Forensic proof of programme execution and resource usage |
  848. | **Persistence** | `velociraptor_services`, `velociraptor_scheduled_tasks`, `velociraptor_autoruns` | Service, scheduled task, and autorun persistence mechanisms |
  849. | **User Activity** | `velociraptor_shellbags`, `velociraptor_recentdocs`, `velociraptor_lnk`, `velociraptor_evidence_of_download` | User browsing, document access, link files, file downloads |
  850. | **Filesystem & Logs** | `velociraptor_ntfs_mft`, `velociraptor_evtx` | MFT search by name/path/timestamp, remote EVTX queries |
  851. | **Cross-Correlation** | `correlate_siem_with_endpoint`, `endpoint_to_siem_pivot`, `endpoint_hunt_from_ioc`, `correlate_by_time`, `build_unified_timeline` | Validate SIEM findings on endpoints, pivot from endpoint IoCs to fleet-wide SIEM searches, unified timelines |
  852. ### Cross-Source IoC Sharing
  853. IoCs extracted from Velociraptor artefacts are automatically captured into the active investigation alongside SIEM, Chainsaw, and Wireshark findings. Use `get_shared_iocs` to retrieve the combined indicator set for cross-source correlation.
  854. ```
  855. SIEM (Elasticsearch) Endpoint (Velociraptor)
  856. │ │
  857. hunt_by_timeframe() velociraptor_pslist()
  858. │ │
  859. auto_capture ──────► Investigation State ◄────── auto_capture
  860. │
  861. get_shared_iocs()
  862. ```
  863. ### DFIR Knowledge Resources
  864. 9 MCP resources provide structured investigation reference data directly to connected AI agents:
  865. | Resource | Content |
  866. |:---------|:--------|
  867. | `crowdsentinel://investigation-workflow` | Mandatory 4-phase IR workflow |
  868. | `crowdsentinel://ioc-reference` | IoC types ranked by Pyramid of Pain |
  869. | `crowdsentinel://ioc-reference/data` | Machine-readable IoC reference data |
  870. | `crowdsentinel://cross-correlation-playbooks` | 5 investigation playbooks (suspicious process, brute force, lateral movement, persistence, exfiltration) |
  871. | `crowdsentinel://velociraptor-guide` | Artefact reference tables and "found X in SIEM → check Y on endpoint" decision tree |
  872. | `crowdsentinel://data-sources` | All data source capabilities and investigation decision matrix |
  873. | `crowdsentinel://tool-recommendations` | Which tool to reach for at each investigation step |
  874. | `crowdsentinel://schemas` | Supported log-source schemas (ECS, Sysmon, Windows Security, auditd, Zeek, …) |
  875. | `crowdsentinel://schemas/list` | Machine-readable schema listing |
  876. ---
  877. <a name="documentation"></a>
  878. ## Documentation
  879. ### User Guides
  880. | Document | Description |
  881. |:---------|:------------|
  882. | [FIRST_TIME_SETUP.md](FIRST_TIME_SETUP.md) | Detailed first-time setup instructions |
  883. | [HOW_TO_USE.md](HOW_TO_USE.md) | Comprehensive usage guide |
  884. | [QUICK_START.md](QUICK_START.md) | 5-minute quick start |
  885. | [TRANSPORT_MODES.md](TRANSPORT_MODES.md) | stdio, SSE, HTTP configuration |
  886. ### Feature Guides
  887. | Document | Description |
  888. |:---------|:------------|
  889. | [THREAT_HUNTING_GUIDE.md](THREAT_HUNTING_GUIDE.md) | Threat hunting workflows |
  890. | [DETECTION_RULES_GUIDE.md](DETECTION_RULES_GUIDE.md) | Using 6,060 detection rules |
  891. | [CYBER_KILL_CHAIN_GUIDE.md](CYBER_KILL_CHAIN_GUIDE.md) | Kill chain analysis |
  892. | [CHAINSAW_GUIDE.md](CHAINSAW_GUIDE.md) | EVTX log analysis with Sigma |
  893. | [INVESTIGATION_PROMPTS_GUIDE.md](INVESTIGATION_PROMPTS_GUIDE.md) | Fast triage spine |
  894. | [AI_AGENT_INTEGRATION.md](AI_AGENT_INTEGRATION.md) | Workflow guidance for AI agents |
  895. ### Developer Guides
  896. | Document | Description |
  897. |:---------|:------------|
  898. | [ARCHITECTURE.md](ARCHITECTURE.md) | Detailed architecture documentation |
  899. | [CONTRIBUTING.md](CONTRIBUTING.md) | Contribution guidelines |
  900. ---
  901. ## Contributing
  902. Contributions are welcome! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
  903. ---
  904. ## Licence
  905. GNU General Public Licence v3.0 — See [LICENSE](LICENSE) for details.
  906. ---
  907. ## Cyber Range
  908. CrowdSentinel ships with a companion **[crowdsentinel-range](https://github.com/thomasxm/crowdsentinel-range)** — a fully ephemeral AWS lab that generates ground-truth-labeled attack telemetry so you can investigate real detections rather than synthetic data.
  909. ### Current range (v1) — 2-host AWS lab
  910. ```
  911. ┌──────────────────────────── AWS VPC (private subnet) ────────────────────────────────┐
  912. │ │
  913. │ ┌──────────────────────────────┐ ┌──────────────────────────────────────────┐ │
  914. │ │ Linux SIEM (Ubuntu 22.04) │ │ Windows Range Box (Server 2022) │ │
  915. │ │ Elasticsearch 8.x (TLS) │◄─────│ Sysmon (olafhartong) + Winlogbeat 9 │ │
  916. │ │ Auditbeat + Filebeat │ │ Atomic Red Team + PurpleSharp │ │
  917. │ └──────────────────────────────┘ └──────────────────────────────────────────┘ │
  918. │ SSM VPC endpoints — no inbound ports open │
  919. └───────────────────────────────────────┬────────────────────────────────────────────--┘
  920. │ SSM port-forward :19200
  921. ┌──────────▼──────────────────────┐
  922. │ ./range-mcp.sh → CrowdSentinel │
  923. │ MCP Server → Claude Code │
  924. └─────────────────────────────────┘
  925. ```
  926. **3 commands to go from zero to hunting:**
  927. ```bash
  928. cd crowdsentinel-range && make up && make attack-windows
  929. cd .. && ./range-mcp.sh
  930. # then in Claude Code: /mcp → "investigate potential intrusions on the windows machine"
  931. ```
  932. > Demo: [73,328 events triaged in ~40 seconds](https://github.com/thomasxm/crowdsentinel-range/blob/main/docs/demo-windows-triage-investigation.md) — 2 shellcode payloads decoded, C2 IP extracted from raw bytes, full IR report with 19 MITRE techniques.
  933. ---
  934. ### Future range (v2) — enterprise AD environment
  935. The next range generation targets **full Active Directory kill-chain coverage**: initial access → lateral movement → domain compromise, with a network security monitor layer, runtime security instrumentation, centralised vulnerability management, and deception infrastructure. Inspired by [DetectionLab](https://github.com/clong/DetectionLab), [Ludus](https://docs.ludus.cloud/), and [SimuLand](https://github.com/Azure/SimuLand).
  936. ```
  937. ┌───────────────────────────────────── AWS / Multi-VPC ───────────────────────────────────────────────────┐
  938. │ │
  939. │ ┌──── Attacker Subnet ────┐ ┌───────────── Corporate Subnet ──────────────────────────────────────┐ │
  940. │ │ │ │ │ │
  941. │ │ Kali Linux (C2 server) │ │ ┌─────────────────┐ ┌──────────────────┐ ┌──────────────────┐ │ │
  942. │ │ Caldera + Metasploit │ │ │ Domain │ │ Windows 11 │ │ Windows 11 │ │ │
  943. │ │ Covenant C2 │──▶│ │ Controller │ │ Workstation 1 │ │ Workstation 2 │ │ │
  944. │ │ │ │ │ (AD DS, DNS) │ │ (domain-joined) │ │ (domain-joined) │ │ │
  945. │ └──────────────────────────┘ │ └─────────────────┘ └──────────────────┘ └──────────────────┘ │ │
  946. │ │ ┌──────────────────────────────────────────────────────────────┐ │ │
  947. │ ┌──── DMZ Subnet ─────────┐ │ │ Linux File/Web Server (IIS / Apache, SMB share) │ │ │
  948. │ │ │ │ └──────────────────────────────────────────────────────────────┘ │ │
  949. │ │ Nginx reverse proxy │ └───────────────────────────────────────────────────────────────────┘ │
  950. │ │ Simulated internet SVC │ │
  951. │ └──────────────────────────┘ ┌───────────── Management / SIEM Subnet ──────────────────────────┐ │
  952. │ │ │ │
  953. │ │ ┌─────────────────────┐ ┌───────────────────────────────┐ │ │
  954. │ │ │ Elasticsearch + │ │ Zeek / Suricata │ │ │
  955. │ │ │ Kibana (SIEM) │ │ (NSM — full packet + alerts) │ │ │
  956. │ │ └─────────────────────┘ └───────────────────────────────┘ │ │
  957. │ │ ┌─────────────────────┐ ┌───────────────────────────────┐ │ │
  958. │ │ │ Velociraptor │ │ Falco (runtime security) │ │ │
  959. │ │ │ (live EDR/forensics)│ │ eBPF probe (CO-RE, default) │ │ │
  960. │ │ └─────────────────────┘ │ + kernel module fallback │ │ │
  961. │ │ ┌─────────────────────┐ └───────────────────────────────┘ │ │
  962. │ │ │ DefectDojo │ ┌───────────────────────────────┐ │ │
  963. │ │ │ (vuln management) │ │ Honeypots + Canary tokens │ │ │
  964. │ │ │ Falco·Zeek·scanners │ │ (deception layer) │ │ │
  965. │ │ └─────────────────────┘ └───────────────────────────────┘ │ │
  966. │ │ ┌─────────────────────────────────────────────────────────┐ │ │
  967. │ │ │ Grafana — centralized dashboards + alerting │ │ │
  968. │ │ │ ES · Falco · Suricata · Velociraptor · DefectDojo │ │ │
  969. │ │ └─────────────────────────────────────────────────────────┘ │ │
  970. │ └───────────────────────────────────────────────────────────────┘ │
  971. │ VyOS router — inter-subnet ACLs + egress filter │
  972. └───────────────────────────────────────────────────────────────────────────────────────────────────────┘
  973. │ SSM / VPN
  974. CrowdSentinel MCP + Claude Code
  975. ```
  976. **v2 target capabilities:**
  977. | Capability | v1 (current) | v2 (planned) |
  978. |------------|-------------|--------------|
  979. | Hosts | 2 (Linux SIEM + Windows) | 7–10 (DC + 2× WS + Linux + NSM + attacker + SIEM) |
  980. | Network segments | 1 (private subnet) | 4 (attacker / corp / DMZ / management) |
  981. | Active Directory | — | Full AD DS domain + GPO + Kerberos |
  982. | Lateral movement coverage | Limited | Pass-the-Hash, Kerberoasting, DCSync |
  983. | Network detection | — | Zeek (NSM) + Suricata IDS |
  984. | Runtime security | — | Falco; modern eBPF probe (CO-RE, bundled, default) + kernel module fallback; syscall-level host + container detection |
  985. | Endpoint forensics | — | Velociraptor live triage |
  986. | Vuln management | — | DefectDojo; aggregates Falco alerts, Zeek/Suricata findings, scanner results (OpenVAS, Trivy) into unified tracking + remediation workflow |
  987. | Dashboards | — | Grafana; centralised metrics + alert visualisation across ES, Falco, Suricata, Velociraptor, and DefectDojo |
  988. | Deception layer | — | Honeypot + canary tokens |
  989. | C2 infrastructure | ART atomics only | Caldera + Metasploit + Covenant |
  990. | Attack scenarios | Windows atomics + EVTX replay | Full kill-chain: initial access → domain compromise |
  991. | CrowdSentinel coverage | ES hunt + IR | ES + Zeek/Suricata + Falco + Velociraptor cross-source correlation; DefectDojo/Grafana as upstream context |
  992. **Design references:** [DetectionLab](https://github.com/clong/DetectionLab) (Sysmon+WEF+Splunk convergence), [Ludus](https://docs.ludus.cloud/) (modular Proxmox/Ansible, VLAN segmentation), [SimuLand](https://github.com/Azure/SimuLand) (multi-domain AD forest, cloud-native), [SANS FOR508](https://www.sans.org/cyber-security-courses/advanced-incident-response-threat-hunting-training/) (artifact-first DFIR workflow), [Falco](https://falco.org/docs/concepts/falco-kernel-testing/) (CO-RE eBPF probe architecture, kernel module fallback).
  993. ## Acknowledgements
  994. CrowdSentinel stands on the shoulders of outstanding open-source projects. Thank you to all of their maintainers and contributors.
  995. **Protocol & server framework**
  996. - [Model Context Protocol](https://modelcontextprotocol.io/) — the protocol connecting LLMs to tools, by Anthropic
  997. - [FastMCP](https://github.com/jlowin/fastmcp) — the Python MCP server framework this project is built on
  998. **Analysis engines & forensic tools**
  999. - [Chainsaw](https://github.com/WithSecureLabs/chainsaw) — EVTX log analyser by WithSecure Labs
  1000. - [Velociraptor](https://docs.velociraptor.app/) — live endpoint forensics and DFIR platform by Velocidex/Rapid7
  1001. - [osquery](https://osquery.io/) — SQL-powered host instrumentation by the osquery Foundation
  1002. - [Wireshark / TShark](https://www.wireshark.org/) — network protocol analysis by the Wireshark Foundation
  1003. - [Chepy](https://github.com/securisec/chepy) — CyberChef-like decoding library powering optional long-tail deobfuscation
  1004. - [sqlglot](https://github.com/tobymao/sqlglot) — SQL parser used to safely validate osquery queries
  1005. **Detection content**
  1006. - [Sigma / SigmaHQ](https://github.com/SigmaHQ/sigma) — the community detection rule standard and rule corpus
  1007. - [Elastic detection-rules](https://github.com/elastic/detection-rules) — Elastic's prebuilt EQL/ES|QL detection rules
  1008. - Community-contributed Sigma and custom rules
  1009. **Data platforms & threat intelligence**
  1010. - [Elasticsearch](https://github.com/elastic/elasticsearch-py) and [OpenSearch](https://github.com/opensearch-project/opensearch-py) Python clients
  1011. - [MISP / PyMISP](https://github.com/MISP/PyMISP) — open-source threat intelligence platform and client
  1012. - [stix2](https://github.com/oasis-open/cti-python-stix2) — OASIS STIX 2.1 library for structured IoC export
  1013. - [Shodan InternetDB](https://internetdb.shodan.io/), [VirusTotal](https://www.virustotal.com/), [AbuseIPDB](https://www.abuseipdb.com/), and [abuse.ch ThreatFox](https://threatfox.abuse.ch/) — IoC enrichment services with free tiers
  1014. **Development & testing**
  1015. - [testcontainers-python](https://github.com/testcontainers/testcontainers-python) — real Elasticsearch containers for the live test tiers
  1016. - [pytest](https://pytest.org/), [Ruff](https://github.com/astral-sh/ruff), [Bandit](https://github.com/PyCQA/bandit) — test runner, linting, and SAST
  1017. **Methodology frameworks**
  1018. - Cyber Kill Chain (Lockheed Martin), Pyramid of Pain (David J. Bianco), Diamond Model (Caltagirone, Pendergast & Betz), [MITRE ATT&CK](https://attack.mitre.org/)
  1019. ---
  1020. <div align="center">
  1021. **Made for the security community by [medjedtxm](https://github.com/thomasxm)**
  1022. [![GitHub](https://img.shields.io/badge/GitHub-CrowdSentinel-181717?style=for-the-badge&logo=github)](https://github.com/thomasxm/CrowdSentinels-AI-MCP)
  1023. [![PyPI](https://img.shields.io/badge/PyPI-crowdsentinel--mcp--server-3775A9?style=for-the-badge&logo=pypi&logoColor=white)](https://pypi.org/project/crowdsentinel-mcp-server/)
  1024. </div>

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.