Skip to content
Back to search

SecurityClaw

A modular, skill-based autonomous Security Operations Center (SOC) agent that monitors OpenSearch/Elasticsearch data, builds RAG-based behavioral memory, and validates real-time anomalies using LLMs.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/securityclaw--securityclaw/badge.svg)](/listings/securityclaw--securityclaw)

HTML

<a href="/listings/securityclaw--securityclaw"><img src="/api/listings/securityclaw--securityclaw/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

These steps run a script from the network that findsafeskills does not scan.

0. Prerequisites

- **Python 3.11+** (check with `python --version`) - **Git** (for cloning the repo) - **OpenSearch 2.x or Elasticsearch 8.x** (or use mock for testing) - **Ollama** (for LLM provider) - **4GB+ RAM** (recommended for Ollama models; 8GB+ for production) - **~2GB disk space** for models and vector indices

0.5 Quick Ollama Setup

The current example configuration in [config.yaml.example](config.yaml.example) uses:

- `qwen2.5:7b-instruct-q4_K_M` for chat/reasoning - `nomic-embed-text:latest` as the lightweight local auxiliary model for embeddings referenced by the sample config

Quick setup:

curl -fsSL https://ollama.com/install.sh | sh
ollama serve
ollama pull qwen2.5:7b-instruct-q4_K_M
ollama pull nomic-embed-text:latest

1. Create Virtual Environment & Install Dependencies

**Step 1a**: Clone the repository

git clone https://github.com/SecurityClaw/SecurityClaw.git
cd SecurityClaw

**Step 1b**: Create a Python virtual environment

# Using venv (built-in)
python3.11 -m venv .venv

# Or using virtualenv (if installed)
virtualenv .venv

**Step 1c**: Activate the virtual environment

# On Linux/macOS
source .venv/bin/activate

# On Windows (PowerShell)
.venv\Scripts\Activate.ps1

# On Windows (Command Prompt)
.venv\Scripts\activate.bat

**Step 1d**: Install Python dependencies

pip install -r requirements.txt

# Or using Pipenv (if you prefer):
pipenv install --dev

**Verify installation**:

python -c "import main; import core; print('✓ Dependencies OK')"

2. Interactive Onboarding

.venv/bin/python main.py onboard

The wizard will guide you through: - **Database**: Host, port, SSL, auth - **LLM**: Ollama configuration - **Connection testing** for both services - **Configuration save** to `config.yaml` and `.env` - **External APIs** (optional): AbuseIPDB, AlienVault OTX, VirusTotal, Talos, MaxMind - **Skill variables**: Auto-discover and prompt for any missing skill-specific env vars

See [ONBOARDING.md](ONBOARDING.md) for detailed walkthrough.

3. Start the Service (Recommended)

.venv/bin/python main.py service

Launches both the background scheduler **and** the web API server: - **Web UI**: `http://localhost:5173` (React frontend with hot reload) - **API**: `http://localhost:7799` (FastAPI REST service) - **Scheduler**: Runs anomaly detection and memory building in background

For API-only mode (no scheduler):

SECURITYCLAW_API_ONLY=1 .venv/bin/python main.py service

3a. Or Run CLI Commands

For pure CLI/background agent operation without the web interface:

.venv/bin/python main.py run                    # Start scheduler loop (anomaly watcher + memory builder)
.venv/bin/python main.py dispatch <skill>      # Fire a skill once (e.g., threat_analyst)
.venv/bin/python main.py chat                  # Interactive terminal-based chat with routing
.venv/bin/python main.py status                # Print compact agent memory snapshot
.venv/bin/python main.py list-skills           # Show loaded skills and intervals

3b. Web Development (Frontend Only)

If you want to develop the React frontend locally:

.venv/bin/python main.py web-dev               # Start Vite dev server on :5173
# In a second terminal:
.venv/bin/python main.py service               # Start backend API on :7799

4. Web Interface Features

The React web UI provides: - **Chat Interface** — Type questions; see real-time reasoning steps with skill invocations - **Skills Dispatch** — Manually trigger skills and view results - **Configuration Editor** — Edit `config.yaml` and `.env` through the UI - **Cron/Interval Management** — View and modify skill schedules - **Conversation History** — Persist and recall previous Q&A sessions - **Status Dashboard** — Real-time scheduler status, skill inventory, memory state

---

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 814 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. # SecurityClaw — Autonomous SOC Agentic Framework
  2. <!-- ALL-CONTRIBUTORS-BADGE:START - Do not remove or modify this section -->
  3. [![All Contributors](https://img.shields.io/badge/all_contributors-2-orange.svg?style=flat-square)](#contributors-)
  4. <!-- ALL-CONTRIBUTORS-BADGE:END -->
  5. [![DeepWiki](https://img.shields.io/badge/DeepWiki-SecurityClaw-blue?style=flat-square)](https://deepwiki.com/SecurityClaw/SecurityClaw)
  6. A modular, skill-based autonomous Security Operations Center (SOC) agent that monitors OpenSearch/Elasticsearch data, builds RAG-based behavioral memory, and validates real-time anomalies using LLMs.
  7. ## Portable launchers
  8. SecurityClaw includes versioned launchers for local and minimal-container deployments. They start services only when requested and keep the managed OpenSearch container on `restart=no`.
  9. Linux, macOS, WSL, or another POSIX-compatible shell:
  10. ```bash
  11. ./securityclaw.sh -h
  12. ./securityclaw.sh start # local .venv application
  13. ./securityclaw.sh docker # Docker Compose application
  14. ./securityclaw.sh status
  15. ./securityclaw.sh stop
  16. ```
  17. Native Windows PowerShell:
  18. ```powershell
  19. .\securityclaw.ps1 -h
  20. .\securityclaw.ps1 start # local Windows .venv application
  21. .\securityclaw.ps1 docker # Docker Compose application
  22. .\securityclaw.ps1 status
  23. .\securityclaw.ps1 stop
  24. ```
  25. Both launchers verify configuration, Ollama, Docker, and OpenSearch. They also expose `chat`, `skills`, and `logs` commands. Use `OLLAMA_BASE_URL` for a remote model endpoint. The Bash launcher works on Windows through WSL or Git Bash; use the PowerShell launcher for native Windows execution.
  26. ## Features
  27. * **Skill Modularity** — Capabilities as isolated folders with `logic.py` (Python) + `instruction.md` (LLM guidance)
  28. * **Heartbeat Loop** — Cron-like scheduler: 1-minute anomaly watcher, 6-hour memory builder
  29. * **Provider Agnostic** — Swap OpenSearch↔Elasticsearch via config
  30. * **RAG-Based Memory** — Vector embeddings stored in OpenSearch; context-aware threat analysis
  31. * **LangGraph Orchestration** — Multi-step DECIDE→EXECUTE→EVALUATE supervisor loop implemented as a `StateGraph`; conversation and chat working memory checkpointed to SQLite via `SqliteSaver`
  32. * **Manifest-Grounded Planning** — Supervisor planning and retry steps are repaired against the currently loaded skill manifests before execution, so prerequisite chains come from manifest contracts instead of invented tool names
  33. * **Working Memory** — Interactive chat working memory stays inside LangGraph state and is checkpointed in `data/conversations.db`; the scheduler/CLI runtime now uses the same checkpoint-backed model via `data/runtime_memory.db`
  34. * **Conversation-based Investigations** — Investigate threats through an interactive chat interface with real-time LLM reasoning steps, manifest-declared capability contracts, and RAG context retrieval
  35. * **Web Interface** — Modern React-based UI for chat, memory visualization, and skill dispatch
  36. Context budgeting notes:
  37. - Chat output budget defaults to `llm.max_tokens: 16384` in [config.yaml](config.yaml).
  38. - Working memory injected into prompts is compacted by [core/memory.py](core/memory.py) with a default `max_context_chars` budget of 4000 characters.
  39. - Supervisor result summaries are clipped before prompt injection, so there is some compaction already, but there is not yet live token-usage telemetry in the chat router.
  40. ---
  41. ## Web Interface
  42. <img src="img/web-screenshot.png" alt="SecurityClaw Web Interface" width="600" height="600">
  43. ---
  44. ## Quick Start
  45. ### 0. Prerequisites
  46. - **Python 3.11+** (check with `python --version`)
  47. - **Git** (for cloning the repo)
  48. - **OpenSearch 2.x or Elasticsearch 8.x** (or use mock for testing)
  49. - **Ollama** (for LLM provider)
  50. - **4GB+ RAM** (recommended for Ollama models; 8GB+ for production)
  51. - **~2GB disk space** for models and vector indices
  52. ### 0.5 Quick Ollama Setup
  53. The current example configuration in [config.yaml.example](config.yaml.example) uses:
  54. - `qwen2.5:7b-instruct-q4_K_M` for chat/reasoning
  55. - `nomic-embed-text:latest` as the lightweight local auxiliary model for embeddings referenced by the sample config
  56. Quick setup:
  57. ```bash
  58. curl -fsSL https://ollama.com/install.sh | sh
  59. ollama serve
  60. ollama pull qwen2.5:7b-instruct-q4_K_M
  61. ollama pull nomic-embed-text:latest
  62. ```
  63. ### 1. Create Virtual Environment & Install Dependencies
  64. **Step 1a**: Clone the repository
  65. ```bash
  66. git clone https://github.com/SecurityClaw/SecurityClaw.git
  67. cd SecurityClaw
  68. ```
  69. **Step 1b**: Create a Python virtual environment
  70. ```bash
  71. # Using venv (built-in)
  72. python3.11 -m venv .venv
  73. # Or using virtualenv (if installed)
  74. virtualenv .venv
  75. ```
  76. **Step 1c**: Activate the virtual environment
  77. ```bash
  78. # On Linux/macOS
  79. source .venv/bin/activate
  80. # On Windows (PowerShell)
  81. .venv\Scripts\Activate.ps1
  82. # On Windows (Command Prompt)
  83. .venv\Scripts\activate.bat
  84. ```
  85. **Step 1d**: Install Python dependencies
  86. ```bash
  87. pip install -r requirements.txt
  88. # Or using Pipenv (if you prefer):
  89. pipenv install --dev
  90. ```
  91. **Verify installation**:
  92. ```bash
  93. python -c "import main; import core; print('✓ Dependencies OK')"
  94. ```
  95. ### 2. Interactive Onboarding
  96. ```bash
  97. .venv/bin/python main.py onboard
  98. ```
  99. The wizard will guide you through:
  100. - **Database**: Host, port, SSL, auth
  101. - **LLM**: Ollama configuration
  102. - **Connection testing** for both services
  103. - **Configuration save** to `config.yaml` and `.env`
  104. - **External APIs** (optional): AbuseIPDB, AlienVault OTX, VirusTotal, Talos, MaxMind
  105. - **Skill variables**: Auto-discover and prompt for any missing skill-specific env vars
  106. See [ONBOARDING.md](ONBOARDING.md) for detailed walkthrough.
  107. ### 3. Start the Service (Recommended)
  108. ```bash
  109. .venv/bin/python main.py service
  110. ```
  111. Launches both the background scheduler **and** the web API server:
  112. - **Web UI**: `http://localhost:5173` (React frontend with hot reload)
  113. - **API**: `http://localhost:7799` (FastAPI REST service)
  114. - **Scheduler**: Runs anomaly detection and memory building in background
  115. For API-only mode (no scheduler):
  116. ```bash
  117. SECURITYCLAW_API_ONLY=1 .venv/bin/python main.py service
  118. ```
  119. ### 3a. Or Run CLI Commands
  120. For pure CLI/background agent operation without the web interface:
  121. ```bash
  122. .venv/bin/python main.py run # Start scheduler loop (anomaly watcher + memory builder)
  123. .venv/bin/python main.py dispatch <skill> # Fire a skill once (e.g., threat_analyst)
  124. .venv/bin/python main.py chat # Interactive terminal-based chat with routing
  125. .venv/bin/python main.py status # Print compact agent memory snapshot
  126. .venv/bin/python main.py list-skills # Show loaded skills and intervals
  127. ```
  128. ### 3b. Web Development (Frontend Only)
  129. If you want to develop the React frontend locally:
  130. ```bash
  131. .venv/bin/python main.py web-dev # Start Vite dev server on :5173
  132. # In a second terminal:
  133. .venv/bin/python main.py service # Start backend API on :7799
  134. ```
  135. ### 4. Web Interface Features
  136. The React web UI provides:
  137. - **Chat Interface** — Type questions; see real-time reasoning steps with skill invocations
  138. - **Skills Dispatch** — Manually trigger skills and view results
  139. - **Configuration Editor** — Edit `config.yaml` and `.env` through the UI
  140. - **Cron/Interval Management** — View and modify skill schedules
  141. - **Conversation History** — Persist and recall previous Q&A sessions
  142. - **Status Dashboard** — Real-time scheduler status, skill inventory, memory state
  143. ---
  144. ## Architecture
  145. SecurityClaw's chat orchestration is moving toward a capability-first contract model:
  146. - The supervisor plans against manifest-declared capabilities, prerequisites, required entities, and artifacts.
  147. - Live routing favors LLM planning plus manifest viability checks instead of manifest keyword guards.
  148. - Follow-up grounding should be expressed through manifest hooks so the core router stays skill-agnostic.
  149. ### Directory Structure
  150. ```
  151. SecurityClaw/
  152. ├── config.yaml # Central DB/LLM/RAG configuration
  153. ├── .env # Secrets (master credentials)
  154. ├── main.py # CLI entrypoint
  155. │
  156. ├── core/
  157. │ ├── config.py # YAML + env loader
  158. │ ├── memory.py # Structured memory helpers for file/state/checkpoint-backed runtimes
  159. │ ├── runner.py # Conductor (skill discovery, scheduling)
  160. │ ├── scheduler.py # APScheduler wrapper
  161. │ ├── skill_loader.py # Dynamic skill discovery
  162. │ ├── db_connector.py # OpenSearch/ES abstraction
  163. │ ├── llm_provider.py # Ollama provider
  164. │ ├── rag_engine.py # Embedding store & retrieval
  165. │ └── chat_router/ # API-only: LangGraph StateGraph orchestrator
  166. │
  167. ├── skills/
  168. │ ├── network_baseliner/ # 6h: Aggregate logs → RAG vectors
  169. │ ├── fields_baseliner/ # 1h: Catalog field schemas and aggregate example values into fields RAG
  170. │ ├── anomaly_triage/ # Manual: Poll AD findings → enrich → escalate
  171. │ ├── threat_analyst/ # Manual: RAG reasoning → verdict
  172. │ ├── opensearch_querier/ # Manual: Execute database queries
  173. │ ├── forensic_examiner/ # Manual: Build incident timelines
  174. │ ├── baseline_querier/ # Manual: Search behavioral baselines
  175. │ ├── fields_querier/ # Manual: Query field schema catalog
  176. │ └── geoip_lookup/ # Cron (Tue/Fri 2 AM UTC): Maintain MaxMind DB
  177. │
  178. ├── data/
  179. │ ├── conversations.db # SQLite — LangGraph checkpoint store (conversation + chat memory)
  180. │ ├── runtime_memory.db # SQLite — LangGraph checkpoint store (scheduler + CLI runtime memory)
  181. │ └── geoip/ # MaxMind GeoLite2 database files
  182. │
  183. ├── web/
  184. │ ├── api/
  185. │ │ ├── server.py # FastAPI REST service with SSE streaming
  186. │ │ └── service.py # SecurityClawService lifecycle (scheduler + cleanup)
  187. │ ├── src/ # React frontend source
  188. │ ├── dist/ # Built frontend (generated by `web-build`)
  189. │ ├── package.json # Frontend dependencies
  190. │ └── vite.config.js # Vite bundler config
  191. │
  192. ├── tests/
  193. │ ├── conftest.py # Shared fixtures
  194. │ ├── mock_opensearch.py # In-memory DB (cosine kNN)
  195. │ ├── mock_llm.py # Deterministic LLM (keyword-dispatched)
  196. │ ├── data_generator.py # Synthetic network logs & anomalies
  197. │ └── test_*.py # Offline tests + coverage
  198. │
  199. ├── requirements.txt / Pipfile # Dependencies
  200. └── ONBOARDING.md # Interactive setup guide
  201. ```
  202. ### Core Design Principles
  203. | Principle | Implementation |
  204. |-----------|---|
  205. | **Skill Modularity** | Each skill is a folder with `logic.py` (entrypoint) and `instruction.md` (LLM system prompt) |
  206. | **Auto-Discovery** | Runner scans `/skills` and dynamically loads all valid skills |
  207. | **LangGraph Orchestration** | `chat_router` runs a `StateGraph` (DECIDE→EXECUTE→EVALUATE loop) compiled with `SqliteSaver`; state includes chat memory, skill results, and conversation history |
  208. | **Stateful Memory** | Chat orchestration uses LangGraph state checkpointed at `data/conversations.db`; the scheduler and CLI runtime use the same bounded structured memory model checkpointed at `data/runtime_memory.db` |
  209. | **Scheduled Execution** | APScheduler fires skills at intervals; intervals defined in skill `instruction.md` front-matter |
  210. | **Provider Agnostic** | Abstract `BaseDBConnector` and `BaseLLMProvider` allow swapping vendors via config |
  211. | **RAG Context** | Embeddings stored in vector index; retrieved during LLM analysis for behavioral context |
  212. | **Testability** | Mock DB, LLM, and data generators enable repeatable offline tests with coverage reporting |
  213. ---
  214. ## Skill Reference
  215. ### NetworkBaseliner (6-hour cycle)
  216. **Purpose**: Build baseline of "normal" network behavior.
  217. **Logic**:
  218. 1. Query recent logs (e.g., last 24 hours)
  219. 2. Aggregate into summaries (typical ports, protocols, byte volumes)
  220. 3. Generate LLM-enhanced descriptions
  221. 4. Store as embedding vectors in the RAG index
  222. **Output**: Baseline vectors used by ThreatAnalyst for context.
  223. ### AnomalyTriage (Manual)
  224. **Purpose**: Poll anomaly detection findings and escalate high-confidence anomalies.
  225. **Publication note**: This skill is in active validation. Convert to scheduled by adding `schedule_interval_seconds: 60` to `instruction.md`.
  226. **Logic**:
  227. 1. Query OpenSearch AD index for new findings (cursor-based, from last poll)
  228. 2. Enrich each finding with LLM description (entity, score, severity)
  229. 3. If severity ≥ threshold: write to the escalation queue in agent memory
  230. 4. Update cursor for next poll
  231. **Output**: Escalated findings in memory, waiting for ThreatAnalyst analysis.
  232. ### ThreatAnalyst (Manual)
  233. **Purpose**: Analyze escalated findings using RAG context; issue verdict.
  234. **Publication note**: This skill is in active validation. Convert to scheduled by adding `schedule_interval_seconds: 300` to `instruction.md`.
  235. **Logic**:
  236. 1. Read the escalation queue from agent memory
  237. 2. For each finding:
  238. - Query RAG engine for similar baseline context
  239. - Build LLM prompt with finding + baseline context
  240. - Request verdict (TRUE_THREAT, FALSE_POSITIVE, UNKNOWN, ERROR)
  241. 3. Write verdicts and actions to "Recent Decisions"
  242. 4. If TRUE_THREAT: set "Current Focus" and trigger IR playbooks
  243. **Output**: Verdicts with confidence, MITRE tactic mapping, recommended actions.
  244. ### GeoIPLookup (weekly refresh + on-demand lookup)
  245. **Purpose**: Maintain a local MaxMind GeoLite2-City database and answer direct IP geolocation questions.
  246. **Logic**:
  247. 1. On first use, download the MMDB if missing
  248. 2. Once per week, refresh it if stale
  249. 3. For a supplied IP, return local city / subdivision / country / timezone / coordinate data
  250. **Output**: Deterministic geolocation fields from the local MaxMind DB.
  251. ### Publication Status Notes
  252. | Skill | Status | Notes |
  253. |-------|--------|-------|
  254. | **chat_router** | Stable | Powers web interface and API |
  255. | **network_baseliner** | Stable | Builds behavioral baselines from logs |
  256. | **fields_baseliner** | Stable | Catalogs OpenSearch field schemas and aggregated example values |
  257. | **anomaly_triage** | In-Progress | Manual skill; enable scheduling in instruction.md |
  258. | **threat_analyst** | In-Progress | Manual skill; enable scheduling in instruction.md |
  259. | **opensearch_querier** | Stable | Single point of contact for DB queries |
  260. | **forensic_examiner** | In-Progress | Timeline reconstruction; active development |
  261. | **baseline_querier** | In-Progress | Search behavioral baselines; not publication-hardened |
  262. | **fields_querier** | Stable | Search field schema catalog |
  263. | **geoip_lookup** | Stable | MaxMind GeoLite2 maintenance and lookups |
  264. **Legend**:
  265. - **Stable**: Publication-ready; tested in production patterns
  266. - **In-Progress**: Under active validation; feedback welcome
  267. - **Deprecated**: Use alternative skill; kept for backwards compatibility
  268. ---
  269. ## Configuration
  270. ### config.yaml
  271. ```yaml
  272. agent:
  273. name: SecurityClaw
  274. version: "1.0.0"
  275. skills_dir: skills
  276. log_level: INFO
  277. scheduler:
  278. heartbeat_interval_seconds: 60
  279. memory_build_interval_hours: 6
  280. db:
  281. provider: opensearch # or: elasticsearch
  282. host: localhost
  283. port: 9200
  284. use_ssl: false
  285. verify_certs: false
  286. username: "" # Loaded from .env
  287. password: "" # Loaded from .env
  288. # Index configuration (configured during onboarding)
  289. logs_index: securityclaw-logs # Where to scan for network logs
  290. anomaly_index: securityclaw-anomalies # Where AD findings are stored
  291. vector_index: securityclaw-vectors # RAG embedding store
  292. llm:
  293. # ollama, openai, chatgpt, openai_compatible, anthropic, claude_api,
  294. # codex_cli, or claude_cli
  295. provider: ollama
  296. ollama_base_url: http://localhost:11434
  297. ollama_model: qwen2.5:7b-instruct-q4_K_M
  298. ollama_embed_model: nomic-embed-text:latest
  299. rag:
  300. embedding_model: all-MiniLM-L6-v2
  301. top_k: 5
  302. similarity_threshold: 0.65
  303. anomaly:
  304. detector_id: default-detector
  305. poll_interval_seconds: 60
  306. severity_threshold: 0.7
  307. max_findings_per_poll: 50
  308. geoip:
  309. enabled: true
  310. provider: auto # Prefer MaxMind; fall back to IPinfo Lite
  311. db_path: data/geoip/GeoLite2-City.mmdb
  312. edition_id: GeoLite2-City
  313. update_interval_days: 7
  314. download_url: https://download.maxmind.com/app/geoip_download
  315. timeout_seconds: 60
  316. license_key: "" # Loaded from .env via MAXMIND_LICENSE_KEY
  317. ipinfo_token: "" # Loaded from .env via IPINFO_TOKEN
  318. ipinfo_url: https://api.ipinfo.io/lite
  319. ```
  320. Configure either `MAXMIND_LICENSE_KEY` for city-level local lookups or
  321. `IPINFO_TOKEN` for country, continent, and ASN enrichment. IPinfo Lite does not
  322. provide city, coordinates, postal code, or timezone fields.
  323. ### Index Configuration Explained
  324. SecurityClaw works with **three indices**:
  325. | Index | Purpose | Used By | Example |
  326. |-------|---------|---------|---------|
  327. | **logs_index** | Historical network logs for baseline building | NetworkBaseliner (6h cycle) | `securityclaw-logs`, `logs-*`, `filebeat-*` |
  328. | **anomaly_index** | Anomaly Detection results (findings) | AnomalyWatcher (1m cycle) | `securityclaw-anomalies`, `.opendistro-anomaly-results*` |
  329. | **vector_index** | RAG embeddings (normal behavior baseline) | ThreatAnalyst (5m cycle) | `securityclaw-vectors` |
  330. **Flow:**
  331. 1. **NetworkBaseliner** → queries `logs_index` → generates summaries → stores embeddings in `vector_index`
  332. 2. **AnomalyWatcher** → polls `anomaly_index` for new findings → escalates to memory
  333. 3. **ThreatAnalyst** → reads escalations → retrieves context from `vector_index` → issues verdict
  334. During onboarding, you can use any index names/patterns your environment provides (e.g., if your logs are in `filebeat-networking-*`, use that instead of `securityclaw-logs`).
  335. ### .env (git-ignored)
  336. ```
  337. OPENSEARCH_USERNAME=<your-opensearch-username>
  338. OPENSEARCH_PASSWORD=<your-opensearch-password>
  339. OLLAMA_BASE_URL=http://localhost:11434
  340. # Select one provider and define only the corresponding values:
  341. LLM_PROVIDER=ollama
  342. OPENAI_API_KEY=
  343. OPENAI_BASE_URL=https://api.openai.com/v1
  344. OPENAI_MODEL=gpt-5
  345. ANTHROPIC_API_KEY=
  346. ANTHROPIC_BASE_URL=https://api.anthropic.com/v1
  347. ANTHROPIC_MODEL=claude-sonnet-4-5
  348. CODEX_CLI_PATH=codex
  349. CLAUDE_CLI_PATH=claude
  350. ```
  351. `openai` and `chatgpt` use the OpenAI API; `openai_compatible` accepts any
  352. Chat Completions-compatible base URL. `anthropic` and `claude_api` use the
  353. Anthropic Messages API. `codex_cli` and `claude_cli` reuse an already
  354. authenticated local CLI. Remote and CLI chat providers continue to use the
  355. configured Ollama embedding model for local RAG, so embeddings remain separate
  356. from chat generation.
  357. ---
  358. ## CLI Commands
  359. ```bash
  360. # Interactive setup
  361. .venv/bin/python main.py onboard
  362. # Start web interface + backend API + scheduler
  363. .venv/bin/python main.py service
  364. # Run the CLI agent (blocks; press Ctrl+C to stop)
  365. .venv/bin/python main.py run
  366. # Interactive chat in CLI
  367. .venv/bin/python main.py chat
  368. # Fire one skill immediately
  369. .venv/bin/python main.py dispatch anomaly_triage
  370. .venv/bin/python main.py dispatch network_baseliner
  371. .venv/bin/python main.py dispatch threat_analyst
  372. # View working memory
  373. .venv/bin/python main.py status
  374. # List skills and intervals
  375. .venv/bin/python main.py list-skills
  376. # Set logging level
  377. .venv/bin/python main.py --log-level DEBUG run
  378. ```
  379. ---
  380. ## Web Interface
  381. ### Starting the Web Server
  382. The web interface provides a modern chat-based UI for interacting with SecurityClaw skills and viewing reasoning steps.
  383. ```bash
  384. # Activate virtual environment first
  385. source .venv/bin/activate # or: .venv\Scripts\activate on Windows
  386. # Start the web server + backend API + scheduler
  387. python main.py service
  388. ```
  389. **Expected output**:
  390. ```
  391. [INFO] Starting web server and API...
  392. [INFO] Frontend available at: http://localhost:3000
  393. [INFO] API backend available at: http://localhost:5000/api
  394. [INFO] Scheduler running in background
  395. [INFO] Press Ctrl+C to stop
  396. ```
  397. ### Web Interface Features
  398. **Chat Interface** (http://localhost:3000)
  399. - **Real-time chat**: Send questions and receive answers from SecurityClaw skills
  400. - **Reasoning steps**: View LLM reasoning and decision-making process
  401. - **Conversation history**: Browse and manage past conversations
  402. - **Skill dispatch**: Manually trigger skills (anomaly_triage, threat_analyst, etc.)
  403. - **Memory view**: Monitor the agent's working memory and findings
  404. **API Endpoints** (http://localhost:5000/api)
  405. - `GET /api/status` — Agent status and memory summary
  406. - `POST /api/chat/stream` — Stream chat responses
  407. - `GET /api/conversations` — List conversation history
  408. - `DELETE /api/conversations/{id}` — Delete a conversation
  409. - `GET /api/skills` — List available skills
  410. - `POST /api/skills/{name}/dispatch` — Manually trigger a skill
  411. - `GET /api/config` (read-only) — View masked configuration
  412. ### Accessing the Web UI
  413. **Local access** (single machine):
  414. - Open your browser to: **http://localhost:3000**
  415. **Remote access** (from another machine):
  416. 1. Find the server's IP address:
  417. ```bash
  418. hostname -I # Linux
  419. ipconfig # Windows
  420. ifconfig # macOS
  421. ```
  422. 2. Access from remote machine (replace `SERVER_IP` with actual IP):
  423. ```
  424. http://SERVER_IP:3000
  425. ```
  426. - Ensure firewall allows port 3000/5000 traffic
  427. - For production, use HTTPS and set up reverse proxy (nginx/Apache)
  428. ### Example: Running Headless (Server Only)
  429. If you only want the API without opening a browser:
  430. ```bash
  431. # Start server
  432. python main.py service &
  433. # In another terminal, query the API
  434. curl http://localhost:5000/api/status
  435. # Or use the CLI in the main terminal
  436. python main.py dispatch threat_analyst
  437. ```
  438. ### Troubleshooting Web Server
  439. **"Port 3000 already in use"**
  440. ```bash
  441. # Kill the process using port 3000
  442. lsof -ti:3000 | xargs kill -9 # Linux/macOS
  443. netstat -ano | findstr :3000 # Windows
  444. ```
  445. **"Cannot connect to API"**
  446. - Verify backend is running: `curl http://localhost:5000/api/status`
  447. - Check firewall rules
  448. - Look for errors in the main terminal
  449. **"Chat not responding"**
  450. - Check LLM availability (Ollama running?)
  451. - View logs: `python main.py --log-level DEBUG service`
  452. - Check config.yaml for correct provider settings
  453. ---
  454. ## Testing
  455. All tests are offline by default (mock DB + mock LLM) and now emit coverage reports via `pytest-cov`.
  456. ```bash
  457. # Run the full suite with coverage
  458. .venv/bin/python -m pytest
  459. # Run a specific test file
  460. .venv/bin/python -m pytest tests/test_rag.py -v
  461. # Optional HTML coverage report
  462. .venv/bin/python -m pytest --cov-report=html
  463. ```
  464. Coverage XML is written to [coverage.xml](coverage.xml) for CI/reporting.
  465. Current publication-prep baseline: the full suite is measured automatically, but aggregate coverage is still dragged down by in-progress modules and provider-specific adapters. Treat the report as a measurement tool, not as a claim that every skill is publication-hardened.
  466. ### What's Tested
  467. | Layer | Tests | Notes |
  468. |-------|-------|-------|
  469. | **Config** | (via conftest) | YAML + env loading |
  470. | **Scheduler** | 13 | Job registration, dispatch, intervals, cron expressions |
  471. | **DB Abstraction** | 20 | Search, kNN, anomaly findings, bulk indexing |
  472. | **LLM Abstraction** | 11 | Embedding, chat, canned responses |
  473. | **RAG Engine** | 15 | Store, retrieve, context building, category filters |
  474. | **Skill Loader** | 14 | Discovery, instruction loading, interval parsing |
  475. | **Skills** | active coverage | Stable orchestration paths are covered; in-progress skills remain under active validation |
  476. | **Data Generator** | 24 | Synthetic logs, anomalies, baseline chunks, embeddings |
  477. Redundant supervisor routing tests were consolidated to keep the publication suite smaller and easier to maintain.
  478. ---
  479. ## Writing a New Skill
  480. ### Anatomy of a Skill
  481. ```
  482. skills/my_skill/
  483. ├── logic.py # Python
  484. └── instruction.md # LLM guidance
  485. ```
  486. **logic.py**:
  487. ```python
  488. """
  489. skills/my_skill/logic.py
  490. Context dict keys:
  491. - db → BaseDBConnector
  492. - llm → BaseLLMProvider
  493. - memory → StateBackedMemory (in-memory) or CheckpointBackedMemory (SQLite-backed)
  494. - config → Config
  495. - skills → dict of loaded Skill objects
  496. """
  497. from pathlib import Path
  498. SKILL_NAME = "my_skill"
  499. INSTRUCTION_PATH = Path(__file__).parent / "instruction.md"
  500. def run(context: dict) -> dict:
  501. """
  502. Main entry point. Called by Runner on schedule.
  503. Return a dict with status, results, etc.
  504. """
  505. db = context.get("db")
  506. llm = context.get("llm")
  507. memory = context.get("memory")
  508. config = context.get("config")
  509. # Your logic here
  510. memory.add_finding("Found something interesting")
  511. return {
  512. "status": "ok",
  513. "findings": 5,
  514. }
  515. ```
  516. **instruction.md**:
  517. ```markdown
  518. ---
  519. schedule_interval_seconds: 300
  520. ---
  521. # My Skill
  522. You are a security analyst specializing in [X].
  523. When given anomalies, your job is to:
  524. 1. [Step 1]
  525. 2. [Step 2]
  526. Respond in JSON format with:
  527. ```json
  528. {
  529. "verdict": "...",
  530. "confidence": ...,
  531. "reasoning": "..."
  532. }
  533. ```
  534. ```
  535. ---
  536. ## Extending SecurityClaw
  537. ### Add a New Skill
  538. 1. Create `skills/my_skill/` directory
  539. 2. Write `logic.py` with `run(context)` function
  540. 3. Write `instruction.md` with LLM guidance and optional `schedule_interval_seconds`
  541. 4. Restart agent or run `.venv/bin/python main.py dispatch my_skill` to test
  542. ### Add a DB Backend
  543. 1. Subclass `BaseDBConnector` in `core/db_connector.py`
  544. 2. Set `db.provider: my_db` in `config.yaml`
  545. 3. Update `build_db_connector()` factory to instantiate your class
  546. ### Add an LLM Backend
  547. 1. Subclass `BaseLLMProvider` in `core/llm_provider.py`
  548. 2. Set `llm.provider: my_llm` in `config.yaml`
  549. 3. Update `build_llm_provider()` factory to instantiate your class
  550. ---
  551. ## Troubleshooting
  552. **"Module 'X' not found"**
  553. ```bash
  554. .venv/bin/pip install -r requirements.txt
  555. ```
  556. **"Cannot connect to OpenSearch"**
  557. - Verify OpenSearch is running: `curl -u admin:admin http://localhost:9200`
  558. - Check config.yaml host/port
  559. - Check firewall rules
  560. **"Cannot connect to Ollama"**
  561. - Start Ollama: `ollama serve`
  562. - Pull the sample models: `ollama pull qwen2.5:7b-instruct-q4_K_M && ollama pull nomic-embed-text:latest`
  563. - Verify base URL in config.yaml
  564. **"Skill not loading"**
  565. - Check `/skills/<name>/logic.py` exists
  566. - Verify `run(context)` function signature
  567. - Check logs: `.venv/bin/python main.py --log-level DEBUG run`
  568. **"No findings detected"**
  569. - Seed mock DB: See `tests/conftest.py` for example synthetic data
  570. - Check anomaly indices: `curl http://localhost:9200/_cat/indices?v`
  571. - Verify detector ID in config.yaml
  572. ---
  573. ## Performance Notes
  574. - **LLM Calls**: Each anomaly watcher and threat analyst cycle calls the LLM 1+ times (Ollama: ~1s per call)
  575. - **RAG Retrieval**: kNN search is O(n) in mock; ~1ms per query on seeded DB
  576. - **Scheduler**: Background APScheduler has minimal overhead (~1% CPU idle)
  577. ---
  578. ## Contributing
  579. Contributions welcome! Areas for enhancement:
  580. - [ ] Elasticsearch compatibility testing
  581. - [ ] anomaly_triage, forensic_examiner currently underperforming or not working depending on the query patterns; active validation and refactor needed
  582. - [ ] Advanced MITRE ATT&CK mapping
  583. - [ ] Enhanced RAG context (e.g., no hard-coded country codes, instead parse through RAG)
  584. - [ ] Multi-tenant support
  585. ---
  586. ## Support
  587. For issues, questions, or feature requests, open an issue or contact the SecurityClaw team.
  588. ---
  589. ## Security / Publication Checklist
  590. - `config.yaml`, `.env`, `data/conversations.db`, and `data/runtime_memory.db` are intended to stay local.
  591. - Use [config.yaml.example](config.yaml.example) as the public template.
  592. - Run a quick scan before publishing:
  593. git grep -nEI '(password|api[_-]?key|BEGIN [A-Z ]*PRIVATE KEY|sk-)' -- .
  594. git log --all -G 'password|api[_-]?key|sk-' --oneline
  595. ## Contributors ✨
  596. Thanks goes to these wonderful people ([emoji key](https://allcontributors.org/docs/en/emoji-key)):
  597. <!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section -->
  598. <!-- prettier-ignore-start -->
  599. <!-- markdownlint-disable -->
  600. <table>
  601. <tbody>
  602. <tr>
  603. <td align="center" valign="top" width="14.28%"><a href="http://michael.tsikerdekis.com/"><img src="https://avatars.githubusercontent.com/u/232134?v=4?s=100" width="100px;" alt="Michael Tsikerdekis"/><br /><sub><b>Michael Tsikerdekis</b></sub></a><br /><a href="https://github.com/tsikerdekis/SecurityClaw/commits?author=tsikerdekis" title="Code">💻</a> <a href="https://github.com/tsikerdekis/SecurityClaw/commits?author=tsikerdekis" title="Documentation">📖</a> <a href="#design-tsikerdekis" title="Design">🎨</a> <a href="#ideas-tsikerdekis" title="Ideas, Planning, & Feedback">🤔</a> <a href="#infra-tsikerdekis" title="Infrastructure (Hosting, Build-Tools, etc)">🚇</a> <a href="https://github.com/tsikerdekis/SecurityClaw/commits?author=tsikerdekis" title="Tests">⚠️</a> <a href="#maintenance-tsikerdekis" title="Maintenance">🚧</a></td>
  604. <td align="center" valign="top" width="14.28%"><a href="https://kroryandev.com/"><img src="https://avatars.githubusercontent.com/u/69607760?v=4?s=100" width="100px;" alt="KroX"/><br /><sub><b>KroX</b></sub></a><br /><a href="https://github.com/tsikerdekis/SecurityClaw/commits?author=kroryan" title="Code">💻</a> <a href="https://github.com/tsikerdekis/SecurityClaw/commits?author=kroryan" title="Documentation">📖</a> <a href="#ideas-kroryan" title="Ideas, Planning, & Feedback">🤔</a></td>
  605. </tr>
  606. </tbody>
  607. <tfoot>
  608. <tr>
  609. <td align="center" size="13px" colspan="7">
  610. <img src="https://raw.githubusercontent.com/all-contributors/all-contributors-cli/1b8533af435da9854653492b1327a23a4dbd0a10/assets/logo-small.svg">
  611. <a href="https://all-contributors.js.org/docs/en/bot/usage">Add your contributions</a>
  612. </img>
  613. </td>
  614. </tr>
  615. </tfoot>
  616. </table>
  617. <!-- markdownlint-restore -->
  618. <!-- prettier-ignore-end -->
  619. <!-- ALL-CONTRIBUTORS-LIST:END -->
  620. This project follows the [all-contributors](https://github.com/all-contributors/all-contributors) specification. Contributions of any kind welcome!

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.