linux-perf-doctor
Diagnose Linux CPU saturation, memory leaks, OOM kills, disk I/O bottlenecks, syscall storms, thread contention, socket backlogs, and cgroup v1/v2 container throttling using real OS telemetry. Use when users report slow Linux servers, high load averages, container throttling, high iowait, OOM events, or process performance regressions.
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Badge for your README
Markdown
[](/listings/anujtiwari-student--linux-perf-doctor)HTML
<a href="/listings/anujtiwari-student--linux-perf-doctor"><img src="/api/listings/anujtiwari-student--linux-perf-doctor/badge.svg" alt="findsafeskills safety rating" /></a>Install
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
Install directly from GitHub (Recommended)
Install the skill from its public repository:
npx skills add AnujTiwari-Student/linux-perf-doctor --skill linux-perf-doctor
To install the published v1.0.0 release specifically:
npx skills add AnujTiwari-Student/linux-perf-doctor@v1.0.0 --skill linux-perf-doctor
Install from a Local Clone
Clone the repository and install from its root directory:
# Download the source from GitHub (Code > Download ZIP) and extract it cd <extracted-directory> npx skills add . --skill linux-perf-doctor
Manual Installation
Run these commands from the parent directory containing `linux-perf-doctor/`.
For Claude Code:
mkdir -p ~/.claude/skills cp -r ./linux-perf-doctor ~/.claude/skills/linux-perf-doctor
For OpenAI Codex:
mkdir -p ~/.agents/skills cp -r ./linux-perf-doctor ~/.agents/skills/linux-perf-doctor
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
SKILL.md · Manifest · 89 lines
- ---
- name: linux-perf-doctor
- description: Diagnose Linux CPU saturation, memory leaks, OOM kills, disk I/O bottlenecks, syscall storms, thread contention, socket backlogs, and cgroup v1/v2 container throttling using real OS telemetry. Use when users report slow Linux servers, high load averages, container throttling, high iowait, OOM events, or process performance regressions.
- license: MIT
- compatibility: Linux hosts and containers with Python 3.8+ and standard /proc and /sys filesystems. Optional tools include perf, strace, ss, iostat, vmstat, pidstat, and bpftool.
- metadata:
- version: "1.0.1"
- category: systems-performance
- ---
- # linux-perf-doctor
- Autonomous eBPF, syscall, process, I/O, and container bottleneck profiler that diagnoses Linux performance problems using operating-system telemetry rather than speculative source-code guesses.
- ## When to Use
- - Diagnose high CPU usage, load average spikes, run-queue saturation, or involuntary context switching.
- - Investigate high memory utilization, RSS growth, swap thrashing, or kernel Out-Of-Memory (OOM) kills.
- - Identify disk I/O bottlenecks, elevated `iowait`, block device saturation, or high per-process read/write amplification.
- - Inspect container resource constraints, cgroup v1 and cgroup v2 CPU throttling (`nr_throttled`), memory high/max events, and Pressure Stall Information (PSI).
- - Analyze network socket backlogs (`Recv-Q`, `Send-Q`), TCP retransmissions, `TIME_WAIT` accumulation, or `SYN_RECV` pressure.
- - Perform opt-in active process profiling (`perf`, `strace`, or eBPF tools) when passive `/proc` telemetry indicates syscall storms or user-space hot loops.
- ## Safety and Operating Principles
- 1. Default to read-only passive telemetry collection (`/proc`, `/sys`, cgroup filesystems, `ss`, `vmstat`, `iostat`).
- 2. Never invoke privilege-escalation wrappers automatically or escalate privileges without user execution.
- 3. Keep active profiling (`scripts/profile_process.sh` using `perf` or `strace`) strictly opt-in via `--enable-active-profiling` because attaching tracers can add latency or pause target threads.
- 4. Bound all sampling windows with `--interval` and `--duration` limits (default 1 second, max 60 seconds).
- 5. Separate measured facts (directly observed counters and ratios) from diagnostic hypotheses (inferred application-level root causes).
- ## Diagnostic Workflow
- ### Step 1: Verify Host and Profiler Capabilities
- Run the capability detector first to see which tools (`perf`, `strace`, `ss`, `iostat`, `vmstat`, `pidstat`, `bpftool`), kernel interfaces (`/proc/pressure`, cgroup v2), and permission levels (`perf_event_paranoid`, `ptrace_scope`) are available:
- ```bash
- bash scripts/check_perf_capabilities.sh --pretty
- ```
- ### Step 2: Collect System, Cgroup, I/O, and Network Baselines
- Collect passive read-only telemetry across the host or container:
- ```bash
- python3 scripts/collect_system_metrics.py --interval 0.2 --samples 2 --pretty
- python3 scripts/inspect_cgroups.py --cgroup-root /sys/fs/cgroup --pretty
- bash scripts/inspect_io.sh --pretty
- python3 scripts/inspect_network.py --pretty
- ```
- ### Step 3: Inspect Target Process Metrics
- When a specific process (`PID`) exhibits anomalous CPU, memory, file descriptor, or context-switch behavior, inspect its `/proc/<pid>` telemetry:
- ```bash
- python3 scripts/collect_process_metrics.py --pid <PID> --interval 0.2 --pretty
- ```
- ### Step 4: Execute Unified Bottleneck Analysis and Report Generation
- Synthesize all subsystem collectors into a unified diagnostic report with evidence-backed findings, confidence scores, and remediation steps:
- ```bash
- python3 scripts/generate_report.py --pid <PID> --format markdown --output report.md
- python3 scripts/generate_report.py --pid <PID> --format json --output report.json
- ```
- ### Step 5: Opt-In Active Process Profiling (Only When Necessary)
- If passive telemetry shows high system CPU (`sy`) or high voluntary/involuntary context switches and the user explicitly consents to active profiling, run `scripts/profile_process.sh`:
- ```bash
- bash scripts/profile_process.sh --pid <PID> --mode strace-summary --duration 3 --enable-active-profiling
- bash scripts/profile_process.sh --pid <PID> --mode perf-stat --duration 3 --enable-active-profiling
- ```
- ## Interpreting Findings
- - **CPU Throttling vs Host Saturation**: If cgroup `cpu.stat` shows rising `nr_throttled` and `throttled_usec` while host idle CPU is above 30%, the container CPU quota (`cpu.max`) is undersized rather than the underlying node being saturated.
- - **System Time vs User Time**: System CPU (`sy`) above 25% accompanied by high `ctxt/s` points to syscall storms, lock contention, or small-buffer I/O. User CPU (`us`) above 80% with low `ctxt/s` points to compute-bound loops, serialization, or GC activity.
- - **Memory Pressure vs Leak**: Steady RSS growth across intervals without anonymous page reclamation indicates an application retention issue or leak. High major page faults (`majflt`) and PSI `memory` full stall percentages indicate active thrashing.
- ## Gotchas
- - Containerized environments often mount host `/proc/meminfo` while restricting memory via `/sys/fs/cgroup/memory.max`. Always run `scripts/inspect_cgroups.py` alongside `scripts/collect_system_metrics.py` to avoid misreading host RAM as container capacity.
- - Non-root users cannot read `/proc/<pid>/io` or attach `strace` when `kernel.yama.ptrace_scope` is `1` or higher unless granted `CAP_SYS_PTRACE`. Handle permission denials gracefully and rely on `/proc/<pid>/stat` and `/proc/<pid>/status`.
- - Refer to `references/technical-reference.md`, `references/diagnostic-workflows.md`, and `references/troubleshooting.md` for metric thresholds and kernel counter definitions.
README.md · README · 140 lines
- # linux-perf-doctor
- <p align="center">
- <img src="assets/branding/logo.svg" alt="linux-perf-doctor logo" width="180" />
- </p>
- **Read-only Linux Performance Diagnostics for AI Agents and SRE Workflows**
- ## 1. Project Name and Positioning
- `linux-perf-doctor` is a production-grade [Agent Skill](https://agentskills.io/specification) that equips AI coding and SRE agents with deterministic Linux operating-system telemetry collection, cgroup v1/v2 container pressure analysis, and evidence-backed bottleneck diagnosis.
- ## 2. Problem Statement
- When applications suffer latency spikes, CPU throttling, or OOM terminations in production or containerized staging environments, AI coding assistants frequently speculate about source-code algorithms without inspecting actual kernel counters. Furthermore, containerized workloads often misreport host `/proc/meminfo` and CPU core counts while silently suffering cgroup CFS bandwidth throttling (`nr_throttled`) or cgroup memory reclaim stalls.
- ## 3. What the Skill Actually Does
- `linux-perf-doctor` executes deterministic, read-only collectors against `/proc`, `/sys/fs/cgroup`, `/proc/pressure` (PSI), `/proc/diskstats`, and `/proc/net/tcp*`, with optional capability detection for `perf`, `strace`, `ss`, `iostat`, `vmstat`, `pidstat`, and eBPF utilities. It correlates findings across subsystems and distinguishes **measured facts** from **diagnostic hypotheses**.
- ## 4. Key Features
- - **Capability & Security Guard Detection**: Checks installed binaries (`perf`, `strace`, `ss`, `iostat`, `vmstat`, `bpftool`) and kernel security parameters (`perf_event_paranoid`, `yama.ptrace_scope`, `kptr_restrict`).
- - **System & PSI Telemetry**: Measures CPU user/system/idle/iowait/steal deltas, context switch rates, `/proc/vmstat` OOM/fault counters, and Linux Pressure Stall Information (`cpu`, `memory`, `io`).
- - **Container & Cgroup v1/v2 Analyzer**: Detects CFS quota throttling (`nr_throttled`, `throttled_usec`), `memory.current` vs `memory.max`, and `memory.events` (`oom_kill`, `high` reclaim stalls).
- - **Targeted Process Inspector**: Analyzes per-PID CPU ticks, RSS/Swap, major page faults, voluntary vs involuntary context switches, open file descriptors, and `/proc/<pid>/io`.
- - **Network & Block I/O Diagnostics**: Identifies `CLOSE_WAIT` socket leaks, `TIME_WAIT` churn, `SYN_RECV` backlog saturation, non-zero `Recv-Q`/`Send-Q` buffers, and high block-device `await` latencies.
- - **Strictly Opt-In Active Profiling**: Requires `--enable-active-profiling` before invoking `strace` or `perf` with bounded duration timeouts.
- ### Project Demo
- Watch Linux Perf Doctor in action:
- [**Watch the demonstration on LinkedIn →**](https://lnkd.in/p/dRXQRwFk)
- ## 5. Architecture Overview
- See [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) for the complete four-stage pipeline:
- 1. `scripts/check_perf_capabilities.sh` -> Capability & permission discovery
- 2. `scripts/collect_system_metrics.py`, `scripts/collect_process_metrics.py`, `scripts/inspect_cgroups.py`, `scripts/inspect_io.sh`, `scripts/inspect_network.py` -> Passive telemetry collection
- 3. `scripts/profile_process.sh` -> Opt-in active profiling (`passive-summary`, `strace-summary`, `perf-stat`)
- 4. `scripts/generate_report.py` -> Cross-subsystem correlation & JSON/Markdown report synthesis
- ## 6. Installation
- ### Install directly from GitHub (Recommended)
- Install the skill from its public repository:
- ```bash
- npx skills add AnujTiwari-Student/linux-perf-doctor --skill linux-perf-doctor
- ```
- To install the published v1.0.0 release specifically:
- ```bash
- npx skills add AnujTiwari-Student/linux-perf-doctor@v1.0.0 --skill linux-perf-doctor
- ```
- ### Install from a Local Clone
- Clone the repository and install from its root directory:
- ```bash
- # Download the source from GitHub (Code > Download ZIP) and extract it
- cd <extracted-directory>
- npx skills add . --skill linux-perf-doctor
- ```
- ### Manual Installation
- Run these commands from the parent directory containing `linux-perf-doctor/`.
- For Claude Code:
- ```bash
- mkdir -p ~/.claude/skills
- cp -r ./linux-perf-doctor ~/.claude/skills/linux-perf-doctor
- ```
- For OpenAI Codex:
- ```bash
- mkdir -p ~/.agents/skills
- cp -r ./linux-perf-doctor ~/.agents/skills/linux-perf-doctor
- ```
- ## 7. Supported Agent Environments
- See [`docs/COMPATIBILITY.md`](docs/COMPATIBILITY.md) for detailed verification classifications across Claude Code, OpenAI Codex, Gemini CLI, Google Antigravity, OpenCode, and Hermes Agent.
- ## 8. Prerequisites
- - Linux kernel with `/proc` and `/sys` mounted (supports both host nodes and unprivileged containers).
- - Python 3.8+ (standard library only; zero external PyPI packages required).
- - GNU Bash 4.0+.
- - Optional CLI tools (`perf`, `strace`, `ss`, `iostat`, `vmstat`, `pidstat`, `bpftool`) are auto-detected and gracefully skipped if absent.
- ## 9. Usage Examples
- Ask your AI agent:
- - *"Use `linux-perf-doctor` to check why our container is experiencing tail latency spikes despite low node CPU."*
- - *"Inspect PID 4242 with `linux-perf-doctor` and generate a Markdown bottleneck report."*
- ## 10. Script Examples
- ```bash
- # 1. Check host capabilities and kernel guards
- bash scripts/check_perf_capabilities.sh --pretty
- # 2. Collect host system metrics across two 200ms intervals
- python3 scripts/collect_system_metrics.py --interval 0.2 --samples 2 --pretty
- # 3. Inspect container cgroup v1/v2 throttling and memory events
- python3 scripts/inspect_cgroups.py --cgroup-root /sys/fs/cgroup --pretty
- # 4. Generate a unified Markdown diagnostic report
- python3 scripts/generate_report.py --format markdown --output examples/live-host-report.md
- ```
- ## 11. Example Diagnostic Results
- See [`examples/sample-saturated-report.md`](examples/sample-saturated-report.md) and [`examples/sample-saturated-report.json`](examples/sample-saturated-report.json) for complete outputs generated from the saturated container/node fixture (`tests/fixtures/saturated_node`).
- ## 12. Security and Safety Approach
- - Read-only passive inspection by default.
- - Never invokes privilege-escalation wrappers or modifies kernel parameters.
- - Never reads `/proc/<pid>/environ` or prints secrets.
- - Active profilers (`strace`, `perf`) are blocked unless `--enable-active-profiling` is explicitly passed. See [`docs/SECURITY.md`](docs/SECURITY.md).
- ## 13. Limitations
- - On macOS or Windows hosts without a Linux `/proc` filesystem, live host collection returns a clean error unless pointed at a captured `/proc` snapshot directory via `--proc-root`.
- - Unprivileged users cannot read `/proc/<pid>/io` or `/proc/<pid>/fd` for processes owned by other UIDs; the collector logs a warning and continues with `/proc/<pid>/stat` and `/proc/<pid>/status`.
- ## 14. Testing Instructions
- ```bash
- python3 -m unittest discover -s tests -v
- ```
- ## 15. Contribution Instructions
- See [`CONTRIBUTING.md`](CONTRIBUTING.md) for code style, stdlib-only rules, and test requirements.
- ## 16. Roadmap
- - Add optional `bpftool` / `bpftrace` read-only map dump parsers when running with `CAP_BPF`.
- - Add `/proc/slabinfo` kernel slab cache pressure analysis.
- ## 17. License
- MIT License. See [`LICENSE`](LICENSE).
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.