Skip to content
Back to search

linux-perf-doctor

Diagnose Linux CPU saturation, memory leaks, OOM kills, disk I/O bottlenecks, syscall storms, thread contention, socket backlogs, and cgroup v1/v2 container throttling using real OS telemetry. Use when users report slow Linux servers, high load averages, container throttling, high iowait, OOM events, or process performance regressions.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/anujtiwari-student--linux-perf-doctor/badge.svg)](/listings/anujtiwari-student--linux-perf-doctor)

HTML

<a href="/listings/anujtiwari-student--linux-perf-doctor"><img src="/api/listings/anujtiwari-student--linux-perf-doctor/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Install directly from GitHub (Recommended)

Install the skill from its public repository:

npx skills add AnujTiwari-Student/linux-perf-doctor --skill linux-perf-doctor

To install the published v1.0.0 release specifically:

npx skills add AnujTiwari-Student/linux-perf-doctor@v1.0.0 --skill linux-perf-doctor

Install from a Local Clone

Clone the repository and install from its root directory:

# Download the source from GitHub (Code > Download ZIP) and extract it
cd <extracted-directory>
npx skills add . --skill linux-perf-doctor

Manual Installation

Run these commands from the parent directory containing `linux-perf-doctor/`.

For Claude Code:

mkdir -p ~/.claude/skills
cp -r ./linux-perf-doctor ~/.claude/skills/linux-perf-doctor

For OpenAI Codex:

mkdir -p ~/.agents/skills
cp -r ./linux-perf-doctor ~/.agents/skills/linux-perf-doctor

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

SKILL.md · Manifest · 89 lines
  1. ---
  2. name: linux-perf-doctor
  3. description: Diagnose Linux CPU saturation, memory leaks, OOM kills, disk I/O bottlenecks, syscall storms, thread contention, socket backlogs, and cgroup v1/v2 container throttling using real OS telemetry. Use when users report slow Linux servers, high load averages, container throttling, high iowait, OOM events, or process performance regressions.
  4. license: MIT
  5. compatibility: Linux hosts and containers with Python 3.8+ and standard /proc and /sys filesystems. Optional tools include perf, strace, ss, iostat, vmstat, pidstat, and bpftool.
  6. metadata:
  7. version: "1.0.1"
  8. category: systems-performance
  9. ---
  10. # linux-perf-doctor
  11. Autonomous eBPF, syscall, process, I/O, and container bottleneck profiler that diagnoses Linux performance problems using operating-system telemetry rather than speculative source-code guesses.
  12. ## When to Use
  13. - Diagnose high CPU usage, load average spikes, run-queue saturation, or involuntary context switching.
  14. - Investigate high memory utilization, RSS growth, swap thrashing, or kernel Out-Of-Memory (OOM) kills.
  15. - Identify disk I/O bottlenecks, elevated `iowait`, block device saturation, or high per-process read/write amplification.
  16. - Inspect container resource constraints, cgroup v1 and cgroup v2 CPU throttling (`nr_throttled`), memory high/max events, and Pressure Stall Information (PSI).
  17. - Analyze network socket backlogs (`Recv-Q`, `Send-Q`), TCP retransmissions, `TIME_WAIT` accumulation, or `SYN_RECV` pressure.
  18. - Perform opt-in active process profiling (`perf`, `strace`, or eBPF tools) when passive `/proc` telemetry indicates syscall storms or user-space hot loops.
  19. ## Safety and Operating Principles
  20. 1. Default to read-only passive telemetry collection (`/proc`, `/sys`, cgroup filesystems, `ss`, `vmstat`, `iostat`).
  21. 2. Never invoke privilege-escalation wrappers automatically or escalate privileges without user execution.
  22. 3. Keep active profiling (`scripts/profile_process.sh` using `perf` or `strace`) strictly opt-in via `--enable-active-profiling` because attaching tracers can add latency or pause target threads.
  23. 4. Bound all sampling windows with `--interval` and `--duration` limits (default 1 second, max 60 seconds).
  24. 5. Separate measured facts (directly observed counters and ratios) from diagnostic hypotheses (inferred application-level root causes).
  25. ## Diagnostic Workflow
  26. ### Step 1: Verify Host and Profiler Capabilities
  27. Run the capability detector first to see which tools (`perf`, `strace`, `ss`, `iostat`, `vmstat`, `pidstat`, `bpftool`), kernel interfaces (`/proc/pressure`, cgroup v2), and permission levels (`perf_event_paranoid`, `ptrace_scope`) are available:
  28. ```bash
  29. bash scripts/check_perf_capabilities.sh --pretty
  30. ```
  31. ### Step 2: Collect System, Cgroup, I/O, and Network Baselines
  32. Collect passive read-only telemetry across the host or container:
  33. ```bash
  34. python3 scripts/collect_system_metrics.py --interval 0.2 --samples 2 --pretty
  35. python3 scripts/inspect_cgroups.py --cgroup-root /sys/fs/cgroup --pretty
  36. bash scripts/inspect_io.sh --pretty
  37. python3 scripts/inspect_network.py --pretty
  38. ```
  39. ### Step 3: Inspect Target Process Metrics
  40. When a specific process (`PID`) exhibits anomalous CPU, memory, file descriptor, or context-switch behavior, inspect its `/proc/<pid>` telemetry:
  41. ```bash
  42. python3 scripts/collect_process_metrics.py --pid <PID> --interval 0.2 --pretty
  43. ```
  44. ### Step 4: Execute Unified Bottleneck Analysis and Report Generation
  45. Synthesize all subsystem collectors into a unified diagnostic report with evidence-backed findings, confidence scores, and remediation steps:
  46. ```bash
  47. python3 scripts/generate_report.py --pid <PID> --format markdown --output report.md
  48. python3 scripts/generate_report.py --pid <PID> --format json --output report.json
  49. ```
  50. ### Step 5: Opt-In Active Process Profiling (Only When Necessary)
  51. If passive telemetry shows high system CPU (`sy`) or high voluntary/involuntary context switches and the user explicitly consents to active profiling, run `scripts/profile_process.sh`:
  52. ```bash
  53. bash scripts/profile_process.sh --pid <PID> --mode strace-summary --duration 3 --enable-active-profiling
  54. bash scripts/profile_process.sh --pid <PID> --mode perf-stat --duration 3 --enable-active-profiling
  55. ```
  56. ## Interpreting Findings
  57. - **CPU Throttling vs Host Saturation**: If cgroup `cpu.stat` shows rising `nr_throttled` and `throttled_usec` while host idle CPU is above 30%, the container CPU quota (`cpu.max`) is undersized rather than the underlying node being saturated.
  58. - **System Time vs User Time**: System CPU (`sy`) above 25% accompanied by high `ctxt/s` points to syscall storms, lock contention, or small-buffer I/O. User CPU (`us`) above 80% with low `ctxt/s` points to compute-bound loops, serialization, or GC activity.
  59. - **Memory Pressure vs Leak**: Steady RSS growth across intervals without anonymous page reclamation indicates an application retention issue or leak. High major page faults (`majflt`) and PSI `memory` full stall percentages indicate active thrashing.
  60. ## Gotchas
  61. - Containerized environments often mount host `/proc/meminfo` while restricting memory via `/sys/fs/cgroup/memory.max`. Always run `scripts/inspect_cgroups.py` alongside `scripts/collect_system_metrics.py` to avoid misreading host RAM as container capacity.
  62. - Non-root users cannot read `/proc/<pid>/io` or attach `strace` when `kernel.yama.ptrace_scope` is `1` or higher unless granted `CAP_SYS_PTRACE`. Handle permission denials gracefully and rely on `/proc/<pid>/stat` and `/proc/<pid>/status`.
  63. - Refer to `references/technical-reference.md`, `references/diagnostic-workflows.md`, and `references/troubleshooting.md` for metric thresholds and kernel counter definitions.
README.md · README · 140 lines
  1. # linux-perf-doctor
  2. <p align="center">
  3. <img src="assets/branding/logo.svg" alt="linux-perf-doctor logo" width="180" />
  4. </p>
  5. **Read-only Linux Performance Diagnostics for AI Agents and SRE Workflows**
  6. ## 1. Project Name and Positioning
  7. `linux-perf-doctor` is a production-grade [Agent Skill](https://agentskills.io/specification) that equips AI coding and SRE agents with deterministic Linux operating-system telemetry collection, cgroup v1/v2 container pressure analysis, and evidence-backed bottleneck diagnosis.
  8. ## 2. Problem Statement
  9. When applications suffer latency spikes, CPU throttling, or OOM terminations in production or containerized staging environments, AI coding assistants frequently speculate about source-code algorithms without inspecting actual kernel counters. Furthermore, containerized workloads often misreport host `/proc/meminfo` and CPU core counts while silently suffering cgroup CFS bandwidth throttling (`nr_throttled`) or cgroup memory reclaim stalls.
  10. ## 3. What the Skill Actually Does
  11. `linux-perf-doctor` executes deterministic, read-only collectors against `/proc`, `/sys/fs/cgroup`, `/proc/pressure` (PSI), `/proc/diskstats`, and `/proc/net/tcp*`, with optional capability detection for `perf`, `strace`, `ss`, `iostat`, `vmstat`, `pidstat`, and eBPF utilities. It correlates findings across subsystems and distinguishes **measured facts** from **diagnostic hypotheses**.
  12. ## 4. Key Features
  13. - **Capability & Security Guard Detection**: Checks installed binaries (`perf`, `strace`, `ss`, `iostat`, `vmstat`, `bpftool`) and kernel security parameters (`perf_event_paranoid`, `yama.ptrace_scope`, `kptr_restrict`).
  14. - **System & PSI Telemetry**: Measures CPU user/system/idle/iowait/steal deltas, context switch rates, `/proc/vmstat` OOM/fault counters, and Linux Pressure Stall Information (`cpu`, `memory`, `io`).
  15. - **Container & Cgroup v1/v2 Analyzer**: Detects CFS quota throttling (`nr_throttled`, `throttled_usec`), `memory.current` vs `memory.max`, and `memory.events` (`oom_kill`, `high` reclaim stalls).
  16. - **Targeted Process Inspector**: Analyzes per-PID CPU ticks, RSS/Swap, major page faults, voluntary vs involuntary context switches, open file descriptors, and `/proc/<pid>/io`.
  17. - **Network & Block I/O Diagnostics**: Identifies `CLOSE_WAIT` socket leaks, `TIME_WAIT` churn, `SYN_RECV` backlog saturation, non-zero `Recv-Q`/`Send-Q` buffers, and high block-device `await` latencies.
  18. - **Strictly Opt-In Active Profiling**: Requires `--enable-active-profiling` before invoking `strace` or `perf` with bounded duration timeouts.
  19. ### Project Demo
  20. Watch Linux Perf Doctor in action:
  21. [**Watch the demonstration on LinkedIn →**](https://lnkd.in/p/dRXQRwFk)
  22. ## 5. Architecture Overview
  23. See [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) for the complete four-stage pipeline:
  24. 1. `scripts/check_perf_capabilities.sh` -> Capability & permission discovery
  25. 2. `scripts/collect_system_metrics.py`, `scripts/collect_process_metrics.py`, `scripts/inspect_cgroups.py`, `scripts/inspect_io.sh`, `scripts/inspect_network.py` -> Passive telemetry collection
  26. 3. `scripts/profile_process.sh` -> Opt-in active profiling (`passive-summary`, `strace-summary`, `perf-stat`)
  27. 4. `scripts/generate_report.py` -> Cross-subsystem correlation & JSON/Markdown report synthesis
  28. ## 6. Installation
  29. ### Install directly from GitHub (Recommended)
  30. Install the skill from its public repository:
  31. ```bash
  32. npx skills add AnujTiwari-Student/linux-perf-doctor --skill linux-perf-doctor
  33. ```
  34. To install the published v1.0.0 release specifically:
  35. ```bash
  36. npx skills add AnujTiwari-Student/linux-perf-doctor@v1.0.0 --skill linux-perf-doctor
  37. ```
  38. ### Install from a Local Clone
  39. Clone the repository and install from its root directory:
  40. ```bash
  41. # Download the source from GitHub (Code > Download ZIP) and extract it
  42. cd <extracted-directory>
  43. npx skills add . --skill linux-perf-doctor
  44. ```
  45. ### Manual Installation
  46. Run these commands from the parent directory containing `linux-perf-doctor/`.
  47. For Claude Code:
  48. ```bash
  49. mkdir -p ~/.claude/skills
  50. cp -r ./linux-perf-doctor ~/.claude/skills/linux-perf-doctor
  51. ```
  52. For OpenAI Codex:
  53. ```bash
  54. mkdir -p ~/.agents/skills
  55. cp -r ./linux-perf-doctor ~/.agents/skills/linux-perf-doctor
  56. ```
  57. ## 7. Supported Agent Environments
  58. See [`docs/COMPATIBILITY.md`](docs/COMPATIBILITY.md) for detailed verification classifications across Claude Code, OpenAI Codex, Gemini CLI, Google Antigravity, OpenCode, and Hermes Agent.
  59. ## 8. Prerequisites
  60. - Linux kernel with `/proc` and `/sys` mounted (supports both host nodes and unprivileged containers).
  61. - Python 3.8+ (standard library only; zero external PyPI packages required).
  62. - GNU Bash 4.0+.
  63. - Optional CLI tools (`perf`, `strace`, `ss`, `iostat`, `vmstat`, `pidstat`, `bpftool`) are auto-detected and gracefully skipped if absent.
  64. ## 9. Usage Examples
  65. Ask your AI agent:
  66. - *"Use `linux-perf-doctor` to check why our container is experiencing tail latency spikes despite low node CPU."*
  67. - *"Inspect PID 4242 with `linux-perf-doctor` and generate a Markdown bottleneck report."*
  68. ## 10. Script Examples
  69. ```bash
  70. # 1. Check host capabilities and kernel guards
  71. bash scripts/check_perf_capabilities.sh --pretty
  72. # 2. Collect host system metrics across two 200ms intervals
  73. python3 scripts/collect_system_metrics.py --interval 0.2 --samples 2 --pretty
  74. # 3. Inspect container cgroup v1/v2 throttling and memory events
  75. python3 scripts/inspect_cgroups.py --cgroup-root /sys/fs/cgroup --pretty
  76. # 4. Generate a unified Markdown diagnostic report
  77. python3 scripts/generate_report.py --format markdown --output examples/live-host-report.md
  78. ```
  79. ## 11. Example Diagnostic Results
  80. See [`examples/sample-saturated-report.md`](examples/sample-saturated-report.md) and [`examples/sample-saturated-report.json`](examples/sample-saturated-report.json) for complete outputs generated from the saturated container/node fixture (`tests/fixtures/saturated_node`).
  81. ## 12. Security and Safety Approach
  82. - Read-only passive inspection by default.
  83. - Never invokes privilege-escalation wrappers or modifies kernel parameters.
  84. - Never reads `/proc/<pid>/environ` or prints secrets.
  85. - Active profilers (`strace`, `perf`) are blocked unless `--enable-active-profiling` is explicitly passed. See [`docs/SECURITY.md`](docs/SECURITY.md).
  86. ## 13. Limitations
  87. - On macOS or Windows hosts without a Linux `/proc` filesystem, live host collection returns a clean error unless pointed at a captured `/proc` snapshot directory via `--proc-root`.
  88. - Unprivileged users cannot read `/proc/<pid>/io` or `/proc/<pid>/fd` for processes owned by other UIDs; the collector logs a warning and continues with `/proc/<pid>/stat` and `/proc/<pid>/status`.
  89. ## 14. Testing Instructions
  90. ```bash
  91. python3 -m unittest discover -s tests -v
  92. ```
  93. ## 15. Contribution Instructions
  94. See [`CONTRIBUTING.md`](CONTRIBUTING.md) for code style, stdlib-only rules, and test requirements.
  95. ## 16. Roadmap
  96. - Add optional `bpftool` / `bpftrace` read-only map dump parsers when running with `CAP_BPF`.
  97. - Add `/proc/slabinfo` kernel slab cache pressure analysis.
  98. ## 17. License
  99. MIT License. See [`LICENSE`](LICENSE).

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.