art-of-debugging
Systematic methodology and concrete tool recipes for debugging Unix, Python, and PyTorch programs - crashes, hangs, segfaults, wrong output, CUDA OOM, NaN/Inf, slowness, and multi-node/multi-GPU issues. Use when a program crashes, hangs, deadlocks, segfaults, runs out of memory (OOM), produces NaN/Inf or wrong numbers, runs too slowly, or when the user mentions gdb, strace, py-spy, core files, CUDA_LAUNCH_BLOCKING, ldd/nm/LD_PRELOAD, cProfile, or distributed training hangs. Distilled from "The Art of Debugging", the latest version of which can be found at https://github.com/stas00/the-art-of-debugging The latest SKILL.md version can be found at https://github.com/stas00/the-art-of-debugging/blob/master/SKILL.md
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Badge for your README
Markdown
[](/listings/stas00--the-art-of-debugging)HTML
<a href="/listings/stas00--the-art-of-debugging"><img src="/api/listings/stas00--the-art-of-debugging/badge.svg" alt="findsafeskills safety rating" /></a>Install
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
SKILL.md · Manifest · 247 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.
- ---
- name: art-of-debugging
- description: >-
- Systematic methodology and concrete tool recipes for debugging Unix, Python,
- and PyTorch programs - crashes, hangs, segfaults, wrong output, CUDA OOM, NaN/Inf,
- slowness, and multi-node/multi-GPU issues. Use when a program crashes, hangs,
- deadlocks, segfaults, runs out of memory (OOM), produces NaN/Inf or wrong
- numbers, runs too slowly, or when the user mentions gdb, strace, py-spy,
- core files, CUDA_LAUNCH_BLOCKING, ldd/nm/LD_PRELOAD, cProfile, or distributed
- training hangs. Distilled from "The Art of Debugging", the latest version of which can be found at https://github.com/stas00/the-art-of-debugging
- The latest SKILL.md version can be found at https://github.com/stas00/the-art-of-debugging/blob/master/SKILL.md
- ---
- # The Art of Debugging
- > Distilled from **The Art of Debugging Open Book** by Stas Bekman - source: https://github.com/stas00/the-art-of-debugging (CC BY-SA 4.0). This skill is a condensed index; each section links back to the full chapter for depth.
- Actionable methodology + copy-paste recipes for debugging Unix / Python / PyTorch programs. Apply the general loop first; then jump to the domain cheatsheet for the failure at hand. For scaling this up to large-model training/inference on real clusters (compute/storage/network, SLURM, throughput/memory, instabilities, fault tolerance, inference), pair this with [Machine Learning Engineering](https://github.com/stas00/ml-engineering/blob/master/skills/ml-engineering/SKILL.md).
- ## The debugging loop
- The single most important idea: **most of the effort is in *locating* the cause; once you truly understand it, the fix is usually easy.** Optimize everything for reaching understanding faster.
- 1. **Reproduce reliably.** Get one command that triggers the bug every time. If it's flaky, pin the nondeterminism (seeds, ordering, timing, network, uninitialized memory) first - you can't debug what you can't repeat.
- 2. **Shrink the payload.** Make the repro *fast*: fewer layers, tiny model/data, one process, one CPU/GPU, one node. A 2-second repro beats a 2-minute one - you'll run it hundreds of times. See [methodology](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md).
- 3. **Localize.** Confirm you're editing the code that actually runs (the `die` trick), then bisect the search space: which commit, which file/function, which line, which input, which rank.
- 4. **Get a usable signal.** Turn a cryptic failure into a precise one: a real traceback (sync mode), a stack dump (py-spy/gdb), a syscall trace (strace), a printed value at the boundary, or a min/max/NaN check on a tensor.
- 5. **Change one thing, re-run, verify.** Fix on the fast repro, confirm, then re-widen to the full payload. Revert anything that didn't help.
- ### Make the loop fast and reliable
- - **Atomic debug cycles.** Make each iteration a single self-contained, repeatable command (setup + run in one shot) so you never hand-redo multi-step state. See [atomic debug cycles](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#atomic-debug-cycles).
- - **Automate diagnostics, minimize typing.** Alias the repro and your most-used commands; one keystroke to re-run. See [alias frequently used commands](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#alias-frequently-used-commands) and [automate diagnostics](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#automate-diagnostics-minimize-or-avoid-typing).
- - **One-liner programs.** `perl`/`awk`/`python -c` to slice logs, extract fields, transform data on the fly instead of writing throwaway scripts. See [the power of one-liner programs](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#the-power-of-one-liner-programs).
- - **Juggle configs cleanly** when running many debug experiments so results don't get confused. See [juggling multiple sets of configs](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#juggling-multiple-sets-of-configs-for-different-debug-experiments).
- ### Localization techniques
- - **Am I editing the right file/class?** Insert a guaranteed break where you *think* execution goes; if the program doesn't die, you're in the wrong file/class/env:
- ```python
- def suspect():
- die # NameError -> proves this code runs; the traceback also names the caller
- ```
- `traceback.print_stack()` shows callers without stopping (useful when the same function is reached via many paths). See [am I editing the right file and the right class?](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#am-i-editing-the-right-file-and-the-right-class).
- - **Bisect a regression.** `git bisect start / bad / good <rev>` walks commits automatically to the one that broke things - script the test for `git bisect run`. See [finding a breaking commit by bisecting](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#finding-a-breaking-commit-by-bisecting-revisions).
- - **Small/synthetic payload first.** Use tiny or synthetic inputs; switch to real data only when the bug is data-dependent. See [real vs random vs synthetic data](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#real-data-vs-random-data-vs-synthetic-data).
- - **Race conditions.** Reordering/timing bugs hide under async; forcing synchronous execution can expose (or mask) them - note which. See [avoiding race conditions](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#avoiding-race-conditions) and [async vs sync mode](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#async-vs-sync-mode).
- ### Reproducing resource & environment issues
- - **Cap resources on purpose** to test failure paths: emulate a nearly-full disk, limited CPU RAM, or limited GPU memory. See [running out of resources](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#running-out-of-resources-disk-space-cpu-memory-gpu-memory).
- - **Watch resources live.** `watch -n1 nvidia-smi` / `free -h` / `df -h` in a second visible terminal to correlate a hang/OOM with what the machine is doing. See [watching and reproducing resource issues](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#watching-and-reproducing-resource-issues).
- - **Inject `sleep`** to freeze a program at the interesting moment so you can attach a debugger or snapshot state. See [uses for sleep](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#uses-for-sleep).
- - **HPC/SLURM:** keep the allocation and re-run with `srun` instead of re-`sbatch`-ing to cut per-iteration overhead. See [SLURM salloc and srun fast debug combo](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#slurm-salloc-and-srun-fast-debug-combo).
- ## Unix / shell
- Full chapter: [Unix Tools for Debugging](https://github.com/stas00/the-art-of-debugging/blob/master/unix/README.md).
- - **Make shell scripts fail loudly and traceably:**
- ```bash
- set -e # abort on first error
- set -o pipefail # a failing command anywhere in a pipe fails the whole pipe
- set -u # abort on undefined variables (catches typos)
- set -x # trace: print each command with expanded values as it runs
- set +x # turn tracing back off around a noisy region
- ```
- Combine as `set -euo pipefail`. See [controlling script execution](https://github.com/stas00/the-art-of-debugging/blob/master/unix/README.md#controlling-script-execution).
- - **`strace` - trace system calls** to see what a program actually does (files, network, why it's stuck):
- ```bash
- strace python -c "print('hi')" # trace from the start
- strace --pid PID # attach to a running/stuck process
- strace -o log.txt -f torchrun ... # -f follows forked children
- strace -e trace=open,openat,read python prog.py # filter to specific syscalls
- strace -e trace=network -p PID # is it stuck on a socket?
- ```
- Classic use: a process at 100% CPU with no output, or hung on I/O/network. See [strace](https://github.com/stas00/the-art-of-debugging/blob/master/unix/README.md#strace).
- - **`nohup` - survive logout/disconnect** (don't lose a long run to a dropped SSH):
- ```bash
- nohup ./long-running-command > log.txt &
- ```
- See [nohup](https://github.com/stas00/the-art-of-debugging/blob/master/unix/README.md#nohup).
- - **`make`** - after editing compiled sources, rebuild before re-testing, or you'll debug a stale binary. See [make](https://github.com/stas00/the-art-of-debugging/blob/master/unix/README.md#make).
- - **Terminal ergonomics:** search long scrollback and copy multi-line commands cleanly; keep an **informative prompt** (host, path, git branch, last exit code) so you always know where/what ran. See [shell environment](https://github.com/stas00/the-art-of-debugging/blob/master/unix/README.md#shell-environment).
- ## Compiled programs (C/C++, extensions, shared libraries)
- Full chapter: [Debugging Compiled Programs](https://github.com/stas00/the-art-of-debugging/blob/master/compiled-programs/README.md). Compile with `-g` for debug symbols.
- - **Segfault -> backtrace from a core file:**
- ```bash
- ulimit -c unlimited # allow core dumps in this shell
- sudo sysctl -w kernel.core_pattern=/tmp/core-%e.%p.%h.%t # control where cores go
- ./program # crash -> core file written
- gdb ./program /tmp/core-... # or: gdb -c core ./program
- ```
- At the `(gdb)` prompt:
- ```
- bt # backtrace (read bottom-up: outermost caller -> crash site)
- bt full # + local variable values at each frame
- thread apply all bt # backtrace for every thread (essential for multithreaded crashes)
- ```
- See [segmentation fault, core files and gdb](https://github.com/stas00/the-art-of-debugging/blob/master/compiled-programs/README.md#segmentation-fault-core-files-and-gdb).
- - **No core? Run it under gdb** and step to the crash:
- ```bash
- gdb ./program
- (gdb) run # then: bt / break FILE:LINE / next / step / print VAR / continue
- ```
- See [run the program under gdb](https://github.com/stas00/the-art-of-debugging/blob/master/compiled-programs/README.md#run-the-program-under-gdb).
- - **Inspect / snapshot a running process:**
- ```bash
- sudo gdb --pid=PID # attach; then: thread apply all bt
- gcore PID # force a core dump without killing (or: kill -ABRT PID)
- ```
- See [get the backtrace from the still running process](https://github.com/stas00/the-art-of-debugging/blob/master/compiled-programs/README.md#get-the-backtrace-from-the-still-running-process).
- - **"symbol not found" / wrong library loaded:**
- ```bash
- ldd ./program # which shared libs resolve, and to what paths
- LD_LIBRARY_PATH=/path/to/libs ./program # prepend a search dir
- nm -D libfoo.so | grep symbol # is the symbol actually exported? (T=defined, U=undefined)
- LD_PRELOAD=/path/to/shim.so ./program # force-load / override a library
- ```
- See [debugging shared libraries and symbol resolution](https://github.com/stas00/the-art-of-debugging/blob/master/compiled-programs/README.md#debugging-shared-libraries-and-symbol-resolution) ([ldd](https://github.com/stas00/the-art-of-debugging/blob/master/compiled-programs/README.md#ldd), [nm](https://github.com/stas00/the-art-of-debugging/blob/master/compiled-programs/README.md#nm)).
- ## Python
- Full chapter: [Debugging Python Programs](https://github.com/stas00/the-art-of-debugging/blob/master/python/README.md).
- - **Print effectively** instead of scattering bare `print`:
- - **auto-print name+value** so you never mislabel output. See [auto-print what's being observed](https://github.com/stas00/the-art-of-debugging/blob/master/python/README.md#auto-print-whats-being-observed).
- - **dump all attributes** of an object to see its real state. See [printing object variables](https://github.com/stas00/the-art-of-debugging/blob/master/python/README.md#printing-object-variables).
- - **trace calls/returns** with `q` (writes to `/tmp/q`, doesn't pollute stdout). See [q](https://github.com/stas00/the-art-of-debugging/blob/master/python/README.md#q).
- - **Run the code you think you're running.** Edits not taking effect? Wrong copy is imported:
- ```bash
- pip install -e . # run from the source tree, not a copied install
- PYTHONPATH=src python prog.py # or point Python straight at the source
- python -c "import pkg; print(pkg.__file__)" # confirm which file is actually loaded
- ```
- See [ensuring the Python package you edit is the one that is run](https://github.com/stas00/the-art-of-debugging/blob/master/python/README.md#ensuring-the-python-package-you-edit-is-the-one-that-is-run) and [make tests use the git repo's packages](https://github.com/stas00/the-art-of-debugging/blob/master/python/README.md#setting-up-your-test-suite-to-always-use-the-git-repos-python-packages).
- - **Who called this?** `traceback.print_stack()` or the `die` trick to reveal the caller in complex codebases. See [who is calling?](https://github.com/stas00/the-art-of-debugging/blob/master/python/README.md#who-is-calling).
- - **Diagnose a hang (process alive but stuck)** with `py-spy` - no code changes, attaches live:
- ```bash
- pip install py-spy
- py-spy dump -n -p PID # -n also shows native (C/C++ extension) frames
- # all Python subprocesses at once (skip the launcher):
- pgrep -P $(pgrep -o python) | xargs -I {} py-spy dump --pid {}
- ```
- No sudo? `echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope`. **The first line of each dump is where it's stuck.** See [py-spy](https://github.com/stas00/the-art-of-debugging/blob/master/python/README.md#py-spy).
- - **Slow code -> profile before optimizing** (measure, don't guess):
- ```bash
- python -m cProfile -s cumtime prog.py # what dominates cumulative time
- kernprof -l -v prog.py # line_profiler: per-line timing of @profile funcs
- ```
- For sub-ms functions, bump `pstats` precision (e.g. `pstats.f8 = lambda x: f"{x:6.3f}"`) so timings aren't all `0.000`. See [profilers](https://github.com/stas00/the-art-of-debugging/blob/master/python/README.md#profilers) and [cProfile](https://github.com/stas00/the-art-of-debugging/blob/master/python/README.md#cprofile).
- ## PyTorch (incl. CUDA / multi-GPU / multi-node)
- Full chapter: [Debugging PyTorch Programs](https://github.com/stas00/the-art-of-debugging/blob/master/pytorch/README.md).
- ### Debug fast
- Shrink the model, not the problem - make a full run finish in seconds:
- - **Fewer layers** via (1) a local clone with config edits, (2) editing the config object on the fly, or (3) hacking the modeling code. See [reducing the number of layers](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#reducing-the-number-of-layers-for-large-models).
- - **Tiny random model + tiny tokenizer + tiny dataset** for near-instant iterations; reproduce at full scale only for scale-only bugs. See [making a tiny model](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#making-a-tiny-model) and [faster debug with tiny models, tokenizers and datasets](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#faster-debug-and-development-with-tiny-models-tokenizers-and-datasets).
- ### Cryptic CUDA errors
- CUDA is async, so the reported line is usually wrong. Force a real traceback:
- ```bash
- CUDA_LAUNCH_BLOCKING=1 python prog.py # sync CUDA -> accurate Python traceback
- CUDA_VISIBLE_DEVICES="" python prog.py # run on CPU (if feasible) for the clearest traceback
- ```
- See [dealing with async CUDA bugs](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#dealing-with-async-cuda-bugs).
- ### CUDA / CPU OOM
- - Distinguish OOM in **`forward`** (activations - batch/seq len) vs **`backward`** (gradients/optimizer state). See [debugging CUDA OOM in forward](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#debugging-cuda-oom-in-forward) / [backward](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#debugging-cuda-oom-in-backward).
- - **Fragmentation** (free memory exists but not contiguous): tune `PYTORCH_ALLOC_CONF` (e.g. `expandable_segments:True`, `max_split_size_mb:...`). See [overcoming CUDA OOM due to memory fragmentation](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#overcoming-cuda-oom-due-to-memory-fragmentation).
- - **See who allocated what** with the memory profiler / allocation tracing; probe the ceiling with the allocatable-GBs test. See [PyTorch memory profiler](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#pytorch-memory-profiler), [strategic memory allocation tracing](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#strategic-memory-allocation-tracing), [discovering allocatable GBs before OOM](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#discovering-how-many-gbs-is-allocatable-before-oom-for-cpu-and-gpu).
- - **CPU OOM / peak RAM:** see [CPU memory](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#cpu-memory).
- ### NaN/Inf & wrong numbers
- ```python
- torch.autograd.set_detect_anomaly(True) # pinpoint the op that first produced NaN/Inf in backward
- ```
- Find where bad values first appear; watch fp underflow/overflow (especially fp16/bf16); expect small, benign cross-device numeric differences. Inspect tensors compactly (shape/device/dtype/stats) and use `lovely-tensors` for one-line summaries that surface bad tensors fast. See [detecting problematic tensor values](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#detecting-problematic-tensor-values), [underflow and overflow detection](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#underflow-and-overflow-detection), [floating point discrepancies across devices](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#floating-point-math-discrepancies-on-different-devices), [dumping tensor values](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#many-ways-to-dump-tensors-values), and [auto-dumping tensor attributes](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#auto-dumping-desired-tensor-attributes).
- ### Segfault in a PyTorch/NCCL extension
- Same core-file + gdb flow as compiled programs, but **activate the exact python env that produced the core** or gdb can't unpack it:
- ```bash
- conda activate my-env
- gdb python core-python-... # then: bt / thread apply all bt
- ```
- See [segfaults and getting a backtrace from a core file](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#segfaults-and-getting-a-backtrace-from-a-core-file).
- ### Multi-GPU / multi-node hang or deadlock
- 1. **Verify comms first** with a minimal all-reduce test (`torch-distributed-gpu-test.py`); rule out network/NCCL before app code. See [getting nodes to talk to each other](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#getting-nodes-to-talk-to-each-other) and [InfiniBand connection](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#solving-the-infiniband-connection-between-multiple-nodes).
- 2. **Dump every rank's stack at once** with `py-spy` (recipes for python/deepspeed/accelerate, across nodes via `srun`/`pdsh`). Ranks stuck at *different* lines reveal the desync (a mismatched collective). See [diagnosing crashes, hangs and tracing execution](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#diagnosing-crashes-hangs-and-tracing-execution).
- 3. **Make distributed output legible:** prefix every log line with `node:rank`, and target `pdb` at one rank. See [prefixing logs](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#prefixing-logs-with-noderank-interleaved-asserts), [pdb on a specific rank](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#invoke-pdb-on-a-specific-rank-in-multi-node-training).
- 4. **Narrow further:** check for a [network-level hang](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#network-level-hanging), [isolate a bad GPU](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#isolate-problematic-gpus), or trace line-by-line with the [python `trace`](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#python-trace) module. On AMD, a slow/hung run may be [IOMMU-related](https://github.com/stas00/ml-engineering/blob/master/compute/accelerator/amd/debug.md#hangs-or-slow-multi-gpu-runs-and-iommu).
- For the cluster-level context around these bugs (verifying node connectivity, NCCL/InfiniBand tuning, network benchmarking, checkpointing/fault tolerance), see [Machine Learning Engineering](https://github.com/stas00/ml-engineering/blob/master/skills/ml-engineering/SKILL.md).
- ### Performance
- - **Time regions precisely.** For GPU work use CUDA events (CPU timers lie because kernels are async):
- ```python
- s, e = torch.cuda.Event(enable_timing=True), torch.cuda.Event(enable_timing=True)
- s.record(); run(); e.record(); torch.cuda.synchronize()
- ms = s.elapsed_time(e)
- ```
- See [measuring durations](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#measuring-durations).
- - **Profile ops** with `torch.profiler` (CPU+GPU, op-level, small overhead); when it's not enough, drop to `cProfile` for pure-Python hot spots. See [performance and profiling](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#performance-and-profiling).
- ## Pick the tool by symptom
- | Symptom | Reach for |
- |---|---|
- | Stuck / 100% CPU / no output | `py-spy dump` (Python), `strace --pid` (syscalls), `gdb --pid` (native) |
- | Multi-GPU/node hang | minimal collective test -> `py-spy` across all ranks -> `node:rank` logs |
- | Segfault / crash in C or extension | core file + `gdb` (`bt`, `bt full`, `thread apply all bt`) |
- | Cryptic CUDA error / wrong line | `CUDA_LAUNCH_BLOCKING=1`, or run on CPU |
- | CUDA/CPU OOM | forward vs backward; fragmentation (`PYTORCH_ALLOC_CONF`); memory profiler |
- | NaN/Inf / wrong numbers | `set_detect_anomaly`, under/overflow detection, per-tensor stats, `lovely-tensors` |
- | "my edits do nothing" | the `die` trick; `pip install -e .` / `PYTHONPATH`; check `pkg.__file__` |
- | Who calls this? | `traceback.print_stack()` / `die` |
- | Wrong/missing shared lib | `ldd`, `nm -D`, `LD_LIBRARY_PATH`, `LD_PRELOAD` |
- | Too slow (Python) | `cProfile -s cumtime`, `line_profiler` |
- | Too slow (PyTorch/GPU) | CUDA events, `torch.profiler` |
- | Regression appeared | `git bisect run` |
- | Script fails silently | `set -euo pipefail`, `set -x` |
- | Flaky / non-deterministic | pin seeds/order/timing; force sync; check race conditions |
- | Long run dies on disconnect | `nohup ... > log &` (or tmux/screen) |
- ## Notes for AI agents
- - **Observe before guessing:** obtain a stack dump / traceback / syscall trace / boundary value / tensor stat *before* proposing a cause; don't speculate from the error string alone.
- - **Secure a fast, reliable repro first,** then optimize its speed - iteration count matters more than any single clever idea.
- - **Change one variable at a time,** re-run the repro, and revert changes that don't move the needle.
- - **Confirm you're running the code you edited** (`pkg.__file__`, the `die` trick) before deeper investigation - a huge share of "impossible" bugs are wrong-file/wrong-env.
- - **Read the linked chapter section** before applying an unfamiliar recipe - each has worked examples, caveats, and copy-paste scripts.
- - Prefer built-in, low-overhead tools (`py-spy`, `strace`, `gdb`, env vars) that need no source changes and work on already-running processes.
- - For large-scale ML training/inference engineering (bottleneck analysis, throughput/memory, distributed hangs at cluster scale, fault tolerance), use the companion skill: [Machine Learning Engineering](https://github.com/stas00/ml-engineering/blob/master/skills/ml-engineering/SKILL.md).
README.md · README · 100 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.
- # The Art of Debugging Open Book
- <!-- not-in-ebook-start -->
- <a href="#ebook-versions-of-the-book"><img src="images/The-Art-of-Debugging-book-cover-200x259.png" align="right" width="200" height="259" alt="The Art of Debugging Open Book cover"></a>
- <!-- not-in-ebook-end -->
- This evolving book will teach you how to:
- 1. **Debug normal issues really fast**
- 2. **Make complicated issues possible to debug**
- footnote: adapted from Perl's slogan: "Easy things should be easy and hard things should be possible".
- This book is a collection of methodologies and copy-n-paste recipes for successful debugging of simple and complicated software problems. Some sections are quite complete, while other are a work in progress.
- Besides debugging methodologies the second focus of this book is on sharing the best tools I have discovered to get the debugging job done successfully, while ideally "losing the least amount of hair" in the process.
- ## Intro
- I have been developing software since 1995 and a lot of this work involved debugging. Over the years I developed various efficient methodologies for discovering the source of the problem, which is the most difficult stage before it can be solved. Since after the problem is understood, typically its resolution is at easy reach.
- Every so often someone I would be debugging some difficult problem with would suggest to share my approaches with the world. I always said that it'd be too difficult to generalize, but recently the planted seed seems to have sprouted and so in the following chapters I will try to share some of the insights to ease this very difficult at times process.
- ## Table of Contents
- 1. **[Fast Debugging Methodology](./methodology/)**
- 2. **[Unix Tools for Debugging](./unix/)** - `bash`, `strace`, `make`, prompt, `nohup`, etc.
- 3. **[Debugging Compiled Programs](./compiled-programs/)** - `gdb`, `ldd`, `nm`, `LD_LIBRARY_PATH`, `LD_PRELOAD`, etc.
- 4. **[Debugging Python Programs](./python/)** - `py-spy`, paths, auto-print, etc.
- 5. **[Debugging PyTorch Programs](./pytorch/)** - cpu and gpu memory, performance, models, tensors, etc.
- 6. **[Debugging Machine Learning Projects](https://github.com/stas00/ml-engineering/tree/master/debug)** (external)
- ## Updates
- I announce any significant updates on my twitter channel [https://twitter.com/StasBekman](https://twitter.com/StasBekman).
- ## Ebook versions of the book
- You can download various ebook formats of this book:
- * [PDF](https://huggingface.co/stas/the-art-of-debugging-book/resolve/main/Stas%20Bekman%20-%20The%20Art%20of%20Debugging.pdf?download=true)
- * [EPUB](https://huggingface.co/stas/the-art-of-debugging-book/resolve/main/Stas%20Bekman%20-%20The%20Art%20of%20Debugging.epub?download=true)
- I will try to rebuild these once in a few weeks or so, but if you want the latest ebook versions, the instructions for building are [here](build).
- ## SKILL.md for AI agents
- I maintain a [SKILL.md](./SKILL.md) file that you can use to teach your AI agent to debug your projects better.
- See also the companion skills: [Machine Learning Engineering](https://github.com/stas00/ml-engineering/blob/master/skills/ml-engineering/SKILL.md) and [Stas' Python Cookbook](https://github.com/stas00/python-cookbook/blob/master/SKILL.md).
- ## Related Open Books
- - [Machine Learning Engineering Open Book](https://github.com/stas00/ml-engineering) — training and operating large-scale ML models.
- - [Stas' Python Cookbook](https://github.com/stas00/python-cookbook) — everyday Python and standard-library recipes.
- ## Contributing
- If you found a bug, typo or would like to propose an improvement please don't hesitate to open an [Issue](https://github.com/stas00/the-art-of-debugging/issues) or contribute a PR.
- ## License
- The content of this site is distributed under [Attribution-ShareAlike 4.0 International](./LICENSE-CC-BY-SA).
- ## My repositories map
- ✔ **Books:**
- [Machine Learning Engineering](https://github.com/stas00/ml-engineering) |
- [The Art of Debugging](https://github.com/stas00/the-art-of-debugging) |
- [Stas' Python Cookbook](https://github.com/stas00/python-cookbook)
- ✔ **Applications:**
- [ipyexperiments](https://github.com/stas00/ipyexperiments)
- ✔ **Tools and Cheatsheets:**
- [bash](https://github.com/stas00/bash-tools) |
- [conda](https://github.com/stas00/conda-tools) |
- [git](https://github.com/stas00/git-tools) |
- [jupyter-notebook](https://github.com/stas00/jupyter-notebook-tools) |
- [make](https://github.com/stas00/make-tools) |
- [python](https://github.com/stas00/python-tools) |
- [tensorboard](https://github.com/stas00/tensorboard-tools) |
- [unix](https://github.com/stas00/unix-tools)
- ✔ **Other Machine Learning:**
- [ML ways](https://github.com/stas00/ml-ways) |
- [Porting](https://github.com/stas00/porting)
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.