Skip to content
Back to search

art-of-debugging

SkillWorks with Claude CodeGitHub

Systematic methodology and concrete tool recipes for debugging Unix, Python, and PyTorch programs - crashes, hangs, segfaults, wrong output, CUDA OOM, NaN/Inf, slowness, and multi-node/multi-GPU issues. Use when a program crashes, hangs, deadlocks, segfaults, runs out of memory (OOM), produces NaN/Inf or wrong numbers, runs too slowly, or when the user mentions gdb, strace, py-spy, core files, CUDA_LAUNCH_BLOCKING, ldd/nm/LD_PRELOAD, cProfile, or distributed training hangs. Distilled from "The Art of Debugging", the latest version of which can be found at https://github.com/stas00/the-art-of-debugging The latest SKILL.md version can be found at https://github.com/stas00/the-art-of-debugging/blob/master/SKILL.md

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/stas00--the-art-of-debugging/badge.svg)](/listings/stas00--the-art-of-debugging)

HTML

<a href="/listings/stas00--the-art-of-debugging"><img src="/api/listings/stas00--the-art-of-debugging/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

SKILL.md · Manifest · 247 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. ---
  2. name: art-of-debugging
  3. description: >-
  4. Systematic methodology and concrete tool recipes for debugging Unix, Python,
  5. and PyTorch programs - crashes, hangs, segfaults, wrong output, CUDA OOM, NaN/Inf,
  6. slowness, and multi-node/multi-GPU issues. Use when a program crashes, hangs,
  7. deadlocks, segfaults, runs out of memory (OOM), produces NaN/Inf or wrong
  8. numbers, runs too slowly, or when the user mentions gdb, strace, py-spy,
  9. core files, CUDA_LAUNCH_BLOCKING, ldd/nm/LD_PRELOAD, cProfile, or distributed
  10. training hangs. Distilled from "The Art of Debugging", the latest version of which can be found at https://github.com/stas00/the-art-of-debugging
  11. The latest SKILL.md version can be found at https://github.com/stas00/the-art-of-debugging/blob/master/SKILL.md
  12. ---
  13. # The Art of Debugging
  14. > Distilled from **The Art of Debugging Open Book** by Stas Bekman - source: https://github.com/stas00/the-art-of-debugging (CC BY-SA 4.0). This skill is a condensed index; each section links back to the full chapter for depth.
  15. Actionable methodology + copy-paste recipes for debugging Unix / Python / PyTorch programs. Apply the general loop first; then jump to the domain cheatsheet for the failure at hand. For scaling this up to large-model training/inference on real clusters (compute/storage/network, SLURM, throughput/memory, instabilities, fault tolerance, inference), pair this with [Machine Learning Engineering](https://github.com/stas00/ml-engineering/blob/master/skills/ml-engineering/SKILL.md).
  16. ## The debugging loop
  17. The single most important idea: **most of the effort is in *locating* the cause; once you truly understand it, the fix is usually easy.** Optimize everything for reaching understanding faster.
  18. 1. **Reproduce reliably.** Get one command that triggers the bug every time. If it's flaky, pin the nondeterminism (seeds, ordering, timing, network, uninitialized memory) first - you can't debug what you can't repeat.
  19. 2. **Shrink the payload.** Make the repro *fast*: fewer layers, tiny model/data, one process, one CPU/GPU, one node. A 2-second repro beats a 2-minute one - you'll run it hundreds of times. See [methodology](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md).
  20. 3. **Localize.** Confirm you're editing the code that actually runs (the `die` trick), then bisect the search space: which commit, which file/function, which line, which input, which rank.
  21. 4. **Get a usable signal.** Turn a cryptic failure into a precise one: a real traceback (sync mode), a stack dump (py-spy/gdb), a syscall trace (strace), a printed value at the boundary, or a min/max/NaN check on a tensor.
  22. 5. **Change one thing, re-run, verify.** Fix on the fast repro, confirm, then re-widen to the full payload. Revert anything that didn't help.
  23. ### Make the loop fast and reliable
  24. - **Atomic debug cycles.** Make each iteration a single self-contained, repeatable command (setup + run in one shot) so you never hand-redo multi-step state. See [atomic debug cycles](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#atomic-debug-cycles).
  25. - **Automate diagnostics, minimize typing.** Alias the repro and your most-used commands; one keystroke to re-run. See [alias frequently used commands](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#alias-frequently-used-commands) and [automate diagnostics](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#automate-diagnostics-minimize-or-avoid-typing).
  26. - **One-liner programs.** `perl`/`awk`/`python -c` to slice logs, extract fields, transform data on the fly instead of writing throwaway scripts. See [the power of one-liner programs](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#the-power-of-one-liner-programs).
  27. - **Juggle configs cleanly** when running many debug experiments so results don't get confused. See [juggling multiple sets of configs](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#juggling-multiple-sets-of-configs-for-different-debug-experiments).
  28. ### Localization techniques
  29. - **Am I editing the right file/class?** Insert a guaranteed break where you *think* execution goes; if the program doesn't die, you're in the wrong file/class/env:
  30. ```python
  31. def suspect():
  32. die # NameError -> proves this code runs; the traceback also names the caller
  33. ```
  34. `traceback.print_stack()` shows callers without stopping (useful when the same function is reached via many paths). See [am I editing the right file and the right class?](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#am-i-editing-the-right-file-and-the-right-class).
  35. - **Bisect a regression.** `git bisect start / bad / good <rev>` walks commits automatically to the one that broke things - script the test for `git bisect run`. See [finding a breaking commit by bisecting](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#finding-a-breaking-commit-by-bisecting-revisions).
  36. - **Small/synthetic payload first.** Use tiny or synthetic inputs; switch to real data only when the bug is data-dependent. See [real vs random vs synthetic data](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#real-data-vs-random-data-vs-synthetic-data).
  37. - **Race conditions.** Reordering/timing bugs hide under async; forcing synchronous execution can expose (or mask) them - note which. See [avoiding race conditions](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#avoiding-race-conditions) and [async vs sync mode](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#async-vs-sync-mode).
  38. ### Reproducing resource & environment issues
  39. - **Cap resources on purpose** to test failure paths: emulate a nearly-full disk, limited CPU RAM, or limited GPU memory. See [running out of resources](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#running-out-of-resources-disk-space-cpu-memory-gpu-memory).
  40. - **Watch resources live.** `watch -n1 nvidia-smi` / `free -h` / `df -h` in a second visible terminal to correlate a hang/OOM with what the machine is doing. See [watching and reproducing resource issues](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#watching-and-reproducing-resource-issues).
  41. - **Inject `sleep`** to freeze a program at the interesting moment so you can attach a debugger or snapshot state. See [uses for sleep](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#uses-for-sleep).
  42. - **HPC/SLURM:** keep the allocation and re-run with `srun` instead of re-`sbatch`-ing to cut per-iteration overhead. See [SLURM salloc and srun fast debug combo](https://github.com/stas00/the-art-of-debugging/blob/master/methodology/README.md#slurm-salloc-and-srun-fast-debug-combo).
  43. ## Unix / shell
  44. Full chapter: [Unix Tools for Debugging](https://github.com/stas00/the-art-of-debugging/blob/master/unix/README.md).
  45. - **Make shell scripts fail loudly and traceably:**
  46. ```bash
  47. set -e # abort on first error
  48. set -o pipefail # a failing command anywhere in a pipe fails the whole pipe
  49. set -u # abort on undefined variables (catches typos)
  50. set -x # trace: print each command with expanded values as it runs
  51. set +x # turn tracing back off around a noisy region
  52. ```
  53. Combine as `set -euo pipefail`. See [controlling script execution](https://github.com/stas00/the-art-of-debugging/blob/master/unix/README.md#controlling-script-execution).
  54. - **`strace` - trace system calls** to see what a program actually does (files, network, why it's stuck):
  55. ```bash
  56. strace python -c "print('hi')" # trace from the start
  57. strace --pid PID # attach to a running/stuck process
  58. strace -o log.txt -f torchrun ... # -f follows forked children
  59. strace -e trace=open,openat,read python prog.py # filter to specific syscalls
  60. strace -e trace=network -p PID # is it stuck on a socket?
  61. ```
  62. Classic use: a process at 100% CPU with no output, or hung on I/O/network. See [strace](https://github.com/stas00/the-art-of-debugging/blob/master/unix/README.md#strace).
  63. - **`nohup` - survive logout/disconnect** (don't lose a long run to a dropped SSH):
  64. ```bash
  65. nohup ./long-running-command > log.txt &
  66. ```
  67. See [nohup](https://github.com/stas00/the-art-of-debugging/blob/master/unix/README.md#nohup).
  68. - **`make`** - after editing compiled sources, rebuild before re-testing, or you'll debug a stale binary. See [make](https://github.com/stas00/the-art-of-debugging/blob/master/unix/README.md#make).
  69. - **Terminal ergonomics:** search long scrollback and copy multi-line commands cleanly; keep an **informative prompt** (host, path, git branch, last exit code) so you always know where/what ran. See [shell environment](https://github.com/stas00/the-art-of-debugging/blob/master/unix/README.md#shell-environment).
  70. ## Compiled programs (C/C++, extensions, shared libraries)
  71. Full chapter: [Debugging Compiled Programs](https://github.com/stas00/the-art-of-debugging/blob/master/compiled-programs/README.md). Compile with `-g` for debug symbols.
  72. - **Segfault -> backtrace from a core file:**
  73. ```bash
  74. ulimit -c unlimited # allow core dumps in this shell
  75. sudo sysctl -w kernel.core_pattern=/tmp/core-%e.%p.%h.%t # control where cores go
  76. ./program # crash -> core file written
  77. gdb ./program /tmp/core-... # or: gdb -c core ./program
  78. ```
  79. At the `(gdb)` prompt:
  80. ```
  81. bt # backtrace (read bottom-up: outermost caller -> crash site)
  82. bt full # + local variable values at each frame
  83. thread apply all bt # backtrace for every thread (essential for multithreaded crashes)
  84. ```
  85. See [segmentation fault, core files and gdb](https://github.com/stas00/the-art-of-debugging/blob/master/compiled-programs/README.md#segmentation-fault-core-files-and-gdb).
  86. - **No core? Run it under gdb** and step to the crash:
  87. ```bash
  88. gdb ./program
  89. (gdb) run # then: bt / break FILE:LINE / next / step / print VAR / continue
  90. ```
  91. See [run the program under gdb](https://github.com/stas00/the-art-of-debugging/blob/master/compiled-programs/README.md#run-the-program-under-gdb).
  92. - **Inspect / snapshot a running process:**
  93. ```bash
  94. sudo gdb --pid=PID # attach; then: thread apply all bt
  95. gcore PID # force a core dump without killing (or: kill -ABRT PID)
  96. ```
  97. See [get the backtrace from the still running process](https://github.com/stas00/the-art-of-debugging/blob/master/compiled-programs/README.md#get-the-backtrace-from-the-still-running-process).
  98. - **"symbol not found" / wrong library loaded:**
  99. ```bash
  100. ldd ./program # which shared libs resolve, and to what paths
  101. LD_LIBRARY_PATH=/path/to/libs ./program # prepend a search dir
  102. nm -D libfoo.so | grep symbol # is the symbol actually exported? (T=defined, U=undefined)
  103. LD_PRELOAD=/path/to/shim.so ./program # force-load / override a library
  104. ```
  105. See [debugging shared libraries and symbol resolution](https://github.com/stas00/the-art-of-debugging/blob/master/compiled-programs/README.md#debugging-shared-libraries-and-symbol-resolution) ([ldd](https://github.com/stas00/the-art-of-debugging/blob/master/compiled-programs/README.md#ldd), [nm](https://github.com/stas00/the-art-of-debugging/blob/master/compiled-programs/README.md#nm)).
  106. ## Python
  107. Full chapter: [Debugging Python Programs](https://github.com/stas00/the-art-of-debugging/blob/master/python/README.md).
  108. - **Print effectively** instead of scattering bare `print`:
  109. - **auto-print name+value** so you never mislabel output. See [auto-print what's being observed](https://github.com/stas00/the-art-of-debugging/blob/master/python/README.md#auto-print-whats-being-observed).
  110. - **dump all attributes** of an object to see its real state. See [printing object variables](https://github.com/stas00/the-art-of-debugging/blob/master/python/README.md#printing-object-variables).
  111. - **trace calls/returns** with `q` (writes to `/tmp/q`, doesn't pollute stdout). See [q](https://github.com/stas00/the-art-of-debugging/blob/master/python/README.md#q).
  112. - **Run the code you think you're running.** Edits not taking effect? Wrong copy is imported:
  113. ```bash
  114. pip install -e . # run from the source tree, not a copied install
  115. PYTHONPATH=src python prog.py # or point Python straight at the source
  116. python -c "import pkg; print(pkg.__file__)" # confirm which file is actually loaded
  117. ```
  118. See [ensuring the Python package you edit is the one that is run](https://github.com/stas00/the-art-of-debugging/blob/master/python/README.md#ensuring-the-python-package-you-edit-is-the-one-that-is-run) and [make tests use the git repo's packages](https://github.com/stas00/the-art-of-debugging/blob/master/python/README.md#setting-up-your-test-suite-to-always-use-the-git-repos-python-packages).
  119. - **Who called this?** `traceback.print_stack()` or the `die` trick to reveal the caller in complex codebases. See [who is calling?](https://github.com/stas00/the-art-of-debugging/blob/master/python/README.md#who-is-calling).
  120. - **Diagnose a hang (process alive but stuck)** with `py-spy` - no code changes, attaches live:
  121. ```bash
  122. pip install py-spy
  123. py-spy dump -n -p PID # -n also shows native (C/C++ extension) frames
  124. # all Python subprocesses at once (skip the launcher):
  125. pgrep -P $(pgrep -o python) | xargs -I {} py-spy dump --pid {}
  126. ```
  127. No sudo? `echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope`. **The first line of each dump is where it's stuck.** See [py-spy](https://github.com/stas00/the-art-of-debugging/blob/master/python/README.md#py-spy).
  128. - **Slow code -> profile before optimizing** (measure, don't guess):
  129. ```bash
  130. python -m cProfile -s cumtime prog.py # what dominates cumulative time
  131. kernprof -l -v prog.py # line_profiler: per-line timing of @profile funcs
  132. ```
  133. For sub-ms functions, bump `pstats` precision (e.g. `pstats.f8 = lambda x: f"{x:6.3f}"`) so timings aren't all `0.000`. See [profilers](https://github.com/stas00/the-art-of-debugging/blob/master/python/README.md#profilers) and [cProfile](https://github.com/stas00/the-art-of-debugging/blob/master/python/README.md#cprofile).
  134. ## PyTorch (incl. CUDA / multi-GPU / multi-node)
  135. Full chapter: [Debugging PyTorch Programs](https://github.com/stas00/the-art-of-debugging/blob/master/pytorch/README.md).
  136. ### Debug fast
  137. Shrink the model, not the problem - make a full run finish in seconds:
  138. - **Fewer layers** via (1) a local clone with config edits, (2) editing the config object on the fly, or (3) hacking the modeling code. See [reducing the number of layers](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#reducing-the-number-of-layers-for-large-models).
  139. - **Tiny random model + tiny tokenizer + tiny dataset** for near-instant iterations; reproduce at full scale only for scale-only bugs. See [making a tiny model](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#making-a-tiny-model) and [faster debug with tiny models, tokenizers and datasets](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#faster-debug-and-development-with-tiny-models-tokenizers-and-datasets).
  140. ### Cryptic CUDA errors
  141. CUDA is async, so the reported line is usually wrong. Force a real traceback:
  142. ```bash
  143. CUDA_LAUNCH_BLOCKING=1 python prog.py # sync CUDA -> accurate Python traceback
  144. CUDA_VISIBLE_DEVICES="" python prog.py # run on CPU (if feasible) for the clearest traceback
  145. ```
  146. See [dealing with async CUDA bugs](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#dealing-with-async-cuda-bugs).
  147. ### CUDA / CPU OOM
  148. - Distinguish OOM in **`forward`** (activations - batch/seq len) vs **`backward`** (gradients/optimizer state). See [debugging CUDA OOM in forward](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#debugging-cuda-oom-in-forward) / [backward](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#debugging-cuda-oom-in-backward).
  149. - **Fragmentation** (free memory exists but not contiguous): tune `PYTORCH_ALLOC_CONF` (e.g. `expandable_segments:True`, `max_split_size_mb:...`). See [overcoming CUDA OOM due to memory fragmentation](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#overcoming-cuda-oom-due-to-memory-fragmentation).
  150. - **See who allocated what** with the memory profiler / allocation tracing; probe the ceiling with the allocatable-GBs test. See [PyTorch memory profiler](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#pytorch-memory-profiler), [strategic memory allocation tracing](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#strategic-memory-allocation-tracing), [discovering allocatable GBs before OOM](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#discovering-how-many-gbs-is-allocatable-before-oom-for-cpu-and-gpu).
  151. - **CPU OOM / peak RAM:** see [CPU memory](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#cpu-memory).
  152. ### NaN/Inf & wrong numbers
  153. ```python
  154. torch.autograd.set_detect_anomaly(True) # pinpoint the op that first produced NaN/Inf in backward
  155. ```
  156. Find where bad values first appear; watch fp underflow/overflow (especially fp16/bf16); expect small, benign cross-device numeric differences. Inspect tensors compactly (shape/device/dtype/stats) and use `lovely-tensors` for one-line summaries that surface bad tensors fast. See [detecting problematic tensor values](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#detecting-problematic-tensor-values), [underflow and overflow detection](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#underflow-and-overflow-detection), [floating point discrepancies across devices](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#floating-point-math-discrepancies-on-different-devices), [dumping tensor values](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#many-ways-to-dump-tensors-values), and [auto-dumping tensor attributes](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#auto-dumping-desired-tensor-attributes).
  157. ### Segfault in a PyTorch/NCCL extension
  158. Same core-file + gdb flow as compiled programs, but **activate the exact python env that produced the core** or gdb can't unpack it:
  159. ```bash
  160. conda activate my-env
  161. gdb python core-python-... # then: bt / thread apply all bt
  162. ```
  163. See [segfaults and getting a backtrace from a core file](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#segfaults-and-getting-a-backtrace-from-a-core-file).
  164. ### Multi-GPU / multi-node hang or deadlock
  165. 1. **Verify comms first** with a minimal all-reduce test (`torch-distributed-gpu-test.py`); rule out network/NCCL before app code. See [getting nodes to talk to each other](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#getting-nodes-to-talk-to-each-other) and [InfiniBand connection](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#solving-the-infiniband-connection-between-multiple-nodes).
  166. 2. **Dump every rank's stack at once** with `py-spy` (recipes for python/deepspeed/accelerate, across nodes via `srun`/`pdsh`). Ranks stuck at *different* lines reveal the desync (a mismatched collective). See [diagnosing crashes, hangs and tracing execution](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#diagnosing-crashes-hangs-and-tracing-execution).
  167. 3. **Make distributed output legible:** prefix every log line with `node:rank`, and target `pdb` at one rank. See [prefixing logs](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#prefixing-logs-with-noderank-interleaved-asserts), [pdb on a specific rank](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#invoke-pdb-on-a-specific-rank-in-multi-node-training).
  168. 4. **Narrow further:** check for a [network-level hang](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#network-level-hanging), [isolate a bad GPU](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#isolate-problematic-gpus), or trace line-by-line with the [python `trace`](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#python-trace) module. On AMD, a slow/hung run may be [IOMMU-related](https://github.com/stas00/ml-engineering/blob/master/compute/accelerator/amd/debug.md#hangs-or-slow-multi-gpu-runs-and-iommu).
  169. For the cluster-level context around these bugs (verifying node connectivity, NCCL/InfiniBand tuning, network benchmarking, checkpointing/fault tolerance), see [Machine Learning Engineering](https://github.com/stas00/ml-engineering/blob/master/skills/ml-engineering/SKILL.md).
  170. ### Performance
  171. - **Time regions precisely.** For GPU work use CUDA events (CPU timers lie because kernels are async):
  172. ```python
  173. s, e = torch.cuda.Event(enable_timing=True), torch.cuda.Event(enable_timing=True)
  174. s.record(); run(); e.record(); torch.cuda.synchronize()
  175. ms = s.elapsed_time(e)
  176. ```
  177. See [measuring durations](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#measuring-durations).
  178. - **Profile ops** with `torch.profiler` (CPU+GPU, op-level, small overhead); when it's not enough, drop to `cProfile` for pure-Python hot spots. See [performance and profiling](https://github.com/stas00/the-art-of-debugging/tree/master/pytorch#performance-and-profiling).
  179. ## Pick the tool by symptom
  180. | Symptom | Reach for |
  181. |---|---|
  182. | Stuck / 100% CPU / no output | `py-spy dump` (Python), `strace --pid` (syscalls), `gdb --pid` (native) |
  183. | Multi-GPU/node hang | minimal collective test -> `py-spy` across all ranks -> `node:rank` logs |
  184. | Segfault / crash in C or extension | core file + `gdb` (`bt`, `bt full`, `thread apply all bt`) |
  185. | Cryptic CUDA error / wrong line | `CUDA_LAUNCH_BLOCKING=1`, or run on CPU |
  186. | CUDA/CPU OOM | forward vs backward; fragmentation (`PYTORCH_ALLOC_CONF`); memory profiler |
  187. | NaN/Inf / wrong numbers | `set_detect_anomaly`, under/overflow detection, per-tensor stats, `lovely-tensors` |
  188. | "my edits do nothing" | the `die` trick; `pip install -e .` / `PYTHONPATH`; check `pkg.__file__` |
  189. | Who calls this? | `traceback.print_stack()` / `die` |
  190. | Wrong/missing shared lib | `ldd`, `nm -D`, `LD_LIBRARY_PATH`, `LD_PRELOAD` |
  191. | Too slow (Python) | `cProfile -s cumtime`, `line_profiler` |
  192. | Too slow (PyTorch/GPU) | CUDA events, `torch.profiler` |
  193. | Regression appeared | `git bisect run` |
  194. | Script fails silently | `set -euo pipefail`, `set -x` |
  195. | Flaky / non-deterministic | pin seeds/order/timing; force sync; check race conditions |
  196. | Long run dies on disconnect | `nohup ... > log &` (or tmux/screen) |
  197. ## Notes for AI agents
  198. - **Observe before guessing:** obtain a stack dump / traceback / syscall trace / boundary value / tensor stat *before* proposing a cause; don't speculate from the error string alone.
  199. - **Secure a fast, reliable repro first,** then optimize its speed - iteration count matters more than any single clever idea.
  200. - **Change one variable at a time,** re-run the repro, and revert changes that don't move the needle.
  201. - **Confirm you're running the code you edited** (`pkg.__file__`, the `die` trick) before deeper investigation - a huge share of "impossible" bugs are wrong-file/wrong-env.
  202. - **Read the linked chapter section** before applying an unfamiliar recipe - each has worked examples, caveats, and copy-paste scripts.
  203. - Prefer built-in, low-overhead tools (`py-spy`, `strace`, `gdb`, env vars) that need no source changes and work on already-running processes.
  204. - For large-scale ML training/inference engineering (bottleneck analysis, throughput/memory, distributed hangs at cluster scale, fault tolerance), use the companion skill: [Machine Learning Engineering](https://github.com/stas00/ml-engineering/blob/master/skills/ml-engineering/SKILL.md).
README.md · README · 100 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. # The Art of Debugging Open Book
  2. <!-- not-in-ebook-start -->
  3. <a href="#ebook-versions-of-the-book"><img src="images/The-Art-of-Debugging-book-cover-200x259.png" align="right" width="200" height="259" alt="The Art of Debugging Open Book cover"></a>
  4. <!-- not-in-ebook-end -->
  5. This evolving book will teach you how to:
  6. 1. **Debug normal issues really fast**
  7. 2. **Make complicated issues possible to debug**
  8. footnote: adapted from Perl's slogan: "Easy things should be easy and hard things should be possible".
  9. This book is a collection of methodologies and copy-n-paste recipes for successful debugging of simple and complicated software problems. Some sections are quite complete, while other are a work in progress.
  10. Besides debugging methodologies the second focus of this book is on sharing the best tools I have discovered to get the debugging job done successfully, while ideally "losing the least amount of hair" in the process.
  11. ## Intro
  12. I have been developing software since 1995 and a lot of this work involved debugging. Over the years I developed various efficient methodologies for discovering the source of the problem, which is the most difficult stage before it can be solved. Since after the problem is understood, typically its resolution is at easy reach.
  13. Every so often someone I would be debugging some difficult problem with would suggest to share my approaches with the world. I always said that it'd be too difficult to generalize, but recently the planted seed seems to have sprouted and so in the following chapters I will try to share some of the insights to ease this very difficult at times process.
  14. ## Table of Contents
  15. 1. **[Fast Debugging Methodology](./methodology/)**
  16. 2. **[Unix Tools for Debugging](./unix/)** - `bash`, `strace`, `make`, prompt, `nohup`, etc.
  17. 3. **[Debugging Compiled Programs](./compiled-programs/)** - `gdb`, `ldd`, `nm`, `LD_LIBRARY_PATH`, `LD_PRELOAD`, etc.
  18. 4. **[Debugging Python Programs](./python/)** - `py-spy`, paths, auto-print, etc.
  19. 5. **[Debugging PyTorch Programs](./pytorch/)** - cpu and gpu memory, performance, models, tensors, etc.
  20. 6. **[Debugging Machine Learning Projects](https://github.com/stas00/ml-engineering/tree/master/debug)** (external)
  21. ## Updates
  22. I announce any significant updates on my twitter channel [https://twitter.com/StasBekman](https://twitter.com/StasBekman).
  23. ## Ebook versions of the book
  24. You can download various ebook formats of this book:
  25. * [PDF](https://huggingface.co/stas/the-art-of-debugging-book/resolve/main/Stas%20Bekman%20-%20The%20Art%20of%20Debugging.pdf?download=true)
  26. * [EPUB](https://huggingface.co/stas/the-art-of-debugging-book/resolve/main/Stas%20Bekman%20-%20The%20Art%20of%20Debugging.epub?download=true)
  27. I will try to rebuild these once in a few weeks or so, but if you want the latest ebook versions, the instructions for building are [here](build).
  28. ## SKILL.md for AI agents
  29. I maintain a [SKILL.md](./SKILL.md) file that you can use to teach your AI agent to debug your projects better.
  30. See also the companion skills: [Machine Learning Engineering](https://github.com/stas00/ml-engineering/blob/master/skills/ml-engineering/SKILL.md) and [Stas' Python Cookbook](https://github.com/stas00/python-cookbook/blob/master/SKILL.md).
  31. ## Related Open Books
  32. - [Machine Learning Engineering Open Book](https://github.com/stas00/ml-engineering) — training and operating large-scale ML models.
  33. - [Stas' Python Cookbook](https://github.com/stas00/python-cookbook) — everyday Python and standard-library recipes.
  34. ## Contributing
  35. If you found a bug, typo or would like to propose an improvement please don't hesitate to open an [Issue](https://github.com/stas00/the-art-of-debugging/issues) or contribute a PR.
  36. ## License
  37. The content of this site is distributed under [Attribution-ShareAlike 4.0 International](./LICENSE-CC-BY-SA).
  38. ## My repositories map
  39. ✔ **Books:**
  40. [Machine Learning Engineering](https://github.com/stas00/ml-engineering) |
  41. [The Art of Debugging](https://github.com/stas00/the-art-of-debugging) |
  42. [Stas' Python Cookbook](https://github.com/stas00/python-cookbook)
  43. ✔ **Applications:**
  44. [ipyexperiments](https://github.com/stas00/ipyexperiments)
  45. ✔ **Tools and Cheatsheets:**
  46. [bash](https://github.com/stas00/bash-tools) |
  47. [conda](https://github.com/stas00/conda-tools) |
  48. [git](https://github.com/stas00/git-tools) |
  49. [jupyter-notebook](https://github.com/stas00/jupyter-notebook-tools) |
  50. [make](https://github.com/stas00/make-tools) |
  51. [python](https://github.com/stas00/python-tools) |
  52. [tensorboard](https://github.com/stas00/tensorboard-tools) |
  53. [unix](https://github.com/stas00/unix-tools)
  54. ✔ **Other Machine Learning:**
  55. [ML ways](https://github.com/stas00/ml-ways) |
  56. [Porting](https://github.com/stas00/porting)

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.