Skip to content
Back to search

precise

SkillWorks with Claude CodeGitHub

Online (incremental) covariance, correlation, and precision estimation in Python — the streaming complement to sklearn.covariance. Use when code needs a covariance/correlation matrix updated per observation, recomputes np.cov/np.corrcoef in a rolling loop, must judge or compare covariance estimates, or proposes a new covariance methodology. Points to task-specific skills.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/microprediction--precise/badge.svg)](/listings/microprediction--precise)

HTML

<a href="/listings/microprediction--precise"><img src="/api/listings/microprediction--precise/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

SKILL.md · Manifest · 54 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. ---
  2. name: precise
  3. description: Online (incremental) covariance, correlation, and precision estimation in Python — the streaming complement to sklearn.covariance. Use when code needs a covariance/correlation matrix updated per observation, recomputes np.cov/np.corrcoef in a rolling loop, must judge or compare covariance estimates, or proposes a new covariance methodology. Points to task-specific skills.
  4. ---
  5. # precise
  6. [`precise`](https://github.com/microprediction/precise) is a small, numpy-only library of **online
  7. (incremental) covariance and correlation estimators** behind one sklearn-style `partial_fit` contract —
  8. plus a panel of assessors for scoring an estimate and a recommender for choosing one. It is the streaming
  9. complement to `sklearn.covariance`, whose estimators are batch-only.
  10. ```bash
  11. pip install precise
  12. ```
  13. ```python
  14. from precise import EwaCovariance
  15. est = EwaCovariance(r=0.05)
  16. for y in stream: # y is one observation (1-D)
  17. est.partial_fit(y)
  18. est.covariance_ # symmetric PSD; also .correlation_ / .precision_ / .location_
  19. ```
  20. ## Reach for precise when you see
  21. - a covariance/correlation matrix being **recomputed in a rolling loop** (`np.cov` / `np.corrcoef`,
  22. `pandas .rolling().cov()`) — that is O(window) per step; precise updates in O(1)–O(d²);
  23. - a need for `partial_fit` covariance where `sklearn.covariance` only offers batch `fit`;
  24. - streaming data **keyed by name** with a universe that changes over time (assets entering/leaving);
  25. - shrinkage / robust / factor covariance wanted online (Ledoit–Wolf, OAS, Huber, Tyler, factor models);
  26. - someone **judging or comparing** covariance estimates, or **proposing a new** covariance method.
  27. ## Task-specific skills
  28. Fetch the relevant one for copy-pasteable code and guardrails:
  29. - **Estimate online** — <https://github.com/microprediction/precise/blob/main/.claude/skills/estimate-online-covariance/SKILL.md>
  30. - **Choose an estimator for your data** — <https://github.com/microprediction/precise/blob/main/.claude/skills/choose-covariance-estimator/SKILL.md>
  31. - **Score / compare estimates** (and the high-dimensional pitfalls) — <https://github.com/microprediction/precise/blob/main/.claude/skills/score-covariance-estimate/SKILL.md>
  32. - **Keyed / dynamic universe** (names that enter and leave) — <https://github.com/microprediction/precise/blob/main/.claude/skills/keyed-dynamic-universe/SKILL.md>
  33. - **Assess a new methodology** (rigorous, honest protocol) — <https://github.com/microprediction/precise/blob/main/.claude/skills/assess-covariance-method/SKILL.md>
  34. ## One guardrail worth knowing up front
  35. In high dimensions (variables comparable to observations), **do not rank covariance estimates by the
  36. held-out Gaussian log-likelihood** — it is dominated by unidentifiable small eigenvalues and ranks below
  37. chance. Use inversion-free / block judges instead (see the scoring skill). Background:
  38. <https://precise.microprediction.org/papers/schur-likelihood/>.
  39. ## Reference
  40. Docs <https://precise.microprediction.org> · PyPI <https://pypi.org/project/precise/> ·
  41. Repo <https://github.com/microprediction/precise>.
README.md · README · 247 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. # precise
  2. [![PyPI](https://img.shields.io/pypi/v/precise)](https://pypi.org/project/precise/)
  3. [![ci](https://github.com/microprediction/precise/actions/workflows/ci.yml/badge.svg)](https://github.com/microprediction/precise/actions/workflows/ci.yml)
  4. [![coverage ≥95%](https://img.shields.io/badge/coverage-%E2%89%A595%25-brightgreen)](https://github.com/microprediction/precise/actions/workflows/ci.yml)
  5. [![Python versions](https://img.shields.io/pypi/pyversions/precise)](https://pypi.org/project/precise/)
  6. [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/microprediction/precise/blob/main/LICENSE)
  7. <!-- The coverage badge states the floor that CI enforces (pytest --cov-fail-under=95 in ci.yml). -->
  8. **Online (incremental) covariance and correlation estimation** — the streaming complement to
  9. [`sklearn.covariance`](https://scikit-learn.org/stable/modules/covariance.html), whose estimators
  10. are batch-only and have no `partial_fit`. Pure Python + numpy; no other required dependencies. And
  11. because no estimator wins everywhere, `precise` also **assesses** an estimate and **recommends** one
  12. for your data (see [Assess &amp; recommend](#assess--recommend)).
  13. 📖 Docs: **[precise.microprediction.org](https://precise.microprediction.org)**
  14. ```bash
  15. pip install precise
  16. ```
  17. ## Who it is for
  18. Researchers and engineers who need a covariance, correlation or precision matrix that is kept up
  19. to date as data arrives, rather than recomputed from scratch: in statistics, econometrics,
  20. empirical finance, signal processing and online machine learning. It is also a test bed for
  21. *studying* covariance estimators: twenty estimators share one contract, so they can be compared on
  22. the same stream, scored with the same assessors, and swapped without changing the calling code.
  23. Research uses so far:
  24. - the author's unrefereed working papers
  25. [*Schur Covariance Evaluation*](https://precise.microprediction.org/papers/schur-likelihood/)
  26. (scoring covariance estimates in high dimensions),
  27. [*Two Sides of Schur Damping*](https://precise.microprediction.org/papers/two-sides-of-schur-damping/)
  28. ([arXiv:2606.14798](https://arxiv.org/abs/2606.14798))
  29. and the note [*Spectral Calibration Without a
  30. Split*](https://precise.microprediction.org/papers/online-spectral-calibration/), with
  31. reproduction scripts in [`research/`](https://github.com/microprediction/precise/tree/main/research);
  32. - [river](https://github.com/online-ml/river), the online machine-learning library, which ported
  33. four of precise's estimators into `river.covariance` (see [Comparison](#comparison-with-other-packages)).
  34. ## Use
  35. ```python
  36. import numpy as np
  37. from precise import EwaCovariance
  38. stream = np.random.default_rng(0).standard_normal((500, 4)) # stand-in for your data
  39. est = EwaCovariance(r=0.05)
  40. for y in stream: # y is a 1d observation; pass a 2d array for a batch
  41. est.partial_fit(y)
  42. est.covariance_ # (n, n) ndarray
  43. est.correlation_ # unit-diagonal correlation
  44. est.precision_ # inverse covariance
  45. est.location_ # running mean
  46. est.fit(X) # sklearn-style batch drop-in (X is 2d)
  47. ```
  48. Every estimator is **truly online** — a constant amount of work per observation, no growing
  49. buffers. State is a plain dict, so you can checkpoint mid-stream with `get_state()` / `set_state()`.
  50. ## Estimators
  51. | Class | What it does |
  52. |---|---|
  53. | `EmpiricalCovariance` | running sample covariance (Welford) |
  54. | `DiagonalCovariance` | variances only (independent variables) |
  55. | `BlockCovariance` | block-diagonal, with O(p·b) state for very large p |
  56. | `EwaCovariance` | exponentially weighted (recency-biased) |
  57. | `AdaptiveEwaCovariance` | EWMA whose forgetting rate speeds up on regime change |
  58. | `LedoitWolfCovariance` | online Ledoit-Wolf shrinkage towards a scaled identity |
  59. | `OASCovariance` | online Oracle Approximating Shrinkage (often better-conditioned than LW) |
  60. | `ShrunkCovariance` | fixed-intensity shrinkage to identity **or** a constant-correlation target |
  61. | `NonlinearShrinkageCovariance` | analytical *nonlinear* spectrum shrinkage — each eigenvalue moved separately, sample eigenvectors kept |
  62. | `WindowedNonlinearShrinkageCovariance` | the same map over a rolling window — forgets, and equal weights keep the asymptotics exact |
  63. | `EwaNonlinearShrinkageCovariance` | the same map over an exponentially weighted sample — approximate, but no window boundary to echo shocks |
  64. | `PartialMomentsCovariance` | exponentially weighted partial-moment (semi-)covariance |
  65. | `HuberCovariance` | online robust estimator that downweights outliers |
  66. | `TylerCovariance` | recursive Tyler M-estimator — robust correlation/shape for elliptical data |
  67. | `GeodesicEwaCovariance` | recency-weighted update along the affine-invariant SPD geodesic |
  68. | `DCCCovariance` | dynamic conditional correlation — decouples volatility from correlation |
  69. | `FactorCovariance` | online low-rank + diagonal (approximate factor model); O(d·k) per step |
  70. | `SchurCovariance` | *experimental*: damps cross-block coupling by a fixed `gamma` |
  71. | `SchurLedoitWolfCovariance` | *experimental*: cross-block damping set by a Ledoit-Wolf reliability estimate |
  72. | `SchurConditionalCovariance` | *experimental*: damped, ridge-regressed block conditionals |
  73. **Experimental** marks methods that are new with this package: they have not been peer-reviewed or
  74. validated outside it, and their behaviour may change between minor releases. The same applies to
  75. the `SchurLikelihood` assessor and to `suggest()` below. Everything else implements an
  76. established method.
  77. ```python
  78. from precise import all_estimators, estimator_from_name
  79. all_estimators() # the list of classes (a bake-off in one loop)
  80. estimator_from_name("LedoitWolfCovariance")
  81. ```
  82. ## Keyed / dynamic universes (river-style)
  83. In streaming/finance settings observations arrive as **dicts keyed by name**, and the set of names
  84. can change over time. `keyed(...)` decorates *any* of the estimators above to consume keyed dicts
  85. (river-style `update` / `learn_one`) and emit keyed output:
  86. ```python
  87. from precise import keyed, EwaCovariance
  88. d = keyed(EwaCovariance(r=0.05), dynamic=True) # changing universe (DynamicUniverse)
  89. d.update({"AAPL": 0.01, "MSFT": -0.02})
  90. d.update({"MSFT": 0.00, "NVDA": 0.03}) # AAPL leaves, NVDA enters
  91. d.covariance_ # dict-of-dicts over the live universe
  92. d.to_frame() # pandas DataFrame (pip install precise[pandas])
  93. k = keyed(EwaCovariance(r=0.05)) # fixed universe, imputes missing keys (FixedUniverse)
  94. ```
  95. `dynamic=False` (the default) gives a `FixedUniverse` (one wrapped estimator, missing keys imputed);
  96. `dynamic=True` gives a `DynamicUniverse` (a wrapped estimator per live key-set). Both work with any
  97. positional estimator — the adapter adds no covariance math of its own.
  98. ## Composing volatility × correlation
  99. `H = D R D` is a composition, not a fixed algorithm. `ConditionalCovariance` lets you pick the
  100. per-series **volatility** model and the **correlation** estimator independently — `DCCCovariance` is
  101. just the EWMA/EWMA special case:
  102. ```python
  103. from precise import ConditionalCovariance, EwaCovariance, LedoitWolfCovariance
  104. est = ConditionalCovariance(vol=EwaCovariance(r=0.02), # any estimator, used per series in 1-D
  105. corr=LedoitWolfCovariance(r=0.05)) # correlation from any estimator
  106. ```
  107. The volatility model can also be any univariate model from
  108. [microprediction/skaters](https://github.com/microprediction/skaters) (Holt, Hosking, …) via
  109. `from_skater` — precise doesn't depend on it; the adapter is duck-typed:
  110. ```python
  111. import skaters
  112. from precise import ConditionalCovariance, from_skater
  113. est = ConditionalCovariance(vol=from_skater(skaters.holt), corr=EwaCovariance(r=0.05))
  114. ```
  115. ## Assess & recommend
  116. No estimator wins everywhere, so `precise` treats *judging* and *choosing* an estimate as
  117. first-class alongside producing one.
  118. ```python
  119. from precise import all_assessors, suggest
  120. all_assessors() # scoring rules: LogLikelihood, BlockPseudoLikelihood, SchurLikelihood,
  121. # SteinLoss, FrobeniusToTruth, GMVVariance, ... (higher = better)
  122. suggest(X, top=3) # recommend estimator classes from observable features of X
  123. ```
  124. `suggest` (*experimental*) maps truth-free features of your data (p/n, effective rank, sphericity,
  125. condition number, off-diagonal mass, excess kurtosis) to an estimator, via a frozen, numpy-only
  126. decision tree trained on simulated data. Its own benchmarks find that choosing per data set is
  127. close to, and sometimes worse than, always using one good estimator, except when the number of
  128. variables approaches the number of observations, so treat its answer as a starting point. The
  129. **Schur pseudo-likelihood** (*experimental*), a one-parameter (`γ`) bridge between the full and
  130. block-diagonal Gaussian likelihoods, is both an assessor here and the subject of an unrefereed
  131. [working paper](https://github.com/microprediction/precise/blob/main/papers/schur_likelihood_paper.pdf).
  132. ## Comparison with other packages
  133. - **[`sklearn.covariance`](https://scikit-learn.org/stable/modules/covariance.html)** is the
  134. reference batch implementation (empirical, Ledoit-Wolf, OAS, shrunk, minimum covariance
  135. determinant, graphical lasso). None of its estimators has `partial_fit`: each needs the whole
  136. sample at once. precise follows its naming and attribute conventions (`covariance_`,
  137. `precision_`, `location_`), so moving between the two is easy.
  138. - **[`river.covariance`](https://riverml.xyz/)** is the closest relative. River is a general
  139. online machine-learning library, and its covariance module had `EmpiricalCovariance` and
  140. `EmpiricalPrecision`. In July 2026 river
  141. [PR #1923](https://github.com/online-ml/river/pull/1923) ported precise's `EwaCovariance`,
  142. `LedoitWolfCovariance`, `OASCovariance` and `ShrunkCovariance` into `river.covariance`, crediting
  143. precise, and added `EwaPrecision`, which tracks the inverse directly by Sherman-Morrison updates
  144. (precise computes `precision_` by inverting on read). They shipped in river 0.26.0. If you
  145. already use river and need one of those, use river's: its estimators are dict-native, take
  146. mini-batches from any narwhals dataframe, and fit river pipelines. River chose to take only estimators that never
  147. invert a stored matrix on read, so what precise still offers beyond it is:
  148. - estimators river does not have: nonlinear spectral shrinkage (expanding, rolling-window and
  149. exponentially weighted), robust M-estimators (Huber, Tyler), DCC and composed
  150. volatility × correlation models, an online factor model, partial-moment (semi-)covariance,
  151. adaptive forgetting, geodesic updates, block-diagonal estimators and the experimental Schur
  152. family;
  153. - one numpy contract shared by all twenty, with a registry, so a bake-off is one loop;
  154. - mid-stream checkpointing: every estimator's state is a plain, JSON-friendly dict;
  155. - assessors for scoring an estimate out of sample, and a recommender;
  156. - keyed adapters that turn *any* of the estimators into a dict-keyed one over a universe whose
  157. names enter and leave.
  158. - **[skfolio](https://skfolio.org)** and **PyPortfolioOpt** include covariance estimators as part of
  159. portfolio optimization, in batch form.
  160. ## Development install
  161. ```bash
  162. git clone https://github.com/microprediction/precise.git
  163. cd precise
  164. python -m venv .venv && source .venv/bin/activate
  165. pip install -e ".[dev]"
  166. pytest # run the test suite
  167. ruff check precise tests research # lint
  168. mypy precise # type check
  169. ```
  170. See [CONTRIBUTING.md](https://github.com/microprediction/precise/blob/main/CONTRIBUTING.md) for the full
  171. workflow.
  172. ## Contributing and support
  173. Questions, bug reports and pull requests are welcome on
  174. [GitHub issues](https://github.com/microprediction/precise/issues). The maintainer answers on a
  175. best-effort basis, and reports of silently wrong results come first. See
  176. [CONTRIBUTING.md](https://github.com/microprediction/precise/blob/main/CONTRIBUTING.md) for how to report a problem or propose a change, and the
  177. [Code of Conduct](https://github.com/microprediction/precise/blob/main/CODE_OF_CONDUCT.md).
  178. Much of the 1.x code was written with AI coding assistants under the author's direction;
  179. [AI_USE.md](https://github.com/microprediction/precise/blob/main/AI_USE.md) says how, and how the code is checked.
  180. ## Citing
  181. If you use precise in your research, please cite it. [CITATION.cff](https://github.com/microprediction/precise/blob/main/CITATION.cff) holds the
  182. metadata, and GitHub's "Cite this repository" button formats it. For example:
  183. > Cotton, P. (2026). *precise: online covariance, correlation and precision matrix estimation in
  184. > Python* (version 1.1.0) [Computer software]. https://github.com/microprediction/precise
  185. ## Related
  186. - **Generating** random covariance/correlation matrices to test against: [`randomcov`](https://github.com/microprediction/randomcov).
  187. - **Portfolio construction** (Schur-complementary allocation, HRP) is no longer part of precise:
  188. it moved out at 1.0 to [allocation.microprediction.org](https://allocation.microprediction.org),
  189. and for production use the [skfolio](https://skfolio.org/auto_examples/clustering/plot_6_schur.html)
  190. implementation is recommended. precise estimates covariance; it does not build portfolios.
  191. - A [Robust Portfolio Literature Reading List](https://github.com/microprediction/precise/blob/main/LITERATURE.md) lives in this repo.
  192. - Part of the [microprediction](https://github.com/microprediction/microprediction) project.
  193. > Migrating from precise &lt; 1.0 (the functional "skater" API)? See [MIGRATING.md](https://github.com/microprediction/precise/blob/main/MIGRATING.md).
  194. ## Disclaimer
  195. Not investment advice. Just code, subject to the MIT License.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.