Skip to content
Back to search

OmicsClaw

Conversational & memory-enabled AI research partner for multi-omics analysis. CLI + Desktop App (installers in Releases). From biological idea to full research paper.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

Its scripts were only partly scanned.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/TianGzlab/OmicsClaw with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/TianGzlab/OmicsClaw'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/tiangzlab--omicsclaw/badge.svg)](/listings/tiangzlab--omicsclaw)

HTML

<a href="/listings/tiangzlab--omicsclaw"><img src="/api/listings/tiangzlab--omicsclaw/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

git clone https://github.com/TianGzlab/OmicsClaw.git
cd OmicsClaw
bash 0_setup_env.sh
conda activate OmicsClaw
oc list
oc run spatial-preprocess --demo

Configure chat and runtime settings:

oc onboard
oc interactive

If `oc` is not on `PATH`, use `python omicsclaw.py <command>`.

<p align="center"> </p>

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 242 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. <a id="top"></a>
  2. <div align="center">
  3. <a href="https://github.com/TianGzlab/OmicsClaw">
  4. <img src="docs/images/OmicsClaw_banner.jpeg" alt="OmicsClaw banner" width="100%"/>
  5. </a>
  6. <h3>Local-first AI research partner for multi-omics analysis</h3>
  7. <p>
  8. <b>English</b> ·
  9. <a href="README_zh-CN.md">简体中文</a> ·
  10. <a href="#quick-start">Quick start</a> ·
  11. <a href="#desktop-app">Desktop App</a> ·
  12. <a href="#domains">Domains</a> ·
  13. <a href="https://TianGzlab.github.io/OmicsClaw/">Docs site</a>
  14. </p>
  15. [![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)
  16. [![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
  17. [![Website](https://img.shields.io/badge/Website-Live-brightgreen.svg)](https://TianGzlab.github.io/OmicsClaw/)
  18. [![Desktop App](https://img.shields.io/github/v/release/TianGzlab/OmicsClaw?label=desktop%20app&color=blue&cacheSeconds=600)](https://github.com/TianGzlab/OmicsClaw/releases/latest)
  19. [![Installer Downloads](https://img.shields.io/github/downloads/TianGzlab/OmicsClaw/total?label=installer%20downloads&color=brightgreen&cacheSeconds=600)](https://github.com/TianGzlab/OmicsClaw/releases)
  20. [![Platform](https://img.shields.io/badge/platform-macOS%20%7C%20Windows%20%7C%20Linux-lightgrey)](https://github.com/TianGzlab/OmicsClaw/releases/latest)
  21. </div>
  22. OmicsClaw is an AI agent for multi-omics analysis. You describe the analysis
  23. you want; the model picks a skill, reads its instructions and runs its Python,
  24. R or command-line tools in your own environment, on your machine or on a
  25. server. It ships 88 skills in seven domains, and the same agent answers in the
  26. terminal, in the desktop app and on chat platforms.
  27. ## What's new
  28. - Across seven domains, 83 skills expose function libraries and replayable examples; five remain CLI-only, including the demo-only XCMS stub, and Python/R modules support IO tracking and review before acceptance ([0074](docs/plans/0074-singlecell-skill-migration.md), [0075](docs/plans/0075-spatial-skill-migration.md), [0076](docs/plans/0076-remaining-modalities-migration.md)).
  29. - A real-model routing eval (`tests/evals/live/`, run by hand) measures whether the model picks the right skill for 26 seed requests, the CI unit-test job now also runs the launch shell, attachments, top-level and desktop HTTP tests, and the four consensus skills whose scripts could not start have since been deleted ([0068](docs/plans/0068-eval-hardening.md), [0070](docs/plans/0070-notebook-skill-runtime.md)).
  30. - The agent framework has been rebuilt around three entry points, `oc cli`, `oc desktop` and `oc channel`, and the old `oc interactive`, `oc tui`, `oc onboard` and `oc run` commands are gone ([rebuild status](docs/FRAMEWORK-REBUILD.md)).
  31. - [Desktop v0.1.8](https://github.com/TianGzlab/OmicsClaw/releases/tag/v0.1.8) is available for macOS, Windows and Linux, with Desktop v3, SSH session recovery and an optional Independent review button below each completed reply.
  32. - The agent tells you which of a skill's packages your environment lacks, and with `OMICSCLAW_SKILL_ENV=install` it can install them after you approve, into a separate overlay that leaves the base environment untouched ([0061](docs/plans/0061-adaptive-env-provisioning.md)).
  33. Earlier entries are in [CHANGELOG.md](CHANGELOG.md).
  34. ## Quick start
  35. ```bash
  36. git clone https://github.com/TianGzlab/OmicsClaw.git
  37. cd OmicsClaw
  38. bash 0_setup_env.sh # creates the OmicsClaw conda environment
  39. conda activate OmicsClaw
  40. oc cli --configure # choose a model provider and enter its API key
  41. oc cli # chat in the terminal
  42. ```
  43. `oc cli --configure` saves its answers to a `.env` file. It asks for one of 13
  44. providers (DeepSeek, OpenAI, Anthropic, Gemini, Qwen on DashScope, Ollama, any
  45. OpenAI-compatible endpoint and others), the API key, the model, an optional
  46. base URL, and the workspace directory the agent may read and write. The agent
  47. loads skills from `<workspace>/skills`, so if the workspace is not the
  48. checkout, also set `OMICSCLAW_SKILLS_DIR=<checkout>/skills` in `.env`.
  49. `oc cli --prompt "..."` answers one question and exits. If `oc` is not on
  50. `PATH`, use `python omicsclaw.py cli`.
  51. Skills are ordinary scripts, so you can also run one without the agent:
  52. ```bash
  53. make list # every skill, by domain
  54. python skills/spatial/spatial-preprocess/spatial_preprocess.py --demo --output /tmp/omicsclaw_demo
  55. ```
  56. ## Ways to use it
  57. | Where | Start it with | What you get |
  58. |---|---|---|
  59. | Terminal | `oc cli` | Chat with the agent. `/resume` reopens an earlier conversation, `/skills` lists skills, `/help` lists the other commands |
  60. | Desktop App | the App starts `oc desktop` | Chat, approvals and file viewing in a desktop window ([below](#desktop-app)) |
  61. | Remote server | `oc desktop` on the server | The App reaches it through its own SSH tunnel, so the data stays on the server ([remote mode](docs/engineering/remote-execution.mdx)) |
  62. | Chat platforms | `oc channel --channels telegram` | Owner-only text chat on Telegram, Feishu, Slack, Discord, DingTalk, QQ or Email. `oc channel --list` shows them. Set `OMICSCLAW_APPROVAL_TIMEOUT_S` in `.env` first, or it refuses to start |
  63. | Skill scripts | `python <skill directory>/<script>.py` | One analysis without the agent. Each skill's `SKILL.md` documents its options |
  64. Conversations and the agent's long-term notes are stored in
  65. `<workspace>/.omicsclaw/memory.db`. MCP servers listed in
  66. `<workspace>/.mcp.json` are connected at start-up, and their tools become
  67. available to the agent.
  68. The agent asks you before it runs a shell command, writes or edits a file,
  69. goes to the web, or calls an MCP tool. In the terminal you can allow a call once, allow that tool for
  70. the rest of the conversation, or always allow that exact call; `/auto` stops
  71. the questions about ordinary work. Dangerous commands and changes to
  72. `.omicsclaw/` or `.env` are still asked about. The dangerous-command check
  73. matches known patterns, and a command written to avoid them gets through, so if
  74. you let the agent work unattended, run its commands in a container without
  75. network access by setting `OMICSCLAW_SANDBOX=docker` and
  76. `OMICSCLAW_SANDBOX_IMAGE` (see [`.env.example`](.env.example)).
  77. ## Desktop App
  78. <p align="center">
  79. <img src="docs/images/omicsclaw-app-overview.png" alt="The OmicsClaw desktop app, with a project's conversations in the sidebar and the chat box in the middle" width="94%"/>
  80. </p>
  81. Download [Desktop v0.1.8](https://github.com/TianGzlab/OmicsClaw/releases/tag/v0.1.8) for your platform below.
  82. Check each download against the
  83. [`SHA256SUMS.txt`](https://github.com/TianGzlab/OmicsClaw/releases/download/v0.1.8/SHA256SUMS.txt)
  84. published with it.
  85. | Platform | Installer |
  86. |---|---|
  87. | macOS, Apple Silicon | [OmicsClaw-0.1.8-arm64.dmg](https://github.com/TianGzlab/OmicsClaw/releases/download/v0.1.8/OmicsClaw-0.1.8-arm64.dmg) |
  88. | macOS, Intel | [OmicsClaw-0.1.8-x64.dmg](https://github.com/TianGzlab/OmicsClaw/releases/download/v0.1.8/OmicsClaw-0.1.8-x64.dmg) |
  89. | Windows, x64 or ARM64 | [OmicsClaw.Setup.0.1.8-x64.exe](https://github.com/TianGzlab/OmicsClaw/releases/download/v0.1.8/OmicsClaw.Setup.0.1.8-x64.exe), [OmicsClaw.Setup.0.1.8-arm64.exe](https://github.com/TianGzlab/OmicsClaw/releases/download/v0.1.8/OmicsClaw.Setup.0.1.8-arm64.exe) |
  90. | Linux, x64 | [AppImage](https://github.com/TianGzlab/OmicsClaw/releases/download/v0.1.8/OmicsClaw-0.1.8-x86_64.AppImage), [deb](https://github.com/TianGzlab/OmicsClaw/releases/download/v0.1.8/OmicsClaw-0.1.8-amd64.deb), [rpm](https://github.com/TianGzlab/OmicsClaw/releases/download/v0.1.8/OmicsClaw-0.1.8-x86_64.rpm) |
  91. | Linux, ARM64 | [AppImage](https://github.com/TianGzlab/OmicsClaw/releases/download/v0.1.8/OmicsClaw-0.1.8-arm64.AppImage) |
  92. The installer contains no Python. **Remote mode** connects to a server through
  93. SSH or a backend URL and needs no Python on the client. **Local mode** asks for
  94. an interpreter with OmicsClaw installed, normally the conda environment from
  95. the quick start:
  96. ```bash
  97. conda run -n OmicsClaw python -c "import sys; print(sys.executable)"
  98. ```
  99. In Local mode, the App starts `oc desktop` for the project you open, one backend per
  100. project. The model comes from the backend's `.env`, which the App's Providers
  101. page can edit; a change takes effect when the backend restarts. Troubleshooting
  102. is in the [App guide](docs/ecosystem/omicsclaw-app.mdx), and the HTTP contract
  103. in [`docs/core-features/surfaces.md`](docs/core-features/surfaces.md) §8.
  104. Independent module review is off by default. After a reply finishes, click
  105. **Independent review** below it to review that reply's analysis and return
  106. the findings in the same conversation. The click applies only to that review;
  107. it does not enable reviews for later analyses. Use App v0.1.8 or later, update
  108. the backend to include [`1540fca7`](https://github.com/TianGzlab/OmicsClaw/commit/1540fca71bc3dbf9ba83557aafe964860a6b81dc),
  109. and restart `oc desktop` to use this control.
  110. ## Installation
  111. | Method | Command | Covers |
  112. |---|---|---|
  113. | conda (recommended) | `bash 0_setup_env.sh` | Python, R and the command-line tools the skills need, plus the Desktop backend |
  114. | pip | `pip install -e .` | Terminal chat only. Most skills need packages that the conda environment provides, and the Desktop backend's FastAPI and uvicorn come from conda |
  115. Chat platforms need their own SDKs: `pip install -e ".[channels]"` for
  116. Telegram and Feishu, `pip install slack-sdk aiohttp` for Slack,
  117. `pip install discord.py` for Discord, `pip install httpx websockets` for
  118. DingTalk and `pip install qq-botpy` for QQ; Email uses only the standard
  119. library. The dependency lists are [`environment.yml`](environment.yml) and
  120. [`pyproject.toml`](pyproject.toml).
  121. ## Domains
  122. 88 skills in seven domains. `make list` prints the current index.
  123. | Domain | Skills | Examples | Guide |
  124. |---|---|---|---|
  125. | Spatial transcriptomics | 17 | QC, domains, annotation, deconvolution, CNV, trajectory | [spatial](docs/domains/spatial.mdx) |
  126. | Single-cell omics | 30 | QC, clustering, annotation, doublets, velocity, GRN | [singlecell](docs/domains/singlecell.mdx) |
  127. | Genomics | 10 | QC, alignment, variants, CNV, assembly, epigenomics | [genomics](docs/domains/genomics.mdx) |
  128. | Proteomics | 8 | DIA/DDA, PTM, networks, biomarkers | [proteomics](docs/domains/proteomics.mdx) |
  129. | Metabolomics | 8 | Peaks, normalization, annotation, pathways | [metabolomics](docs/domains/metabolomics.mdx) |
  130. | Bulk RNA-seq | 14 | DE, enrichment, co-expression, deconvolution, survival, cosinor rhythms | [bulkrna](docs/domains/bulkrna.mdx) |
  131. | Literature | 1 | PDF, DOI, PubMed and GEO parsing, dataset handoff | |
  132. ## Documentation
  133. The [docs site](https://TianGzlab.github.io/OmicsClaw/) is written in Chinese.
  134. | Topic | Where |
  135. |---|---|
  136. | Quickstart | [introduction/quickstart](docs/introduction/quickstart.mdx) |
  137. | Desktop App | [ecosystem/omicsclaw-app](docs/ecosystem/omicsclaw-app.mdx) |
  138. | Remote mode | [engineering/remote-execution](docs/engineering/remote-execution.mdx) |
  139. | Domain guides | [spatial](docs/domains/spatial.mdx) · [singlecell](docs/domains/singlecell.mdx) · [genomics](docs/domains/genomics.mdx) · [proteomics](docs/domains/proteomics.mdx) · [metabolomics](docs/domains/metabolomics.mdx) · [bulkrna](docs/domains/bulkrna.mdx) |
  140. | Safety and data privacy | [data privacy](docs/safety/data-privacy.mdx) · [rules and disclaimer](docs/safety/rules-and-disclaimer.mdx) |
  141. | Writing a skill | [CONTRIBUTING.md](CONTRIBUTING.md) · [`templates/skill/`](templates/skill/) |
  142. | Framework design | [`docs/FRAMEWORK-REBUILD.md`](docs/FRAMEWORK-REBUILD.md) |
  143. | Changelog | [CHANGELOG.md](CHANGELOG.md) |
  144. ## Safety and data
  145. Skills read and process your data in your own runtime, local or remote; the
  146. model receives the conversation and what the tools return. OmicsClaw is a
  147. research tool. It is not a medical device and gives no clinical diagnoses, so
  148. have a domain expert check results before acting on them. On a server, keep
  149. `oc desktop` bound to localhost and reach it through SSH; a non-loopback bind
  150. requires `OMICSCLAW_REMOTE_AUTH_TOKEN`. See [data privacy](docs/safety/data-privacy.mdx)
  151. and [rules and disclaimer](docs/safety/rules-and-disclaimer.mdx).
  152. ## Community
  153. Maintainers: Luyi Tian (Principal Investigator), Weige Zhou (Lead Developer), Liying Chen (Developer), and Pengfei Yin (Developer).
  154. [Issues](https://github.com/TianGzlab/OmicsClaw/issues) · [Discussions](https://github.com/TianGzlab/OmicsClaw/discussions) · [Docs](https://TianGzlab.github.io/OmicsClaw/)
  155. <table>
  156. <tr>
  157. <td align="center" width="30%">
  158. <img src="docs/images/IMG_3729.JPG" alt="OmicsClaw WeChat group QR code" width="180"/>
  159. <br/>
  160. WeChat group
  161. </td>
  162. <td valign="middle" width="70%">
  163. Scan the code to join our WeChat group for analysis tips and help with problems.
  164. </td>
  165. </tr>
  166. </table>
  167. <a href="https://github.com/TianGzlab/OmicsClaw/graphs/contributors">
  168. <img src="https://contrib.rocks/image?repo=TianGzlab/OmicsClaw" alt="OmicsClaw contributors"/>
  169. </a>
  170. ## Acknowledgments
  171. The architecture, skill design and local-first approach are inspired by
  172. [ClawBio](https://github.com/ClawBio/ClawBio), an early bioinformatics-native
  173. AI agent skill library. The memory and session-continuity patterns are inspired
  174. by [Nocturne Memory](https://github.com/Dataojitori/nocturne_memory).
  175. ## Contributing
  176. To add a skill, see [CONTRIBUTING.md](CONTRIBUTING.md) and the scaffold under
  177. [`templates/skill/`](templates/skill/). For work on the framework itself, start
  178. with [AGENTS.md](AGENTS.md).
  179. ## License
  180. Apache-2.0. See [LICENSE](LICENSE).
  181. ## Citation
  182. ```bibtex
  183. @software{omicsclaw2026,
  184. title = {OmicsClaw: A Memory-Enabled AI Agent for Multi-Omics Analysis},
  185. author = {Zhou, Weige and Chen, Liying and Yin, Pengfei and Tian, Luyi},
  186. year = {2026},
  187. url = {https://github.com/TianGzlab/OmicsClaw}
  188. }
  189. ```
  190. [Back to top](#top)
skills/singlecell/scrna/sc-cell-annotation/_api.py · Bundled script · 1,167 lines · 1 finding marked
  1. """sc-cell-annotation's function library: label cells with cell types.
  2. In a step: ``annotation = load_skill("sc-cell-annotation")``. Every annotate
  3. function writes ``obs['cell_type']`` (and ``obs['annotation_score']`` when the
  4. method gives one) in place and returns the same AnnData; :func:`run_info`
  5. returns what the run did, including any fallback. The functions read and write
  6. no files except a marker file or reference you name. ``singler`` and ``scmap``
  7. run R through ``Rscript``.
  8. """
  9. from __future__ import annotations
  10. import json
  11. import logging
  12. import tempfile
  13. from pathlib import Path
  14. import numpy as np
  15. import pandas as pd
  16. import scanpy as sc
  17. from skills._sdk.deps import validate_r_environment
  18. from skills._sdk.r_script_runner import R_SCRIPTS_DIR as _SDK_R_SCRIPTS_DIR
  19. from skills._sdk.r_script_runner import RScriptRunner
  20. from skills.singlecell._lib import annotation as sc_annotation_utils
  21. from skills.singlecell._lib.adata_utils import get_matrix_contract
  22. __all__ = [
  23. "annotate",
  24. "annotate_markers",
  25. "annotate_manual",
  26. "annotate_celltypist",
  27. "annotate_popv",
  28. "annotate_knnpredict",
  29. "annotate_singler",
  30. "annotate_scmap",
  31. "annotate_scsa",
  32. "run_info",
  33. "annotation_table",
  34. "cluster_annotation_matrix",
  35. "annotation_figure",
  36. ]
  37. logger = logging.getLogger(__name__)
  38. METHODS = ("markers", "manual", "celltypist", "popv", "knnpredict", "singler", "scmap", "scsa")
  39. _RUN_KEY = "omicsclaw_sc_cell_annotation_run"
  40. def _json_default(value):
  41. if hasattr(value, "item"):
  42. return value.item()
  43. if hasattr(value, "tolist"):
  44. return value.tolist()
  45. return str(value)
  46. def _remember(adata, cluster_key, summary: dict):
  47. adata.uns[_RUN_KEY] = json.dumps({"cluster_key": cluster_key, "summary": summary}, default=_json_default)
  48. return adata
  49. def annotate(
  50. adata,
  51. *,
  52. method: str = "markers",
  53. cluster_key: str | None = None,
  54. markers: dict[str, list[str]] | None = None,
  55. marker_file: str | None = None,
  56. model: str = "Immune_All_Low",
  57. majority_voting: bool = False,
  58. reference: str = "HPCA",
  59. manual_map: str | None = None,
  60. manual_map_file: str | None = None,
  61. species: str = "Human",
  62. tissue: str = "All",
  63. scsa_foldchange: float = 1.5,
  64. scsa_pvalue: float = 0.05,
  65. ):
  66. """Annotate cell types with the chosen method; the single entry point to the method functions.
  67. :param method: One of ``markers`` (default), ``manual``, ``celltypist``, ``popv``,
  68. ``knnpredict``, ``singler``, ``scmap``, ``scsa``. See the method functions for
  69. what each needs.
  70. :param cluster_key: The ``obs`` column with cluster labels. Default ``None``: the
  71. matrix contract's primary cluster key, else the first of ``leiden``,
  72. ``louvain``, ``seurat_clusters``, ``cluster``, ``cell_type`` present.
  73. :param markers: ``markers`` only: cell type to marker genes. Default: the built-in human set.
  74. :param marker_file: ``markers`` only: a JSON or CSV marker file, used instead of *markers*.
  75. :param model: ``celltypist`` only. Default ``"Immune_All_Low"``.
  76. :param majority_voting: ``celltypist`` only. Default ``False``.
  77. :param reference: ``popv``, ``knnpredict``: a labelled ``.h5ad`` path; ``singler``,
  78. ``scmap``: a celldex atlas name. Default ``"HPCA"``.
  79. :param manual_map: ``manual`` only: inline mapping such as ``"0=T cell;1,2=Myeloid"``.
  80. :param manual_map_file: ``manual`` only: a mapping file (json/csv/tsv/txt).
  81. :param species: ``scsa`` only: ``"Human"`` or ``"Mouse"``. Default ``"Human"``.
  82. :param tissue: ``scsa`` only: CellMarker tissue filter. Default ``"All"``.
  83. :param scsa_foldchange: ``scsa`` only. Default 1.5.
  84. :param scsa_pvalue: ``scsa`` only. Default 0.05.
  85. :returns: The same AnnData.
  86. :raises ValueError: an unknown method, or what the chosen method raises.
  87. """
  88. if method not in METHODS:
  89. raise ValueError(f"unknown annotation method {method!r}; choose one of {', '.join(METHODS)}")
  90. if method == "markers":
  91. return annotate_markers(adata, cluster_key=cluster_key, markers=markers, marker_file=marker_file)
  92. if method == "manual":
  93. return annotate_manual(adata, cluster_key=cluster_key, manual_map=manual_map, manual_map_file=manual_map_file)
  94. if method == "celltypist":
  95. return annotate_celltypist(adata, model=model, majority_voting=majority_voting, cluster_key=cluster_key)
  96. if method == "popv":
  97. return annotate_popv(adata, reference=reference, cluster_key=cluster_key)
  98. if method == "knnpredict":
  99. return annotate_knnpredict(adata, reference=reference, cluster_key=cluster_key)
  100. if method == "singler":
  101. return annotate_singler(adata, reference=reference, cluster_key=cluster_key)
  102. if method == "scmap":
  103. return annotate_scmap(adata, reference=reference, cluster_key=cluster_key)
  104. return annotate_scsa(adata, cluster_key=cluster_key, species=species, tissue=tissue,
  105. foldchange=scsa_foldchange, pvalue=scsa_pvalue)
  106. def annotate_markers(adata, *, cluster_key: str | None = None, markers: dict[str, list[str]] | None = None,
  107. marker_file: str | None = None):
  108. """Label each cluster with the cell type whose marker genes have the highest mean expression in it.
  109. Every cell of a cluster gets the cluster's label; a cluster where no marker
  110. gene is expressed is ``Unknown``. Gene names are matched exactly, or
  111. case-insensitively when nothing matches exactly (human markers on mouse data).
  112. :param cluster_key: As in :func:`annotate`.
  113. :param markers: Cell type to marker genes. Default: the built-in human set
  114. (PBMC, brain and general stromal types). Give tissue-specific markers for other tissues.
  115. :param marker_file: A JSON (``{"T cell": ["CD3D", ...]}``) or CSV (``T cell,CD3D;CD3E``)
  116. marker file, used instead of *markers*.
  117. :returns: The same AnnData with ``obs['cell_type']`` and ``obs['annotation_score']``
  118. (the winning mean expression).
  119. :raises ValueError: the cluster column is missing.
  120. """
  121. key = _resolve_cluster_key(adata, cluster_key) or "leiden"
  122. return _remember(adata, key, _annotate_markers(adata, markers=markers, cluster_key=key, marker_file=marker_file))
  123. def annotate_manual(adata, *, cluster_key: str | None = None, manual_map: str | None = None,
  124. manual_map_file: str | None = None):
  125. """Relabel clusters from a mapping the user gives.
  126. :param cluster_key: As in :func:`annotate`.
  127. :param manual_map: Inline mapping such as ``"0=T cell;1,2=Myeloid"``.
  128. :param manual_map_file: A mapping file (json, csv, tsv or txt), used instead of *manual_map*.
  129. :returns: The same AnnData with ``obs['cell_type']``.
  130. :raises ValueError: neither mapping is given.
  131. """
  132. key = _resolve_cluster_key(adata, cluster_key)
  133. return _remember(adata, key, _annotate_manual(adata, cluster_key=key, manual_map=manual_map,
  134. manual_map_file=manual_map_file))
  135. def annotate_celltypist(adata, *, model: str = "Immune_All_Low", majority_voting: bool = False,
  136. cluster_key: str | None = None):
  137. """Per-cell labels from a pretrained CellTypist model.
  138. CellTypist needs log1p-normalised expression to 10,000 counts; when the input
  139. fails that check, or the model cannot be loaded, the function falls back to
  140. :func:`annotate_markers` and records the reason (``run_info(adata)["summary"]["fallback_reason"]``).
  141. :param model: A CellTypist model name or ``.pkl``. Default ``"Immune_All_Low"`` (immune
  142. cells, fine labels); choose a tissue model for other data. Models download on first use.
  143. :param majority_voting: Smooth labels over over-clustered communities. Default ``False``.
  144. :param cluster_key: Recorded for later steps; CellTypist labels cells individually.
  145. :returns: The same AnnData with ``obs['cell_type']``, ``obs['annotation_score']`` and
  146. ``obsm['cell_type_prob']``.
  147. """
  148. key = _resolve_cluster_key(adata, cluster_key)
  149. return _remember(adata, key, _annotate_celltypist(adata, model, majority_voting=majority_voting))
  150. def annotate_popv(adata, *, reference: str = "HPCA", cluster_key: str | None = None):
  151. """Labels transferred from a labelled reference AnnData by PopV-style consensus.
  152. :param reference: Path to a labelled ``.h5ad`` (a ``cell_type`` column in ``obs``).
  153. :param cluster_key: As in :func:`annotate`; used for the cluster consensus.
  154. :returns: The same AnnData with ``obs['cell_type']``.
  155. """
  156. key = _resolve_cluster_key(adata, cluster_key)
  157. return _remember(adata, key, _annotate_popv(adata, reference, cluster_key=key))
  158. def annotate_knnpredict(adata, *, reference: str = "HPCA", cluster_key: str | None = None):
  159. """Labels transferred from a labelled reference AnnData by nearest neighbours (SCOP KNNPredict style).
  160. :param reference: Path to a labelled ``.h5ad`` (a ``cell_type`` column in ``obs``).
  161. :param cluster_key: As in :func:`annotate`.
  162. :returns: The same AnnData with ``obs['cell_type']`` and ``obs['annotation_score']``.
  163. """
  164. key = _resolve_cluster_key(adata, cluster_key)
  165. return _remember(adata, key, _annotate_knnpredict(adata, reference, cluster_key=key))
  166. def annotate_singler(adata, *, reference: str = "HPCA", cluster_key: str | None = None):
  167. """Per-cell labels from SingleR against a celldex atlas, run in R.
  168. :param reference: A celldex atlas: ``HPCA`` (default), ``BlueprintEncode``, ``Monaco``, ...
  169. :param cluster_key: Recorded for later steps; SingleR labels cells individually.
  170. :returns: The same AnnData with ``obs['cell_type']`` and ``obs['annotation_score']``.
  171. :raises RuntimeError: R, SingleR or celldex is missing, or SingleR returns nothing.
  172. """
  173. key = _resolve_cluster_key(adata, cluster_key)
  174. return _remember(adata, key, _annotate_singler(adata, reference))
  175. def annotate_scmap(adata, *, reference: str = "HPCA", cluster_key: str | None = None):
  176. """Per-cell labels projected with scmap onto a celldex atlas, run in R.
  177. :param reference: A celldex atlas. Default ``HPCA``.
  178. :param cluster_key: Recorded for later steps.
  179. :returns: The same AnnData with ``obs['cell_type']``.
  180. :raises RuntimeError: R, scmap or celldex is missing, or scmap returns nothing.
  181. """
  182. key = _resolve_cluster_key(adata, cluster_key)
  183. return _remember(adata, key, _annotate_scmap(adata, reference))
  184. def annotate_scsa(adata, *, cluster_key: str | None = None, species: str = "Human", tissue: str = "All",
  185. foldchange: float = 1.5, pvalue: float = 0.05):
  186. """Cluster labels from CellMarker 2.0 genes scored against each cluster's markers (SCSA).
  187. Runs a Wilcoxon test per cluster and scores each cell type's markers with a
  188. Fisher exact test. The CellMarker table downloads once to
  189. ``~/.cache/omicsclaw/scsa``; without network a small built-in set is used.
  190. :param cluster_key: As in :func:`annotate`.
  191. :param species: ``"Human"`` (default) or ``"Mouse"``.
  192. :param tissue: CellMarker tissue filter such as ``"Blood"``. Default ``"All"``.
  193. :param foldchange: Minimum fold change of a cluster marker. Default 1.5.
  194. :param pvalue: Maximum adjusted p-value of a cluster marker. Default 0.05.
  195. :returns: The same AnnData with ``obs['cell_type']``.
  196. """
  197. key = _resolve_cluster_key(adata, cluster_key)
  198. return _remember(adata, key, _annotate_scsa(adata, cluster_key=key, species=species, tissue=tissue,
  199. foldchange=foldchange, pvalue=pvalue))
  200. def run_info(adata, *, keep: bool = True) -> dict:
  201. """What the last annotate function recorded: ``cluster_key`` and ``summary``.
  202. ``summary`` has ``requested_method``, ``actual_method``, ``used_fallback``,
  203. ``fallback_reason``, ``n_cell_types``, ``cell_type_counts`` and the method's
  204. own details (reference, marker source, ...).
  205. :param keep: Leave the record in ``adata.uns``; ``False`` removes it.
  206. :returns: The record, or an empty dict when no annotate function has run on *adata*.
  207. """
  208. raw = adata.uns.get(_RUN_KEY) if keep else adata.uns.pop(_RUN_KEY, None)
  209. return json.loads(raw) if raw else {}
  210. def annotation_table(adata, *, key: str = "cell_type") -> pd.DataFrame:
  211. """Cells per cell type, largest first.
  212. :param key: The ``obs`` column with the labels. Default ``"cell_type"``.
  213. :returns: Columns ``cell_type``, ``n_cells`` and ``proportion_pct``.
  214. """
  215. counts = adata.obs[key].astype(str).value_counts()
  216. if counts.empty:
  217. return pd.DataFrame(columns=["cell_type", "n_cells"])
  218. df = pd.DataFrame({"cell_type": [str(k) for k in counts.index], "n_cells": [int(v) for v in counts.values]})
  219. df["proportion_pct"] = (df["n_cells"] / max(int(df["n_cells"].sum()), 1) * 100).round(2)
  220. return df.sort_values(["n_cells", "cell_type"], ascending=[False, True]).reset_index(drop=True)
  221. def cluster_annotation_matrix(adata, *, cluster_key: str, key: str = "cell_type") -> pd.DataFrame:
  222. """For each cluster, the fraction of its cells given each label.
  223. :param cluster_key: The ``obs`` column with cluster labels.
  224. :param key: The ``obs`` column with cell-type labels. Default ``"cell_type"``.
  225. :returns: One row per cluster (first column named after *cluster_key*), one column per label;
  226. empty when either column is missing.
  227. """
  228. if cluster_key not in adata.obs.columns or key not in adata.obs.columns:
  229. return pd.DataFrame()
  230. matrix = pd.crosstab(adata.obs[cluster_key].astype(str), adata.obs[key].astype(str), normalize="index")
  231. matrix.index.name = cluster_key
  232. return matrix.reset_index()
  233. def annotation_figure(adata, *, key: str = "cell_type", basis: str | None = None):
  234. """A scatter plot of the 2-D embedding coloured by cell type.
  235. :param key: The ``obs`` column to colour by. Default ``"cell_type"``.
  236. :param basis: The ``obsm`` key to plot. Default: the first of ``X_umap``, ``X_tsne``, ``X_pca``.
  237. :returns: A matplotlib Figure.
  238. :raises KeyError: no embedding is present.
  239. """
  240. import matplotlib.pyplot as plt
  241. name = basis or next((k for k in ("X_umap", "X_tsne", "X_pca") if k in adata.obsm), None)
  242. if name is None:
  243. raise KeyError("no embedding in obsm to plot")
  244. coords = np.asarray(adata.obsm[name])[:, :2]
  245. labels = adata.obs[key].astype(str)
  246. fig, ax = plt.subplots(figsize=(7, 5))
  247. palette = plt.get_cmap("tab20")
  248. for index, label in enumerate(labels.value_counts().index):
  249. mask = (labels == label).to_numpy()
  250. ax.scatter(coords[mask, 0], coords[mask, 1], s=4, color=palette(index % 20), label=label, linewidths=0)
  251. ax.set_xlabel(f"{name[2:]}1")
  252. ax.set_ylabel(f"{name[2:]}2")
  253. ax.set_title(f"{name[2:]} coloured by {key}")
  254. ax.legend(markerscale=3, fontsize=7, frameon=False, bbox_to_anchor=(1.0, 1.0), loc="upper left")
  255. fig.tight_layout()
  256. return fig
  257. BUILTIN_MARKERS_HUMAN: dict[str, list[str]] = {
  258. # --- Blood / PBMC ---
  259. "CD4+ T cell": ["CD3D", "CD3E", "CD4", "IL7R"],
  260. "CD8+ T cell": ["CD3D", "CD3E", "CD8A", "CD8B"],
  261. "Regulatory T cell": ["FOXP3", "IL2RA", "CTLA4"],
  262. "B cell": ["MS4A1", "CD79A", "CD79B", "CD19"],
  263. "Plasma cell": ["MZB1", "SDC1", "IGHA1", "JCHAIN"],
  264. "NK cell": ["GNLY", "NKG7", "KLRD1", "NCAM1"],
  265. "CD14+ Monocyte": ["CD14", "LYZ", "S100A9", "S100A8"],
  266. "CD16+ Monocyte": ["FCGR3A", "MS4A7"],
  267. "Dendritic cell": ["FCER1A", "CD1C", "CLEC10A"],
  268. "Platelet": ["PPBP", "PF4"],
  269. # --- Brain ---
  270. "Neuron": ["SNAP25", "SYT1", "RBFOX3", "STMN2"],
  271. "Astrocyte": ["AQP4", "GFAP", "SLC1A3"],
  272. "Oligodendrocyte": ["MBP", "PLP1", "MOG"],
  273. "Microglia": ["CX3CR1", "P2RY12", "CSF1R"],
  274. "OPC": ["PDGFRA", "CSPG4", "OLIG2"],
  275. # --- General tissue / stroma ---
  276. "Epithelial": ["EPCAM", "KRT18", "KRT19"],
  277. "Fibroblast": ["COL1A1", "COL1A2", "DCN", "LUM"],
  278. "Endothelial": ["PECAM1", "VWF", "CDH5"],
  279. "Smooth muscle cell": ["ACTA2", "TAGLN", "MYH11"],
  280. "Macrophage": ["CD68", "CD163", "MRC1"],
  281. "Mast cell": ["KIT", "TPSAB1", "TPSB2"],
  282. }
  283. def _load_marker_file(path: str | Path) -> dict[str, list[str]]:
  284. """Load custom marker genes from a JSON or CSV file.
  285. JSON format::
  286. {"T cell": ["CD3D", "CD3E"], "B cell": ["MS4A1", "CD79A"]}
  287. CSV format (two columns, no header or header ``cell_type,markers``)::
  288. T cell,CD3D;CD3E;CD4
  289. B cell,MS4A1;CD79A
  290. """
  291. p = Path(path)
  292. if not p.exists():
  293. raise FileNotFoundError(f"Marker file not found: {p}")
  294. text = p.read_text(encoding="utf-8").strip()
  295. if p.suffix == ".json":
  296. markers = json.loads(text)
  297. else:
  298. markers: dict[str, list[str]] = {}
  299. for line in text.splitlines():
  300. line = line.strip()
  301. if not line or line.startswith("#") or line.lower().startswith("cell_type"):
  302. continue
  303. parts = line.split(",", 1)
  304. if len(parts) != 2:
  305. continue
  306. cell_type = parts[0].strip().strip('"').strip("'")
  307. genes = [g.strip().strip('"').strip("'") for g in parts[1].replace(";", ",").split(",") if g.strip()]
  308. if cell_type and genes:
  309. markers[cell_type] = genes
  310. if not markers:
  311. raise ValueError(f"No valid marker entries found in {p}")
  312. return markers
  313. def _detect_species_hint(var_names) -> str:
  314. """Heuristic species detection from gene naming convention.
  315. Human genes: UPPER (CD3D, MS4A1). Mouse genes: Title case (Cd3d, Ms4a1).
  316. """
  317. sample = list(var_names[:min(500, len(var_names))])
  318. if not sample:
  319. return "unknown"
  320. upper_count = sum(1 for g in sample if g == g.upper())
  321. title_count = sum(1 for g in sample if g != g.upper() and g[0].isupper())
  322. ratio_upper = upper_count / len(sample)
  323. ratio_title = title_count / len(sample)
  324. if ratio_upper > 0.7:
  325. return "human"
  326. if ratio_title > 0.5:
  327. return "mouse"
  328. return "unknown"
  329. def _build_case_insensitive_map(var_names) -> dict[str, str]:
  330. """Map UPPER gene names -> actual var_names for case-insensitive matching."""
  331. return {g.upper(): g for g in var_names}
  332. # ---------------------------------------------------------------------------
  333. # Method implementations
  334. # ---------------------------------------------------------------------------
  335. def _record_annotation_execution(
  336. adata,
  337. *,
  338. requested_method: str,
  339. actual_method: str,
  340. fallback_reason: str = "",
  341. ) -> None:
  342. adata.obs["annotation_requested_method"] = requested_method
  343. adata.obs["annotation_actual_method"] = actual_method
  344. adata.obs["annotation_method"] = actual_method
  345. adata.uns["annotation_runtime"] = {
  346. "requested_method": requested_method,
  347. "actual_method": actual_method,
  348. "used_fallback": bool(fallback_reason),
  349. "fallback_reason": fallback_reason,
  350. }
  351. def _annotation_summary(
  352. adata,
  353. *,
  354. requested_method: str,
  355. actual_method: str,
  356. fallback_reason: str = "",
  357. expression_source: str | None = None,
  358. ) -> dict:
  359. counts = adata.obs["cell_type"].astype(str).value_counts().to_dict()
  360. summary = {
  361. "method": actual_method,
  362. "requested_method": requested_method,
  363. "actual_method": actual_method,
  364. "used_fallback": bool(fallback_reason),
  365. "fallback_reason": fallback_reason,
  366. "n_cell_types": len(counts),
  367. "cell_type_counts": {str(k): int(v) for k, v in counts.items()},
  368. }
  369. if expression_source:
  370. summary["expression_source"] = expression_source
  371. return summary
  372. def _candidate_cluster_keys(adata) -> list[str]:
  373. matrix_contract = get_matrix_contract(adata)
  374. candidates: list[str] = []
  375. primary = matrix_contract.get("primary_cluster_key")
  376. if primary and primary in adata.obs.columns:
  377. candidates.append(str(primary))
  378. for key in ("leiden", "louvain", "seurat_clusters", "cluster", "cell_type"):
  379. if key in adata.obs.columns and key not in candidates:
  380. candidates.append(key)
  381. return candidates
  382. def _resolve_cluster_key(adata, cluster_key: str | None) -> str | None:
  383. if cluster_key and cluster_key in adata.obs.columns:
  384. return cluster_key
  385. candidates = _candidate_cluster_keys(adata)
  386. return candidates[0] if candidates else None
  387. def _annotate_markers(adata, markers=None, cluster_key: str = "leiden", marker_file: str | None = None):
  388. """Marker-based annotation with gene-overlap detection and species awareness."""
  389. # ---- 1. Resolve marker source ----
  390. if marker_file:
  391. markers = _load_marker_file(marker_file)
  392. marker_source = f"custom file: {marker_file}"
  393. elif markers is not None:
  394. marker_source = "caller-provided"
  395. else:
  396. markers = BUILTIN_MARKERS_HUMAN
  397. marker_source = "built-in (human, multi-tissue)"
  398. expression_source, expr = "adata.X", adata
  399. if cluster_key not in adata.obs:
  400. raise ValueError(
  401. f"Marker-based annotation requires an existing cluster/label column. "
  402. f"`{cluster_key}` was not found in adata.obs."
  403. )
  404. # ---- 2. Gene overlap diagnostic ----
  405. all_marker_genes = sorted({g for genes in markers.values() for g in genes})
  406. expr_var_names = set(expr.var_names)
  407. matched_genes = [g for g in all_marker_genes if g in expr_var_names]
  408. overlap_rate = len(matched_genes) / len(all_marker_genes) if all_marker_genes else 0.0
  409. case_remap: dict[str, str] | None = None
  410. species_hint = _detect_species_hint(expr.var_names)
  411. if overlap_rate == 0:
  412. # Attempt case-insensitive rescue (human markers vs mouse gene names)
  413. upper_map = _build_case_insensitive_map(expr.var_names)
  414. case_matched = [g for g in all_marker_genes if g.upper() in upper_map]
  415. if case_matched:
  416. case_remap = {g: upper_map[g.upper()] for g in all_marker_genes if g.upper() in upper_map}
  417. logger.warning(
  418. "0/%d marker genes matched by exact name, but %d/%d matched case-insensitively. "
  419. "Dataset appears to use %s gene naming (detected species: %s). "
  420. "Proceeding with case-insensitive matching.",
  421. len(all_marker_genes), len(case_matched), len(all_marker_genes),
  422. "Title-case" if species_hint == "mouse" else "non-UPPER",
  423. species_hint,
  424. )
  425. else:
  426. logger.warning(
  427. "NONE of the %d built-in marker genes were found in the dataset (%d genes). "
  428. "Detected species hint: %s. Marker source: %s.\n"
  429. " This usually means the built-in markers do not match your tissue/organism.\n"
  430. " Solutions:\n"
  431. " 1. Provide custom markers: --marker-file markers.json\n"
  432. " 2. Use CellTypist: --method celltypist --model <model_name>\n"
  433. " 3. Use reference mapping: --method knnpredict --reference <ref.h5ad>\n"
  434. " See SKILL.md 'Reference Data Guide' for download instructions.",
  435. len(all_marker_genes), len(expr_var_names), species_hint, marker_source,
  436. )
  437. elif overlap_rate < 0.3:
  438. logger.warning(
  439. "Only %d/%d (%.0f%%) marker genes found in the dataset. "
  440. "Annotation quality may be limited. Species hint: %s.",
  441. len(matched_genes), len(all_marker_genes), overlap_rate * 100, species_hint,
  442. )
  443. else:
  444. logger.info(
  445. "Marker gene overlap: %d/%d (%.0f%%). Species hint: %s. Source: %s.",
  446. len(matched_genes), len(all_marker_genes), overlap_rate * 100,
  447. species_hint, marker_source,
  448. )
  449. # ---- 3. Score each cluster ----
  450. cluster_annotations: dict[str, str] = {}
  451. cluster_scores: dict[str, float] = {}
  452. for cluster in adata.obs[cluster_key].astype(str).unique():
  453. cluster_mask = adata.obs[cluster_key].astype(str) == cluster
  454. cluster_data = adata[cluster_mask]
  455. best_type = "Unknown"
  456. best_score = 0.0
  457. for cell_type, marker_genes in markers.items():
  458. if case_remap:
  459. available = [case_remap[g] for g in marker_genes if g in case_remap]
  460. else:
  461. available = [g for g in marker_genes if g in expr_var_names]
  462. if not available:
  463. continue
  464. score = float(np.asarray(cluster_data[:, available].X.mean()).item())
  465. if score > best_score:
  466. best_score = score
  467. best_type = cell_type
  468. cluster_annotations[cluster] = best_type
  469. cluster_scores[cluster] = best_score
  470. # ---- 4. Detect all-Unknown and warn ----
  471. unknown_clusters = [c for c, t in cluster_annotations.items() if t == "Unknown"]
  472. if len(unknown_clusters) == len(cluster_annotations):
  473. logger.error(
  474. "ALL %d clusters were annotated as 'Unknown' — the markers did not match any genes "
  475. "in this dataset. This is almost certainly because the built-in markers are not "
  476. "appropriate for your tissue type or organism.\n"
  477. " Recommended actions:\n"
  478. " 1. --marker-file markers.json (provide tissue-specific markers)\n"
  479. " 2. --method celltypist --model <model> (100+ pre-trained models)\n"
  480. " Run: python -c \"import celltypist; celltypist.models.models_description()\" to list models\n"
  481. " 3. --method knnpredict --reference <ref.h5ad> (your own labeled reference)\n"
  482. " See SKILL.md 'Reference Data Guide' for details.",
  483. len(cluster_annotations),
  484. )
  485. elif unknown_clusters:
  486. logger.warning(
  487. "%d/%d clusters annotated as 'Unknown' (clusters: %s). "
  488. "Consider providing more specific markers via --marker-file.",
  489. len(unknown_clusters), len(cluster_annotations),
  490. ", ".join(unknown_clusters[:10]),
  491. )
  492. adata.obs["cell_type"] = adata.obs[cluster_key].astype(str).map(cluster_annotations)
  493. adata.obs["annotation_score"] = adata.obs[cluster_key].astype(str).map(cluster_scores).astype(float)
  494. _record_annotation_execution(
  495. adata,
  496. requested_method="markers",
  497. actual_method="markers",
  498. )
  499. logger.info(
  500. "Annotated %d clusters (%d cell types, %d Unknown). Marker source: %s.",
  501. len(cluster_annotations),
  502. len(set(cluster_annotations.values()) - {"Unknown"}),
  503. len(unknown_clusters),
  504. marker_source,
  505. )
  506. return _annotation_summary(
  507. adata,
  508. requested_method="markers",
  509. actual_method="markers",
  510. expression_source=expression_source,
  511. )
  512. def _annotate_manual(adata, *, cluster_key: str, manual_map: str | None = None, manual_map_file: str | None = None):
  513. """Apply an explicit user-supplied cluster-to-label mapping."""
  514. if manual_map_file:
  515. annotations = sc_annotation_utils.load_manual_annotation_map(manual_map_file)
  516. mapping_source = str(manual_map_file)
  517. elif manual_map:
  518. annotations = sc_annotation_utils.parse_manual_annotation_map(manual_map)
  519. mapping_source = "inline"
  520. else:
  521. raise ValueError("Manual annotation requires --manual-map or --manual-map-file.")
  522. sc_annotation_utils.annotate_clusters_manual(
  523. adata,
  524. annotations=annotations,
  525. cluster_key=cluster_key,
  526. annotation_key="cell_type",
  527. inplace=True,
  528. )
  529. adata.obs["annotation_score"] = np.nan
  530. _record_annotation_execution(
  531. adata,
  532. requested_method="manual",
  533. actual_method="manual",
  534. )
  535. summary = _annotation_summary(
  536. adata,
  537. requested_method="manual",
  538. actual_method="manual",
  539. expression_source="manual_mapping",
  540. )
  541. summary["manual_mapping_source"] = mapping_source
  542. summary["manual_mapping"] = annotations
  543. return summary
  544. def _annotate_celltypist(adata, model: str = "Immune_All_Low", majority_voting: bool = False):
  545. """CellTypist annotation with explicit fallback recording."""
  546. celltypist_input, expression_source = sc_annotation_utils.build_celltypist_input_adata(adata)
  547. is_valid, reason = sc_annotation_utils.validate_celltypist_input_matrix(celltypist_input)
  548. if not is_valid:
  549. logger.warning("CellTypist input validation failed: %s", reason)
  550. _annotate_markers(adata)
  551. _record_annotation_execution(
  552. adata,
  553. requested_method="celltypist",
  554. actual_method="markers",
  555. fallback_reason=reason,
  556. )
  557. summary = _annotation_summary(
  558. adata,
  559. requested_method="celltypist",
  560. actual_method="markers",
  561. fallback_reason=reason,
  562. expression_source=expression_source,
  563. )
  564. return summary
  565. try:
  566. model_name = model if model.endswith(".pkl") else f"{model}.pkl"
  567. sc_annotation_utils.annotate_with_celltypist(
  568. celltypist_input,
  569. model=model_name,
  570. majority_voting=majority_voting,
  571. annotation_key="cell_type",
  572. inplace=True,
  573. )
  574. adata.obs["cell_type"] = celltypist_input.obs["cell_type"].values
  575. if "cell_type_score" in celltypist_input.obs.columns:
  576. adata.obs["annotation_score"] = pd.to_numeric(celltypist_input.obs["cell_type_score"], errors="coerce").values
  577. if "cell_type_prob" in celltypist_input.obsm:
  578. adata.obsm["cell_type_prob"] = celltypist_input.obsm["cell_type_prob"]
  579. _record_annotation_execution(
  580. adata,
  581. requested_method="celltypist",
  582. actual_method="celltypist",
  583. )
  584. return _annotation_summary(
  585. adata,
  586. requested_method="celltypist",
  587. actual_method="celltypist",
  588. expression_source=expression_source,
  589. )
  590. except Exception as exc:
  591. reason = str(exc)
  592. logger.warning("CellTypist annotation unavailable (%s); falling back to marker-based annotation", exc)
  593. _annotate_markers(adata)
  594. _record_annotation_execution(
  595. adata,
  596. requested_method="celltypist",
  597. actual_method="markers",
  598. fallback_reason=reason,
  599. )
  600. summary = _annotation_summary(
  601. adata,
  602. requested_method="celltypist",
  603. actual_method="markers",
  604. fallback_reason=reason,
  605. expression_source=expression_source,
  606. )
  607. return summary
  608. def _annotate_popv(adata, reference: str = "HPCA", cluster_key: str = "leiden"):
  609. """PopV-style reference mapping with cluster consensus."""
  610. metadata = sc_annotation_utils.apply_popv_annotation(
  611. adata,
  612. reference,
  613. cluster_key=cluster_key,
  614. )
  615. actual_method = metadata.get("backend", "popv")
  616. _record_annotation_execution(
  617. adata,
  618. requested_method="popv",
  619. actual_method=actual_method,
  620. )
  621. summary = _annotation_summary(
  622. adata,
  623. requested_method="popv",
  624. actual_method=actual_method,
  625. expression_source=metadata.get("expression_source"),
  626. )
  627. summary.update(
  628. {
  629. "backend": metadata.get("backend"),
  630. "reference": reference,
  631. "reference_label_key": metadata.get("reference_label_key"),
  632. "reference_cell_types": metadata.get("reference_cell_types"),
  633. "reference_gene_overlap": metadata.get("reference_gene_overlap"),
  634. "reference_path": metadata.get("reference_path"),
  635. "popv_methods": metadata.get("popv_methods"),
  636. }
  637. )
  638. return summary
  639. def _annotate_knnpredict(adata, reference: str = "HPCA", cluster_key: str = "leiden"):
  640. """Lightweight reference mapping inspired by SCOP KNNPredict."""
  641. metadata = sc_annotation_utils.apply_knnpredict_annotation(
  642. adata,
  643. reference,
  644. cluster_key=cluster_key,
  645. )
  646. _record_annotation_execution(
  647. adata,
  648. requested_method="knnpredict",
  649. actual_method="knnpredict",
  650. )
  651. summary = _annotation_summary(
  652. adata,
  653. requested_method="knnpredict",
  654. actual_method="knnpredict",
  655. expression_source=metadata.get("expression_source"),
  656. )
  657. summary.update(
  658. {
  659. "backend": metadata.get("backend"),
  660. "reference": reference,
  661. "reference_label_key": metadata.get("reference_label_key"),
  662. "reference_cell_types": metadata.get("reference_cell_types"),
  663. "reference_gene_overlap": metadata.get("reference_gene_overlap"),
  664. "reference_path": metadata.get("reference_path"),
  665. }
  666. )
  667. return summary
  668. def _apply_r_annotations(adata, df: pd.DataFrame, *, requested_method: str, actual_method: str) -> dict:
  669. df = df.copy()
  670. if df.empty:
  671. raise RuntimeError(f"R annotation method '{requested_method}' returned no predictions")
  672. df.index = df.index.astype(str)
  673. df = df.reindex(adata.obs_names)
  674. labels = df["pruned_label"].fillna(df["cell_type"]).astype(str)
  675. adata.obs["cell_type"] = labels.values
  676. if "score" in df.columns:
  677. adata.obs["annotation_score"] = pd.to_numeric(df["score"], errors="coerce").values
  678. _record_annotation_execution(
  679. adata,
  680. requested_method=requested_method,
  681. actual_method=actual_method,
  682. )
  683. return _annotation_summary(adata, requested_method=requested_method, actual_method=actual_method)
  684. def _annotate_singler(adata, reference: str = "HPCA"):
  685. """SingleR annotation via the shared R bridge."""
  686. validate_r_environment(required_r_packages=["SingleR", "celldex", "SingleCellExperiment", "zellkonverter"])
  687. export_adata, expression_source = sc_annotation_utils.build_celltypist_input_adata(adata)
  688. scripts_dir = _SDK_R_SCRIPTS_DIR
  689. runner = RScriptRunner(scripts_dir=scripts_dir, timeout=1800)
  690. with tempfile.TemporaryDirectory(prefix="omicsclaw_singler_") as tmpdir:
  691. tmpdir = Path(tmpdir)
  692. input_h5ad = tmpdir / "input.h5ad"
  693. output_dir = tmpdir / "output"
  694. output_dir.mkdir(parents=True, exist_ok=True)
  695. r_home = tmpdir / "r_home"
  696. xdg_cache = tmpdir / "xdg_cache"
  697. eh_cache = tmpdir / "experimenthub"
  698. for path in (r_home, xdg_cache, eh_cache):
  699. path.mkdir(parents=True, exist_ok=True)
  700. export_adata.write_h5ad(input_h5ad)
  701. runner.run_script(
  702. "sc_singler_annotate.R",
  703. args=[str(input_h5ad), str(output_dir), reference],
  704. expected_outputs=["singler_results.csv"],
  705. output_dir=output_dir,
  706. env={
  707. "HOME": str(r_home),
  708. "XDG_CACHE_HOME": str(xdg_cache),
  709. "OMICSCLAW_EXPERIMENTHUB_CACHE": str(eh_cache),
  710. "ZELLKONVERTER_USE_BASILISK": "FALSE",
  711. },
  712. )
  713. df = pd.read_csv(output_dir / "singler_results.csv", index_col=0)
  714. summary = _apply_r_annotations(adata, df, requested_method="singler", actual_method="singler")
  715. summary["expression_source"] = expression_source
  716. summary["reference"] = reference
  717. return summary
  718. def _annotate_scmap(adata, reference: str = "HPCA"):
  719. """scmap annotation via the shared R bridge."""
  720. validate_r_environment(required_r_packages=["scmap", "celldex", "SingleCellExperiment", "zellkonverter"])
  721. export_adata, expression_source = sc_annotation_utils.build_celltypist_input_adata(adata)
  722. scripts_dir = _SDK_R_SCRIPTS_DIR
  723. runner = RScriptRunner(scripts_dir=scripts_dir, timeout=1800)
  724. with tempfile.TemporaryDirectory(prefix="omicsclaw_scmap_") as tmpdir:
  725. tmpdir = Path(tmpdir)
  726. input_h5ad = tmpdir / "input.h5ad"
  727. output_dir = tmpdir / "output"
  728. output_dir.mkdir(parents=True, exist_ok=True)
  729. r_home = tmpdir / "r_home"
  730. xdg_cache = tmpdir / "xdg_cache"
  731. eh_cache = tmpdir / "experimenthub"
  732. for path in (r_home, xdg_cache, eh_cache):
  733. path.mkdir(parents=True, exist_ok=True)
  734. export_adata.write_h5ad(input_h5ad)
  735. runner.run_script(
  736. "sc_scmap_annotate.R",
  737. args=[str(input_h5ad), str(output_dir), reference],
  738. expected_outputs=["scmap_results.csv"],
  739. output_dir=output_dir,
  740. env={
  741. "HOME": str(r_home),
  742. "XDG_CACHE_HOME": str(xdg_cache),
  743. "OMICSCLAW_EXPERIMENTHUB_CACHE": str(eh_cache),
  744. "ZELLKONVERTER_USE_BASILISK": "FALSE",
  745. },
  746. )
  747. df = pd.read_csv(output_dir / "scmap_results.csv", index_col=0)
  748. summary = _apply_r_annotations(adata, df, requested_method="scmap", actual_method="scmap")
  749. summary["expression_source"] = expression_source
  750. summary["reference"] = reference
  751. return summary
  752. def _annotate_scsa(
  753. adata,
  754. cluster_key: str = "leiden",
  755. species: str = "Human",
  756. tissue: str = "All",
  757. foldchange: float = 1.5,
  758. pvalue: float = 0.05,
  759. ):
  760. """SCSA-style annotation: marker DE -> CellMarker database Fisher exact test scoring.
  761. Adapted from pySCSA (Cao et al.). Runs Wilcoxon DE per cluster, then scores
  762. each cluster against the CellMarker database using Fisher's exact test and
  763. Z-score ranking.
  764. """
  765. from scipy.stats import fisher_exact
  766. if cluster_key not in adata.obs.columns:
  767. raise ValueError(
  768. f"SCSA requires a cluster column. '{cluster_key}' not found in adata.obs."
  769. )
  770. # ---- 1. Run Wilcoxon DE to get markers per cluster ----
  771. logger.info("SCSA: running Wilcoxon DE for cluster markers (key=%s)", cluster_key)
  772. sc.tl.rank_genes_groups(adata, cluster_key, method="wilcoxon", use_raw=False)
  773. result = adata.uns["rank_genes_groups"]
  774. groups = result["names"].dtype.names
  775. # Extract significant markers per cluster
  776. cluster_markers: dict[str, list[str]] = {}
  777. for group in groups:
  778. names = result["names"][group]
  779. logfcs = result["logfoldchanges"][group]
  780. pvals = result["pvals_adj"][group]
  781. sig_genes = []
  782. for name, lfc, pval in zip(names, logfcs, pvals):
  783. if abs(float(lfc)) >= foldchange and float(pval) <= pvalue:
  784. sig_genes.append(str(name))
  785. cluster_markers[str(group)] = sig_genes
  786. # ---- 2. Load or build CellMarker database ----
  787. cellmarker_db = _load_scsa_cellmarker_db(species=species, tissue=tissue)
  788. if not cellmarker_db:
  789. logger.error(
  790. "SCSA: CellMarker database is empty for species=%s, tissue=%s. "
  791. "This means no cell type entries matched. Try:\n"
  792. " 1. --species Human or --species Mouse\n"
  793. " 2. --tissue All (to search all tissues)\n"
  794. " 3. --method markers (with custom --marker-file)\n",
  795. species, tissue,
  796. )
  797. # Fall back to Unknown
  798. adata.obs["cell_type"] = "Unknown"
  799. adata.obs["annotation_score"] = 0.0
  800. _record_annotation_execution(adata, requested_method="scsa", actual_method="scsa")
  801. return _annotation_summary(adata, requested_method="scsa", actual_method="scsa")
  802. # ---- 3. Score each cluster using Fisher exact test ----
  803. all_detected_genes = set(adata.var_names)
  804. n_total_genes = len(all_detected_genes)
  805. cluster_annotations: dict[str, str] = {}
  806. cluster_scores: dict[str, float] = {}
  807. for cluster_id, sig_genes in cluster_markers.items():
  808. sig_set = set(sig_genes) & all_detected_genes
  809. if not sig_set:
  810. cluster_annotations[cluster_id] = "Unknown"
  811. cluster_scores[cluster_id] = 0.0
  812. continue
  813. best_type = "Unknown"
  814. best_zscore = -np.inf
  815. n_sig = len(sig_set)
  816. for cell_type, db_genes in cellmarker_db.items():
  817. db_set = set(db_genes) & all_detected_genes
  818. if not db_set:
  819. continue
  820. # Overlap between cluster markers and cell-type markers
  821. overlap = sig_set & db_set
  822. n_overlap = len(overlap)
  823. if n_overlap == 0:
  824. continue
  825. # Fisher exact test (one-sided, greater)
  826. n_db = len(db_set)
  827. contingency = [
  828. [n_overlap, n_sig - n_overlap],
  829. [n_db - n_overlap, n_total_genes - n_sig - n_db + n_overlap],
  830. ]
  831. # Ensure no negative values
  832. contingency = [[max(0, x) for x in row] for row in contingency]
  833. try:
  834. _, p_val = fisher_exact(contingency, alternative="greater")
  835. except Exception:
  836. continue
  837. # Convert to Z-score-like measure: -log10(p) * sign(enrichment)
  838. zscore = -np.log10(max(p_val, 1e-300)) * (n_overlap / max(n_sig, 1))
  839. if zscore > best_zscore:
  840. best_zscore = zscore
  841. best_type = cell_type
  842. cluster_annotations[cluster_id] = best_type
  843. cluster_scores[cluster_id] = float(best_zscore) if best_zscore > -np.inf else 0.0
  844. # ---- 4. Apply annotations ----
  845. adata.obs["cell_type"] = adata.obs[cluster_key].astype(str).map(cluster_annotations)
  846. adata.obs["annotation_score"] = adata.obs[cluster_key].astype(str).map(cluster_scores).astype(float)
  847. _record_annotation_execution(adata, requested_method="scsa", actual_method="scsa")
  848. # ---- 5. Detect degenerate output ----
  849. unknown_clusters = [c for c, t in cluster_annotations.items() if t == "Unknown"]
  850. if len(unknown_clusters) == len(cluster_annotations):
  851. logger.error(
  852. " *** ALL %d clusters were labeled 'Unknown' by SCSA. ***\n"
  853. " This usually means no marker genes overlapped with the CellMarker database.\n"
  854. " How to fix:\n"
  855. " Option 1 -- Adjust DE thresholds:\n"
  856. " --scsa-foldchange 1.0 --scsa-pvalue 0.1\n"
  857. " Option 2 -- Widen tissue filter:\n"
  858. " --tissue All --species Human\n"
  859. " Option 3 -- Use a different method:\n"
  860. " --method markers --marker-file custom_markers.json\n"
  861. " --method celltypist --model <model>.pkl",
  862. len(cluster_annotations),
  863. )
  864. elif unknown_clusters:
  865. logger.warning(
  866. "SCSA: %d/%d clusters annotated as 'Unknown' (clusters: %s).",
  867. len(unknown_clusters), len(cluster_annotations),
  868. ", ".join(unknown_clusters[:10]),
  869. )
  870. logger.info(
  871. "SCSA: annotated %d clusters (%d cell types, %d Unknown). "
  872. "Species=%s, tissue=%s.",
  873. len(cluster_annotations),
  874. len(set(cluster_annotations.values()) - {"Unknown"}),
  875. len(unknown_clusters),
  876. species, tissue,
  877. )
  878. return _annotation_summary(
  879. adata,
  880. requested_method="scsa",
  881. actual_method="scsa",
  882. expression_source="adata.X",
  883. )
  884. def _load_scsa_cellmarker_db(
  885. species: str = "Human",
  886. tissue: str = "All",
  887. ) -> dict[str, list[str]]:
  888. """Load a CellMarker-style database for SCSA annotation.
  889. Downloads and caches CellMarker2.0 data, then filters by species/tissue.
  890. Returns dict mapping cell_type -> list of marker gene symbols.
  891. """
  892. cache_dir = Path.home() / ".cache" / "omicsclaw" / "scsa"
  893. cache_dir.mkdir(parents=True, exist_ok=True)
  894. db_path = cache_dir / "cellmarker2_markers.csv"
  895. if not db_path.exists():
  896. logger.info("SCSA: downloading CellMarker 2.0 database...")
  897. try:
  898. _download_cellmarker_db(db_path)
  899. except Exception as exc:
  900. logger.warning("Failed to download CellMarker database: %s", exc)
  901. logger.info("SCSA: falling back to built-in compact marker database.")
  902. return _builtin_scsa_markers(species)
  903. # Parse the database
  904. try:
  905. db_df = pd.read_csv(db_path, sep=",", encoding="utf-8", on_bad_lines="skip")
  906. except Exception:
  907. try:
  908. db_df = pd.read_csv(db_path, sep="\t", encoding="utf-8", on_bad_lines="skip")
  909. except Exception as exc:
  910. logger.warning("Failed to parse CellMarker database: %s", exc)
  911. return _builtin_scsa_markers(species)
  912. # Detect column names (CellMarker 2.0 format)
  913. species_col = None
  914. tissue_col = None
  915. celltype_col = None
  916. marker_col = None
  917. for col in db_df.columns:
  918. col_lower = col.lower().strip()
  919. if "species" in col_lower or "specie" in col_lower:
  920. species_col = col
  921. elif "tissue" in col_lower and "sub" not in col_lower:
  922. tissue_col = col
  923. elif "cell_name" in col_lower or "cell_type" in col_lower or col_lower == "cellname" or "cell name" in col_lower:
  924. celltype_col = col
  925. elif "marker" in col_lower or "symbol" in col_lower or "gene" in col_lower:
  926. marker_col = col
  927. if not celltype_col or not marker_col:
  928. logger.warning(
  929. "SCSA: CellMarker CSV columns not recognized (%s). Using built-in markers.",
  930. list(db_df.columns),
  931. )
  932. return _builtin_scsa_markers(species)
  933. # Filter by species
  934. if species_col:
  935. species_norm = species.strip().lower()
  936. db_df = db_df[db_df[species_col].astype(str).str.lower().str.contains(species_norm, na=False)]
  937. # Filter by tissue
  938. if tissue_col and tissue.lower() != "all":
  939. tissue_norm = tissue.strip().lower()
  940. db_df = db_df[db_df[tissue_col].astype(str).str.lower().str.contains(tissue_norm, na=False)]
  941. # Build marker dict
  942. result: dict[str, list[str]] = {}
  943. for _, row in db_df.iterrows():
  944. ct = str(row[celltype_col]).strip()
  945. markers_raw = str(row[marker_col]).strip()
  946. if not ct or ct == "nan" or not markers_raw or markers_raw == "nan":
  947. continue
  948. # Markers can be comma-separated, semicolon-separated, or space-separated
  949. genes = [g.strip() for g in markers_raw.replace(";", ",").replace("/", ",").split(",") if g.strip()]
  950. if ct in result:
  951. result[ct].extend(genes)
  952. else:
  953. result[ct] = genes
  954. # Deduplicate
  955. for ct in result:
  956. result[ct] = list(set(result[ct]))
  957. logger.info("SCSA: loaded %d cell types from CellMarker database.", len(result))
  958. return result
  959. def _download_cellmarker_db(db_path: Path) -> None:
  960. """Download CellMarker 2.0 database."""
  961. import urllib.request
  962. urls = [
  963. "http://bio-bigdata.hrbmu.edu.cn/CellMarker/CellMarker_download_files/file/Cell_marker_All.csv",
  964. "http://117.50.127.228/CellMarker/CellMarker_download_files/file/Cell_marker_All.csv",
    • Warning: Talks to a raw IP address or throwaway host, where malware is often staged
    • Line 1,107 on GitHub
  965. ]
  966. for url in urls:
  967. try:
  968. logger.info(" Trying %s ...", url)
  969. urllib.request.urlretrieve(url, str(db_path))
  970. if db_path.exists() and db_path.stat().st_size > 1000:
  971. logger.info(" Download successful: %s (%d bytes)", db_path, db_path.stat().st_size)
  972. return
  973. except Exception as exc:
  974. logger.warning(" Failed: %s", exc)
  975. continue
  976. raise RuntimeError(
  977. "Could not download CellMarker database from any URL.\n"
  978. " Please download manually from http://bio-bigdata.hrbmu.edu.cn/CellMarker/\n"
  979. " and place it at: %s" % db_path
  980. )
  981. def _builtin_scsa_markers(species: str = "Human") -> dict[str, list[str]]:
  982. """Compact built-in markers as fallback when CellMarker DB is unavailable."""
  983. if species.lower() in ("mouse", "mm", "mus musculus"):
  984. # Mouse markers in Title case
  985. return {
  986. "T cell": ["Cd3d", "Cd3e", "Cd3g"],
  987. "B cell": ["Cd79a", "Cd79b", "Ms4a1", "Cd19"],
  988. "NK cell": ["Nkg7", "Klrb1c", "Gzma"],
  989. "Monocyte": ["Cd14", "Lyz2", "Csf1r"],
  990. "Macrophage": ["Cd68", "Adgre1", "Mrc1"],
  991. "Dendritic cell": ["Itgax", "Flt3", "H2-Aa"],
  992. "Neutrophil": ["S100a8", "S100a9", "Ly6g"],
  993. "Epithelial": ["Epcam", "Krt18", "Krt19"],
  994. "Fibroblast": ["Col1a1", "Col1a2", "Dcn"],
  995. "Endothelial": ["Pecam1", "Cdh5", "Vwf"],
  996. }
  997. # Human markers (UPPER)
  998. return {
  999. "T cell": ["CD3D", "CD3E", "CD3G", "CD2"],
  1000. "CD4+ T cell": ["CD3D", "CD4", "IL7R"],
  1001. "CD8+ T cell": ["CD3D", "CD8A", "CD8B"],
  1002. "B cell": ["CD79A", "CD79B", "MS4A1", "CD19"],
  1003. "Plasma cell": ["MZB1", "SDC1", "IGHA1", "JCHAIN"],
  1004. "NK cell": ["GNLY", "NKG7", "KLRD1", "NCAM1"],
  1005. "CD14+ Monocyte": ["CD14", "LYZ", "S100A9", "S100A8"],
  1006. "CD16+ Monocyte": ["FCGR3A", "MS4A7"],
  1007. "Macrophage": ["CD68", "CD163", "MRC1"],
  1008. "Dendritic cell": ["FCER1A", "CD1C", "CLEC10A"],
  1009. "Mast cell": ["KIT", "TPSAB1", "TPSB2"],
  1010. "Platelet": ["PPBP", "PF4"],
  1011. "Neutrophil": ["S100A8", "S100A9", "FCGR3B", "CSF3R"],
  1012. "Epithelial": ["EPCAM", "KRT18", "KRT19"],
  1013. "Fibroblast": ["COL1A1", "COL1A2", "DCN", "LUM"],
  1014. "Endothelial": ["PECAM1", "VWF", "CDH5"],
  1015. "Smooth muscle cell": ["ACTA2", "TAGLN", "MYH11"],
  1016. "Neuron": ["SNAP25", "SYT1", "RBFOX3"],
  1017. "Astrocyte": ["AQP4", "GFAP", "SLC1A3"],
  1018. "Oligodendrocyte": ["MBP", "PLP1", "MOG"],
  1019. "Microglia": ["CX3CR1", "P2RY12", "CSF1R"],
  1020. }

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.