Skip to content
Back to search

lance

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/lance-format/lance with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/lance-format/lance'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/lance-format--lance/badge.svg)](/listings/lance-format--lance)

HTML

<a href="/listings/lance-format--lance"><img src="/api/listings/lance-format--lance/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

**Installation**

pip install pylance

To install a preview release:

pip install --pre --extra-index-url https://pypi.fury.io/lance-format pylance

> [!TIP] > Preview releases are released more often than full releases and contain the > latest features and bug fixes. They receive the same level of testing as full releases. > We guarantee they will remain published and available for download for at > least 6 months. When you want to pin to a specific version, prefer a stable release.

**Converting to Lance**

import lance

import pandas as pd
import pyarrow as pa
import pyarrow.dataset

df = pd.DataFrame({"a": [5], "b": [10]})
uri = "/tmp/test.parquet"
tbl = pa.Table.from_pandas(df)
pa.dataset.write_dataset(tbl, uri, format='parquet')

parquet = pa.dataset.dataset(uri, format='parquet')
lance.write_dataset(parquet, "/tmp/test.lance")

**Reading Lance data**

dataset = lance.dataset("/tmp/test.lance")
assert isinstance(dataset, pa.dataset.Dataset)

**Pandas**

df = dataset.to_table().to_pandas()
df

**DuckDB**

import duckdb

# If this segfaults, make sure you have duckdb v0.7+ installed
duckdb.query("SELECT * FROM dataset LIMIT 10").to_df()

**Vector search**

Download the sift1m subset

wget ftp://ftp.irisa.fr/local/texmex/corpus/sift.tar.gz
tar -xzf sift.tar.gz

Convert it to Lance

import lance
from lance.vector import vec_to_table
import numpy as np
import struct

nvecs = 1000000
ndims = 128
with open("sift/sift_base.fvecs", mode="rb") as fobj:
    buf = fobj.read()
    data = np.array(struct.unpack("<128000000f", buf[4 : 4 + 4 * nvecs * ndims])).reshape((nvecs, ndims))
    dd = dict(zip(range(nvecs), data))

table = vec_to_table(dd)
uri = "vec_data.lance"
sift1m = lance.write_dataset(table, uri, max_rows_per_group=8192, max_rows_per_file=1024*1024)

Build the index

sift1m.create_index("vector",
                    index_type="IVF_PQ",
                    num_partitions=256,  # IVF
                    num_sub_vectors=16)  # PQ

Search the dataset

# Get top 10 similar vectors
import duckdb

dataset = lance.dataset(uri)

# Sample 100 query vectors. If this segfaults, make sure you have duckdb v0.7+ installed
sample = duckdb.query("SELECT vector FROM dataset USING SAMPLE 100").to_df()
query_vectors = np.array([np.array(x) for x in sample.vector])

# Get nearest neighbors for all of them
rs = [dataset.to_table(nearest={"column": "vector", "k": 10, "q": q})
      for q in query_vectors]

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 272 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. <div align="center">
  2. <p align="center">
  3. <img width="257" alt="Lance Logo" src="https://user-images.githubusercontent.com/917119/199353423-d3e202f7-0269-411d-8ff2-e747e419e492.png">
  4. **The Open Lakehouse Format for Multimodal AI**<br/>
  5. **High-performance vector search, full-text search, random access, and feature engineering capabilities for the lakehouse.**<br/>
  6. **Compatible with Pandas, DuckDB, Polars, PyArrow, Ray, Spark, and more integrations on the way.**
  7. <a href="https://lance.org">Documentation</a> •
  8. <a href="https://lance.org/community">Community</a> •
  9. <a href="https://discord.gg/lance">Discord</a> •
  10. <a href="https://groups.google.com/a/lance.org/g/dev">Mailing List</a>
  11. [CI]: https://github.com/lance-format/lance/actions/workflows/rust.yml
  12. [CI Badge]: https://github.com/lance-format/lance/actions/workflows/rust.yml/badge.svg
  13. [Docs]: https://lance.org
  14. [Docs Badge]: https://img.shields.io/badge/docs-passing-brightgreen
  15. [crates.io]: https://crates.io/crates/lance
  16. [crates.io badge]: https://img.shields.io/crates/v/lance.svg
  17. [Python versions]: https://pypi.org/project/pylance/
  18. [Python versions badge]: https://img.shields.io/pypi/pyversions/pylance
  19. [![CI Badge]][CI]
  20. [![Docs Badge]][Docs]
  21. [![crates.io badge]][crates.io]
  22. [![Python versions badge]][Python versions]
  23. </p>
  24. </div>
  25. <hr />
  26. Lance is an open lakehouse format for multimodal AI. It contains a file format, table format, and catalog spec that allows you to build a complete lakehouse on top of object storage to power your AI workflows. Lance is perfect for:
  27. 1. Building search engines and feature stores with hybrid search capabilities.
  28. 2. Large-scale ML training requiring high performance IO and random access.
  29. 3. Storing, querying, and managing multimodal data including images, videos, audio, text, and embeddings.
  30. The key features of Lance include:
  31. - **Expressive hybrid search:** Combine vector similarity search, full-text search (BM25), and SQL analytics on the same dataset with accelerated secondary indices.
  32. - **Lightning-fast random access:** 100x faster than Parquet or Iceberg for random access without sacrificing scan performance.
  33. - **Native multimodal data support:** Store images, videos, audio, text, and embeddings in a single unified format with efficient blob encoding and lazy loading.
  34. - **Data evolution:** Efficiently add columns with backfilled values without full table rewrites, perfect for ML feature engineering.
  35. - **Zero-copy versioning:** Automatic versioning with ACID transactions, time travel, tags, and branches—no extra infrastructure needed.
  36. - **Rich ecosystem integrations:** Apache Arrow, Pandas, Polars, DuckDB, Apache Spark, Ray, Trino, Apache Flink, and open catalogs (Apache Polaris, Unity Catalog, Apache Gravitino).
  37. For more details, see the full [Lance format specification](https://lance.org/format).
  38. > [!TIP]
  39. > Lance is in active development and we welcome contributions. Please see our [contributing guide](https://lance.org/community/contributing/) for more information.
  40. ## File format stability
  41. Lance releases frequently because the SDKs, integrations, and performance work are moving quickly. This does not mean the Lance file format changes incompatibly in every release. The Lance file format is identified by the `data_storage_version` stored in each dataset, and stable storage versions are a long-term compatibility contract.
  42. - Once a dataset is written with a stable `data_storage_version`, future Lance releases will continue to support reading that storage version.
  43. - SDK and API compatibility is separate from file format compatibility. SDK/API changes follow semantic versioning and are documented in the [migration guide](https://lance.org/guide/migration/).
  44. - Older Lance releases may not understand file format versions introduced later. If you run mixed Lance versions, pin `data_storage_version` for deterministic writes.
  45. - The `next` file format alias is unstable and should only be used for experimentation, never for production data.
  46. For production, write data with a stable `data_storage_version`. See the [format versioning guide](https://lance.org/format/file/versioning/) for the current compatibility matrix.
  47. ## Quick Start
  48. **Installation**
  49. ```shell
  50. pip install pylance
  51. ```
  52. To install a preview release:
  53. ```shell
  54. pip install --pre --extra-index-url https://pypi.fury.io/lance-format pylance
  55. ```
  56. > [!TIP]
  57. > Preview releases are released more often than full releases and contain the
  58. > latest features and bug fixes. They receive the same level of testing as full releases.
  59. > We guarantee they will remain published and available for download for at
  60. > least 6 months. When you want to pin to a specific version, prefer a stable release.
  61. **Converting to Lance**
  62. ```python
  63. import lance
  64. import pandas as pd
  65. import pyarrow as pa
  66. import pyarrow.dataset
  67. df = pd.DataFrame({"a": [5], "b": [10]})
  68. uri = "/tmp/test.parquet"
  69. tbl = pa.Table.from_pandas(df)
  70. pa.dataset.write_dataset(tbl, uri, format='parquet')
  71. parquet = pa.dataset.dataset(uri, format='parquet')
  72. lance.write_dataset(parquet, "/tmp/test.lance")
  73. ```
  74. **Reading Lance data**
  75. ```python
  76. dataset = lance.dataset("/tmp/test.lance")
  77. assert isinstance(dataset, pa.dataset.Dataset)
  78. ```
  79. **Pandas**
  80. ```python
  81. df = dataset.to_table().to_pandas()
  82. df
  83. ```
  84. **DuckDB**
  85. ```python
  86. import duckdb
  87. # If this segfaults, make sure you have duckdb v0.7+ installed
  88. duckdb.query("SELECT * FROM dataset LIMIT 10").to_df()
  89. ```
  90. **Vector search**
  91. Download the sift1m subset
  92. ```shell
  93. wget ftp://ftp.irisa.fr/local/texmex/corpus/sift.tar.gz
  94. tar -xzf sift.tar.gz
  95. ```
  96. Convert it to Lance
  97. ```python
  98. import lance
  99. from lance.vector import vec_to_table
  100. import numpy as np
  101. import struct
  102. nvecs = 1000000
  103. ndims = 128
  104. with open("sift/sift_base.fvecs", mode="rb") as fobj:
  105. buf = fobj.read()
  106. data = np.array(struct.unpack("<128000000f", buf[4 : 4 + 4 * nvecs * ndims])).reshape((nvecs, ndims))
  107. dd = dict(zip(range(nvecs), data))
  108. table = vec_to_table(dd)
  109. uri = "vec_data.lance"
  110. sift1m = lance.write_dataset(table, uri, max_rows_per_group=8192, max_rows_per_file=1024*1024)
  111. ```
  112. Build the index
  113. ```python
  114. sift1m.create_index("vector",
  115. index_type="IVF_PQ",
  116. num_partitions=256, # IVF
  117. num_sub_vectors=16) # PQ
  118. ```
  119. Search the dataset
  120. ```python
  121. # Get top 10 similar vectors
  122. import duckdb
  123. dataset = lance.dataset(uri)
  124. # Sample 100 query vectors. If this segfaults, make sure you have duckdb v0.7+ installed
  125. sample = duckdb.query("SELECT vector FROM dataset USING SAMPLE 100").to_df()
  126. query_vectors = np.array([np.array(x) for x in sample.vector])
  127. # Get nearest neighbors for all of them
  128. rs = [dataset.to_table(nearest={"column": "vector", "k": 10, "q": q})
  129. for q in query_vectors]
  130. ```
  131. ## Directory structure
  132. | Directory | Description |
  133. | ------------------ | ------------------------ |
  134. | [rust](./rust) | Core Rust implementation |
  135. | [python](./python) | Python bindings (PyO3) |
  136. | [java](./java) | Java bindings (JNI) |
  137. | [docs](./docs) | Documentation source |
  138. ## Benchmarks
  139. ### Vector search
  140. We used the SIFT dataset to benchmark our results with 1M vectors of 128D
  141. 1. For 100 randomly sampled query vectors, we get <1ms average response time (on a 2023 m2 MacBook Air)
  142. ![avg_latency.png](docs/src/images/avg_latency.png)
  143. 2. ANNs are always a trade-off between recall and performance
  144. ![avg_latency.png](docs/src/images/recall_vs_latency.png)
  145. ### Vs. parquet
  146. We create a Lance dataset using the Oxford Pet dataset to do some preliminary performance testing of Lance as compared to Parquet and raw image/XMLs. For analytics queries, Lance is 50-100x better than reading the raw metadata. For batched random access, Lance is 100x better than both parquet and raw files.
  147. ![](docs/src/images/lance_perf.png)
  148. ## Why Lance for AI/ML workflows?
  149. The machine learning development cycle involves multiple stages:
  150. ```mermaid
  151. graph LR
  152. A[Collection] --> B[Exploration];
  153. B --> C[Analytics];
  154. C --> D[Feature Engineer];
  155. D --> E[Training];
  156. E --> F[Evaluation];
  157. F --> C;
  158. E --> G[Deployment];
  159. G --> H[Monitoring];
  160. H --> A;
  161. ```
  162. Traditional lakehouse formats were designed for SQL analytics and struggle with AI/ML workloads that require:
  163. - **Vector search** for similarity and semantic retrieval
  164. - **Fast random access** for sampling and interactive exploration
  165. - **Multimodal data** storage (images, videos, audio alongside embeddings)
  166. - **Data evolution** for feature engineering without full table rewrites
  167. - **Hybrid search** combining vectors, full-text, and SQL predicates
  168. While existing formats (Parquet, Iceberg, Delta Lake) excel at SQL analytics, they require additional specialized systems for AI capabilities. Lance brings these AI-first features directly into the lakehouse format.
  169. A comparison of different formats across ML development stages:
  170. | | Lance | Parquet & ORC | JSON & XML | TFRecord | Database | Warehouse |
  171. | ------------------- | ----- | ------------- | ---------- | -------- | -------- | --------- |
  172. | Analytics | Fast | Fast | Slow | Slow | Decent | Fast |
  173. | Feature Engineering | Fast | Fast | Decent | Slow | Decent | Good |
  174. | Training | Fast | Decent | Slow | Fast | N/A | N/A |
  175. | Exploration | Fast | Slow | Fast | Slow | Fast | Decent |
  176. | Infra Support | Rich | Rich | Decent | Limited | Rich | Rich |
  177. ## Supported Storage
  178. Lance reads and writes datasets stored on local disk or in object storage. The
  179. storage backend is selected automatically by the URI scheme of the dataset path.
  180. | Storage | URI scheme | Notes |
  181. | ---------------------------------------- | -------------------- | ----- |
  182. | Local filesystem | `file://` | Default; also accepts plain paths. `file+uring://` (Linux only) uses `io_uring` for better local I/O performance. |
  183. | AWS S3 | `s3://` | Supports S3 Express One Zone buckets. Use `s3+ddb://` to version datasets with DynamoDB-managed metadata. |
  184. | S3-compatible stores | `s3://` | MinIO, Cloudflare R2, Wasabi, and other S3-compatible stores via the `endpoint` storage option. |
  185. | Microsoft Azure Blob Storage | `az://` | Also supports Azure Data Lake Storage Gen2 via `abfss://`. |
  186. | Google Cloud Storage | `gs://` | |
  187. | Alibaba Cloud Object Storage Service | `oss://` | |
  188. | Tencent Cloud Object Storage | `cos://` | |
  189. | Volcengine Object Storage (TOS) | `tos://` | |
  190. | Hugging Face Hub | `hf://` | |
  191. | GooseFS | `goosefs://` | |
  192. | In-memory | `memory://` | Intended for testing only. |
  193. For credentials, authentication, and per-store configuration options, see the
  194. [object store configuration guide](./docs/src/guide/object_store.md).

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.