Skip to content
Back to search

kaito

Kubernetes AI Toolchain Operator

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/kaito-project/kaito with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/kaito-project/kaito'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/kaito-project--kaito/badge.svg)](/listings/kaito-project--kaito)

HTML

<a href="/listings/kaito-project--kaito"><img src="/api/listings/kaito-project--kaito/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/marketplace.json · Manifest · 1 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. ../.github/plugin/marketplace.json
README.md · README · 101 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. # Kubernetes AI Toolchain Operator (KAITO)
  2. ![GitHub Release](https://img.shields.io/github/v/release/kaito-project/kaito)
  3. [![Go Report Card](https://goreportcard.com/badge/github.com/kaito-project/kaito)](https://goreportcard.com/report/github.com/kaito-project/kaito)
  4. ![GitHub go.mod Go version](https://img.shields.io/github/go-mod/go-version/kaito-project/kaito)
  5. [![codecov](https://codecov.io/gh/kaito-project/kaito/graph/badge.svg?token=XAQLLPB2AR)](https://codecov.io/gh/kaito-project/kaito)
  6. [![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Fkaito-project%2Fkaito.svg?type=shield)](https://app.fossa.com/projects/git%2Bgithub.com%2Fkaito-project%2Fkaito?ref=badge_shield)
  7. | ![notification](website/static/img/bell.svg) What is NEW! |
  8. | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
  9. | ALL vLLM supported models can be run in KAITO now. |
  10. | Latest Release: Oct 24th, 2026. KAITO v0.13.0. |
  11. | First Release: Nov 15th, 2023. KAITO v0.1.0. |
  12. KAITO is an operator suite that automates LLM model inference, fine-tuning, and RAG (Retrieval Augmented Generation) engine deployment in a Kubernetes cluster.
  13. KAITO has the following key differentiations compared to other inference model deployment methodologies:
  14. - Simplify the CRD API by removing detailed deployment parameters. The controller provides optimized preset configurations for key inference engine scheduling parameters such as pipeline parallelism (PP), data parallelism (DP), tensor parallelism (TP), max model length, etc.
  15. - Use node auto provisioner (NAP) to provision GPU resources with accurate model memory estimation, enabling the controller to pick the optimal node count for distributed inference.
  16. - Leverage GPU node built-in local NVMe as model storage — no extra storage is required for inference.
  17. - Support any [vLLM](https://github.com/vllm-project/vllm)-supported HuggingFace models.
  18. ## Architecture
  19. KAITO follows the classic Kubernetes Custom Resource Definition (CRD)/controller design pattern for workload orchestration and integrates with [Gateway API Inference Extension](https://gateway-api-inference-extension.sigs.k8s.io/) to support LLM-based routing.
  20. <div align="center">
  21. <img src="website/static/img/arch.png" width=100% title="KAITO architecture" alt="KAITO architecture">
  22. </div>
  23. - **Workspace**: The CRD that serves as the basic building block for managing LLM inference/tuning workloads. The API provides a largely simplified experience for deploying an LLM model in Kubernetes - the user provides the GPU instance type and the HuggingFace model ID, the controller will:
  24. - Estimate the GPU memory requirement based on the GPU instance type and model metadata, and calculate the required GPU count;
  25. - Trigger GPU node auto-provisioning by integrating with Karpenter APIs ([NodePool](https://karpenter.sh/docs/concepts/nodepools/));
  26. - Configure the inference engine parameters for single node/multiple nodes inference with optimized scheduling based on the GPU hardware topology.
  27. Currently, only the **vLLM** engine is supported. LoRA adapters are supported. KVCache offloading is enabled by default.
  28. - **InferenceSet**: The CRD designed for managing the number of replicas of workspace instances for the same model. It is primarily used to autoscale the workspace based on inference request load. It reacts to scale-up/down actions determined by a KEDA autoscaler that uses vLLM metrics collected by a [KEDA plugin](https://github.com/kaito-project/keda-kaito-scaler).
  29. - **InferencePool**: KAITO integrates [Gateway API Inference Extension](https://gateway-api-inference-extension.sigs.k8s.io/) by creating corresponding InferencePool object and EPP (Endpoint Picker, which enables KVCache-aware routing) per InferenceSet. It can work with any external gateway that supports the inference extension.
  30. > Note: In this repo, an open-source [gpu-provisioner](https://github.com/Azure/gpu-provisioner) is used in the E2E test and is referred to in various documents. KAITO can work with any other node provisioners that support the [Karpenter-core](https://sigs.k8s.io/karpenter) APIs.
  31. KAITO also supports a **RAGEngine** operator. It streamlines the process of managing a Retrieval Augmented Generation (RAG) service.
  32. <div align="center">
  33. <img src="website/static/img/ragarch.png" width=90% title="KAITO RAGEngine architecture" alt="KAITO RAGEngine architecture">
  34. </div>
  35. - **RAGEngine**: The CRD that defines the components of a RAG service, including the LLM endpoint (optional), the embedding service and the vector DB. The controller will create all required components.
  36. - **Vector database**: Supports a built-in [FAISS](https://github.com/facebookresearch/faiss) in-memory vector database (default), and Qdrant/Milvus persistent databases if specified.
  37. - **Embedding**: Supports both local and remote embedding services to embed documents in the vector database.
  38. - **RAGService**: The core service that leverages the [LlamaIndex](https://github.com/run-llama/llama_index) orchestration. It supports commonly used APIs such as `/index` for indexing documents, `/v1/chat/completion` for intercepting LLM calls to append retrieved context automatically, and `/retrieve` for integrating with MCP servers. The `/retrieve` API uses the Reciprocal Rank Fusion (RRF) hybrid search algorithm to combine the results from both BM25 sparse retrieval and vector dense retrieval.
  39. The details of the service APIs can be found in this [document](https://kaito-project.github.io/kaito/docs/rag).
  40. ## Getting Started
  41. - **Installation**: Please check the guidance [here](https://kaito-project.github.io/kaito/docs/installation) for installing core components (Workspace, InferenceSet) using helm and [here](https://github.com/kaito-project/kaito/blob/main/terraform/README.md) for installation using Terraform.
  42. - **Quick Start**: Please check the quick start guidance [here](https://kaito-project.github.io/kaito/docs/quick-start) for running your first model using KAITO!
  43. - **AutoScaling**: Please check this [doc](https://kaito-project.github.io/kaito/docs/keda-autoscaler-inference) for configuring KAITO and KEDA to enable autoscaling inference workload.
  44. - **BYO models using HuggingFace runtime**: If you plan to run any BYO models using the HuggingFace runtime, check this [doc](https://kaito-project.github.io/kaito/docs/custom-model). Note: KAITO only supports BYO models hosted in HuggingFace.
  45. - **CPU models**: Please check this [doc](https://kaito-project.github.io/kaito/docs/aikit) for running CPU models using [aikit](https://github.com/kaito-project/aikit/).
  46. - **RAGEngine**: Please check the installation guidance and usage documents [here](https://kaito-project.github.io/kaito/docs/rag).
  47. - **RAGEngine Output Guardrails**: Please check the current behavior, configuration, and limitations [here](https://kaito-project.github.io/kaito/docs/rag-output-guardrails).
  48. - **Prefill/Decode Disaggregation**: Please check this [doc](https://kaito-project.github.io/kaito/docs/prefill-decode-disaggregation) for deploying models with P/D disaggregation using MultiRoleInference and Gateway API Inference Extension.
  49. ## Contributing
  50. [Read more](https://kaito-project.github.io/kaito/docs/contributing)
  51. <!-- markdown-link-check-disable -->
  52. This project welcomes contributions and suggestions. The contributions require you to agree to a
  53. Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
  54. the rights to use your contribution. For details, visit [CLAs for CNCF](https://github.com/cncf/cla?tab=readme-ov-file).
  55. When you submit a pull request, a CLA bot will automatically determine whether you need to provide
  56. a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions
  57. provided by the bot. You will only need to do this once across all repos using our CLA.
  58. This project has adopted the CLAs for CNCF, please electronically sign the CLA via
  59. https://easycla.lfx.linuxfoundation.org. If you encounter issues, you can submit a ticket with the
  60. Linux Foundation ID group through the [Linux Foundation Support website](https://jira.linuxfoundation.org/plugins/servlet/desk/portal/4/create/143).
  61. ## Get Involved!
  62. - Visit [#KAITO channel in CNCF Slack](https://cloud-native.slack.com/archives/C09B4EWCZ5M) to discuss features in development and proposals.
  63. - We host a weekly community meeting for contributors on Tuesdays at 4:00pm PST. Please join here: [meeting link](https://zoom-lfx.platform.linuxfoundation.org/meeting/99948431028?password=05912bb9-53fb-4b22-a634-ab5f8261e94c).
  64. - Reference the weekly meeting notes in our [KAITO community calls doc](https://docs.google.com/document/d/1OEC-WUQ2wn0TDQPsU09shMoXn5cW3dSrdu-M43Q79dA/edit?usp=sharing)!
  65. ## License
  66. See [Apache License 2.0](LICENSE).
  67. [![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Fkaito-project%2Fkaito.svg?type=large)](https://app.fossa.com/projects/git%2Bgithub.com%2Fkaito-project%2Fkaito?ref=badge_large)
  68. ## Code of Conduct
  69. KAITO has adopted the [Cloud Native Compute Foundation Code of Conduct](https://github.com/cncf/foundation/blob/main/code-of-conduct.md). For more information see the [KAITO Code of Conduct](CODE_OF_CONDUCT.md).
  70. <!-- markdown-link-check-enable -->
  71. ## Contact
  72. - Please send emails to "KAITO devs" <kaito-dev@microsoft.com> for any issues.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.