Skip to content
Back to search

sample-genai-on-eks-starter-kit

A comprehensive toolkit for deploying production-ready Generative AI infrastructure on Amazon EKS. Includes pre-configured components for: πŸš€ AI Gateway (LiteLLM) πŸ€– LLM Serving (vLLM, SGLang, Ollama) πŸ“Š Vector Databases, πŸ” Embedding Models (TEI) πŸ“ˆ Observability (Langfuse, Phoenix) etc. Fast-track your GenAI deployment with Kubernetes

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/aws-samples/sample-genai-on-eks-starter-kit with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/aws-samples/sample-genai-on-eks-starter-kit'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/aws-samples--sample-genai-on-eks-starter-kit/badge.svg)](/listings/aws-samples--sample-genai-on-eks-starter-kit)

HTML

<a href="/listings/aws-samples--sample-genai-on-eks-starter-kit"><img src="/api/listings/aws-samples--sample-genai-on-eks-starter-kit/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

1. Install dependencies:

npm install

2. Configure environment variables:

./cli configure
# Example:
# βœ” Enter value for REGION: us-west-2
# βœ” Enter value for EKS_CLUSTER_NAME: genai-on-eks
# ? Enter value for DOMAIN:

This will prompt you to enter values for environment variables. Then, it will save the values on `.env.local`. There are a few important ones, including:

- REGION - AWS region to be used to provision the infrastructure - EKS_CLUSTER_NAME - Name of the EKS cluster - EKS_MODE - EKS mode, `auto` (default) or `standard` - DOMAIN - Recommend to use a domain name already configured with a Route 53 hosted zone, check [FAQs](#faqs) more details - HF_TOKEN - Hugging Face [user access token](https://huggingface.co/docs/hub/en/security-tokens)

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md Β· README Β· 378 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. # GenAI on EKS Starter Kit
  2. [![Documentation](https://img.shields.io/badge/docs-live-blue)](https://aws-samples.github.io/sample-genai-on-eks-starter-kit/)
  3. [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
  4. A production-ready starter kit for deploying **generative AI infrastructure** on **Amazon EKS** (Elastic Kubernetes Service). Deploy LLM serving engines (vLLM, SGLang, TGI), AI gateways, vector databases, GPU-accelerated inference platforms, and AI agent frameworks on **Kubernetes** with a single CLI command. Built for teams running self-hosted LLMs and AI workloads on AWS.
  5. The starter kit includes the configurable components and examples from several categories:
  6. - AI Gateway - [LiteLLM](https://www.litellm.ai), [Kong AI Gateway OSS](https://github.com/Kong/kong)
  7. - LLM Model - [vLLM](https://docs.vllm.ai), [SGLang](https://docs.sglang.ai), [Ollama](https://ollama.com)
  8. - Embedding Model - [Text Embedding Inference (TEI)](https://huggingface.co/docs/text-embeddings-inference)
  9. - Observability (o11y) - [Langfuse](https://langfuse.com), [Phoenix](https://phoenix.arize.com)
  10. - GUI App - [Open WebUI](https://docs.openwebui.com)
  11. - Vector Database - [Qdrant](https://qdrant.tech), [Chroma](https://docs.trychroma.com), [Milvus](https://milvus.io)
  12. - Workflow Automation - [n8n](https://docs.n8n.io)
  13. - AI Agent - [OpenClaw](https://github.com/openclaw/openclaw)
  14. - MCP Server - [FastMCP 2.0](https://gofastmcp.com)
  15. - AI Agent Framework - [Strands Agents ](https://strandsagents.com), [Agno](https://docs.agno.com)
  16. ## Prerequisites
  17. Before you begin, ensure you have the following tools installed:
  18. - [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html)
  19. - [Terraform](https://www.terraform.io/downloads)
  20. - [Node.js](https://nodejs.org/en/download)
  21. - [kubectl](https://kubernetes.io/docs/tasks/tools)
  22. - [Helm](https://helm.sh/docs/intro/install)
  23. - [Docker](https://docs.docker.com/get-started/) and [Buildx](https://docs.docker.com/build/concepts/overview/#buildx)
  24. ## Initial Setup
  25. 1. Install dependencies:
  26. ```bash
  27. npm install
  28. ```
  29. 2. Configure environment variables:
  30. ```bash
  31. ./cli configure
  32. # Example:
  33. # βœ” Enter value for REGION: us-west-2
  34. # βœ” Enter value for EKS_CLUSTER_NAME: genai-on-eks
  35. # ? Enter value for DOMAIN:
  36. ```
  37. This will prompt you to enter values for environment variables. Then, it will save the values on `.env.local`. There are a few important ones, including:
  38. - REGION - AWS region to be used to provision the infrastructure
  39. - EKS_CLUSTER_NAME - Name of the EKS cluster
  40. - EKS_MODE - EKS mode, `auto` (default) or `standard`
  41. - DOMAIN - Recommend to use a domain name already configured with a Route 53 hosted zone, check [FAQs](#faqs) more details
  42. - HF_TOKEN - Hugging Face [user access token](https://huggingface.co/docs/hub/en/security-tokens)
  43. ## Environment Setup
  44. There are two methods to setup your environment:
  45. ### Method 1: Quick Demo Setup
  46. To quickly set up a demo environment with infrastructure and essential components and examples:
  47. ```bash
  48. ./cli demo-setup
  49. ```
  50. This command will:
  51. 1. Set up the required infrastructure using Terraform (check [Infrastructure Setup](docs/INFRA_SETUP.md) for more information)
  52. 2. Deploy the demo components and examples specified in the config.json file in the right order
  53. Check [Demo Walkthrough](docs/DEMO_WALKTHROUGH.md) on how to setup and use the demo
  54. ### Method 2: Interactive Setup
  55. For a more customized setup, you can use the interactive setup command:
  56. ```bash
  57. ./cli interactive-setup
  58. # Example:
  59. # βœ” Select AI Gateway components to install: litellm
  60. # βœ” Select LLM Model components to install: vllm
  61. # ? Select Embedding Model components to install: (Press <space> to select, <a> to toggle all, <i> to invert selection, and <enter> to proceed)
  62. # ❯◉ Text Embedding Inference (TEI)
  63. ```
  64. This command will:
  65. 1. Present you with a list of available components and examples organized by category
  66. 2. Allow you to select which components and examples you want to install
  67. 3. Set up the required infrastructure using Terraform
  68. 4. Install all the selected components and examples
  69. Note. Unlike the quick demo setup, the selected components and examples may not be deployed in the required order. Some components/examples might need to be refreshed by running the CLI install command again.
  70. ## NVIDIA Dynamo Platform Setup
  71. This starter kit supports deploying [NVIDIA Dynamo](https://developer.nvidia.com/dynamo) for optimized LLM inference on Amazon EKS.
  72. ### Quick Start
  73. 1. Install components in order:
  74. ```bash
  75. ./cli nvidia-platform monitoring install # Prometheus + Grafana
  76. ./cli nvidia-platform gpu-operator install # NVIDIA GPU Operator
  77. ./cli nvidia-platform dynamo-platform install # Dynamo CRDs, Operator, etcd, NATS
  78. ./cli nvidia-platform dynamo-vllm install # Deploy a model with vLLM
  79. ```
  80. 2. Optionally run benchmarks and auto-configuration:
  81. ```bash
  82. ./cli nvidia-platform benchmark install # AIPerf concurrency sweep
  83. ./cli nvidia-platform aiconfigurator install # TP/PP recommendation + SLA deploy
  84. ```
  85. For full details on platform prerequisites, deployment modes (aggregated vs disaggregated), KV cache routing, monitoring dashboards, benchmarking, and AIConfigurator, see the [NVIDIA Platform README](components/nvidia-platform/README.md).
  86. ## Components & Examples Management
  87. You can install or uninstall individual components/examples using the CLI:
  88. ### Install a Component/Example
  89. ```bash
  90. ./cli <category> <component/example> install
  91. # Examples:
  92. # ./cli ai-gateway litellm install
  93. # ./cli strands-agents calculator-agent install
  94. ```
  95. ### Uninstall a Component/Example
  96. ```bash
  97. ./cli <category> <component/example> uninstall
  98. # Examples:
  99. # ./cli ai-gateway litellm uninstall
  100. # ./cli strands-agents calculator-agent uninstall
  101. ```
  102. ## LLM/Embedding Model Management
  103. The CLI provides commands to manage LLM/Embedding models for the hosting components:
  104. ### Configure Models
  105. Configure which models should be deployed for a specific component:
  106. ```bash
  107. ./cli llm-model <component> configure-models
  108. ./cli embedding-model <component> configure-models
  109. # Example:
  110. # ./cli llm-model vllm configure-models
  111. # ./cli embedding-model tei configure-models
  112. ```
  113. ### Update Models
  114. Add and/or remove models for a specific component:
  115. ```bash
  116. ./cli llm-model <component> update-models
  117. ./cli embedding-model <component> update-models
  118. # Example:
  119. # ./cli llm-model vllm update-models
  120. # ./cli embedding-model tei update-models
  121. ```
  122. ### Add Models
  123. Only add missing models for a specific component:
  124. ```bash
  125. ./cli llm-model <component> add-models
  126. ./cli embedding-model <component> add-models
  127. # Example:
  128. # ./cli llm-model vllm add-models
  129. # ./cli embedding-model tei add-models
  130. ```
  131. ### Remove All Models
  132. Remove all models for a specific component:
  133. ```bash
  134. ./cli llm-model <component> remove-all-models
  135. ./cli embedding-model <component> remove-all-models
  136. # Example:
  137. # ./cli llm-model vllm remove-all-models
  138. # ./cli embedding-model tei remove-all-models
  139. ```
  140. ## Cleanup
  141. There are two methods to clean up your environment:
  142. ### Method 1: Uninstall Components/Examples Individually and Destroy Infrastructure
  143. This method gives you more control over the cleanup process:
  144. 1. Uninstall each component/example:
  145. ```bash
  146. # Examples:
  147. # ./cli strands-agents calculator-agent uninstall
  148. # ./cli ai-gateway litellm uninstall
  149. # ... uninstall other components/examples as needed
  150. ```
  151. 2. Destroy the infrastructure:
  152. ```bash
  153. ./cli cleanup-infra
  154. ```
  155. ### Method 2: Cleanup Everything at Once
  156. This method provides a one-command solution to clean up all examples, components and infrastructure:
  157. ```bash
  158. ./cli cleanup-everything
  159. ```
  160. This command will:
  161. 1. Attempt to uninstall all deployed examples and components
  162. 2. Destroy the infrastructure using Terraform
  163. ## FAQs
  164. ### How the config files work?
  165. `.env` and `config.json` will be loaded first. Then, the configs will be merged/overridden with the values from `.env.local` and `config.local.json` if exist.
  166. ### How can I use this starter kit without having a Route 53 hosted zone?
  167. With a domain name already configured with a Route 53 hosted zone, a single shared ALB with HTTPS is used together with a wildcard ACM cert and Route 53 DNS records to expose all public facing services e.g. litellm.<DOMAIN> and openwebui.<DOMAIN>.
  168. Alternatively, when the `DOMAIN` field on `.env` (or `.env.local`) is empty, multiple ALBs with HTTP will be created for each public facing service. In this case, only one service requiring the Nginx Ingress basic auth (e.g. Milvus and Qdrant) can be exposed.
  169. ### How can I configure and update the LiteLLM proxy model list?
  170. Run `./cli litellm install` again to update the LiteLLM models.
  171. For Bedrock models, the model list hardcoded on config.json.
  172. ```json
  173. # Example:
  174. "bedrock": {
  175. "llm": {
  176. "models": [
  177. { "name": "amazon-nova-premier", "model": "us.amazon.nova-premier-v1:0" },
  178. { "name": "claude-4-opus", "model": "us.anthropic.claude-opus-4-20250514-v1:0" },
  179. ]
  180. }
  181. }
  182. ```
  183. ### How can I change the EC2 GPU instance families and purchasing options?
  184. The default instance families are g6e, g6 and g5g and the default purchasing options are spot and on-demand. You can change the values on `terraform/variables.tf` and then run `./cli terraform apply again`.
  185. Note that the model deployment manifests use `nodeSelector` like `eks.amazonaws.com/instance-family: g6e` to lock the specific tested instance family which you will need to adjust accordingly.
  186. For self-hosted models, they will be dynamically detected from the running model pods.
  187. ### How can I use LLM models with AWS Neuron and EC2 Inferentia 2?
  188. The supported models will have the `-neuron` suffix. To enable the support, on `config.json` (or `config.local.json`), change `enableNeuron` to `true` and then install the component again (e.g `./cli llm-model vllm install`) which will take ~20-30 mins to build the vLLM Neuron container image and push it to ECR.
  189. When a supported LLM model is deployed for the first time, Neorun just-in-time (JIT) compilation will compile the model which will take ~20-30 mins. The compiled model then will be cached on EFS file system for the subsequent deployments.
  190. Llama-3.1-8B-Instruct, DeepSeek-R1-Distill-Llama-8B, Mistral-7B-Instruct-v0.3 models can the INT8 quantization to run on single inf2.xlarge, but inf2.8xlarge is still required to compile.
  191. - 1st step is to deploy the model with will used inf2.8xlarge first to compile and cache the model
  192. - Then, on `config.json` (or `config.local.json`), change from `"compile": true` to `"compile": false`
  193. - Then, delete the model deploymen and then deploy again which will use inf2.xlarge
  194. ### How can I disable the multi-arch container image build?
  195. By default, `docker buildx` is used to build the multi-arch container images. To disable it, modify the `docker` section on `config.json` (or `config.local.json`) to set `"useBuildx": false` and `arch` based on your machine OS arch.
  196. ### How can I use the different AWS region for Bedrock?
  197. By default, the same region as the EKS cluster will be used. To change it, modify the `bedrock` section on `config.json` (or `config.local.json`) to set `region` to the preferred region.
  198. ### How can I provision and manage multiple EKS clusters?
  199. You can change the values of the `REGION`, `EKS_CLUSTER_NAME`, and `DOMAIN` fields on `.env` (or `.env.local`). Then, Terraform workspace and kubectl context will automatically use those values when running the related `./cli` commands.
  200. ### What is ECR Pull Through Cache and should I enable it?
  201. ECR Pull Through Cache caches external container images (from Docker Hub, GitHub Container Registry) in your private ECR registry. This avoids rate limits from public registries and keeps images within your AWS infrastructure.
  202. **Default: Disabled** (`enable_ecr_pull_through_cache = false`)
  203. **Why disabled by default?**
  204. - Cached images are stored in your private ECR, which incurs storage costs
  205. - Public registries (Docker Hub, GHCR) work fine for most use cases since EKS nodes have internet access
  206. **When to enable:**
  207. - You're hitting Docker Hub rate limits (anonymous: 100 pulls/6hrs, authenticated: 200 pulls/6hrs)
  208. - You want faster, more reliable pulls from within AWS
  209. - Your organization requires images to be stored in private registries
  210. **To enable:**
  211. Docker Hub and GitHub Container Registry require authentication for ECR pull through cache. You'll need to provide credentials for both registries.
  212. 1. **Get Docker Hub credentials:**
  213. - Create a Docker Hub account at [hub.docker.com](https://hub.docker.com)
  214. - Generate an access token at [Docker Hub Security Settings](https://hub.docker.com/settings/security) β†’ "New Access Token"
  215. - For more information, see [Create and manage access tokens](https://docs.docker.com/security/for-developers/access-tokens/) in the Docker documentation
  216. 2. **Get GitHub credentials:**
  217. - Generate a Personal Access Token (classic) at [GitHub Settings β†’ Developer settings β†’ Personal access tokens](https://github.com/settings/tokens)
  218. - The token needs `read:packages` scope to pull from GitHub Container Registry
  219. - For more information, see [Managing your personal access tokens](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens) in the GitHub documentation
  220. 3. **Configure credentials in `config.local.json`:**
  221. ```json
  222. // config.local.json
  223. {
  224. "terraform": {
  225. "vars": {
  226. "enable_ecr_pull_through_cache": true,
  227. "dockerhub_username": "your-dockerhub-username",
  228. "dockerhub_access_token": "your-dockerhub-access-token",
  229. "github_username": "your-github-username",
  230. "github_token": "your-github-personal-access-token"
  231. }
  232. }
  233. }
  234. ```
  235. 4. **Apply the changes:**
  236. ```bash
  237. ./cli terraform apply
  238. ```
  239. **Important:** Keep your credentials in `config.local.json` (which is gitignored) and never commit them to version control. The credentials are stored securely in AWS Secrets Manager with the `ecr-pullthroughcache/` prefix.
  240. **Supported registries:**
  241. - `vllm/*` β†’ Docker Hub
  242. - `lmsysorg/*` β†’ Docker Hub
  243. - `ollama/*` β†’ Docker Hub
  244. - `huggingface/*` β†’ GitHub Container Registry
  245. **Note:** When you `terraform destroy`, the cache rules are deleted but the cached ECR repositories remain. To fully clean up, manually delete repositories starting with `vllm/`, `lmsysorg/`, `ollama/`, or `huggingface/` from ECR.
  246. ## Disclaimer
  247. ⚠️ **This repository is intended for demonstration and learning purposes only.**
  248. It is **not** intended for production use. The code provided here is for educational purposes and should not be used in a live environment without proper testing, validation, and modifications.
  249. Use at your own risk. The authors are not responsible for any issues, damages, or losses that may result from using this code in production.
  250. Check [Security Considerations](docs/SECURITY.md) for more information on the security scans.
  251. ## Contributing
  252. Contributions welcome! Please read our [Contributing Guidelines](CONTRIBUTING.md) and [Code of Conduct](CODE_OF_CONDUCT.md) for more information.
  253. ## License
  254. This project is licensed under the MIT License - see [LICENSE](LICENSE) file.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.