Skip to content
Back to search

klag

MarketplaceWorks with Claude CodeGitHub

The Klag Kafka consumer lag exporter: install it, connect its MCP endpoint, and triage consumer lag from your agent.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/themoah--klag/badge.svg)](/listings/themoah--klag)

HTML

<a href="/listings/themoah--klag"><img src="/api/listings/themoah--klag/badge.svg" alt="findsafeskills safety rating" /></a>

Install

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

docker run -e KAFKA_BOOTSTRAP_SERVERS=kafka:9092 \
           -e METRICS_REPORTER=prometheus \
           -p 8888:8888 \
           themoah/klag:latest

Metrics available at `http://localhost:8888/metrics`.

A GraalVM native image (`themoah/klag:native`) starts in ~70-100 ms using ~44 MB RSS, versus ~500 ms / ~119 MB for the JVM image — same config, endpoints, and metrics. See [Native Image](https://klag.dev/deployment/native-image/).

Helm

helm repo add klag https://themoah.github.io/klag
helm repo update
helm install klag klag/klag --set kafka.bootstrapServers="kafka-broker:9092"

Published to [Artifact Hub](https://artifacthub.io/packages/helm/klag/klag). Full chart config, SASL, and Strimzi: [Kubernetes deployment](https://klag.dev/deployment/kubernetes/) and [charts/klag/README.md](charts/klag/README.md).

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/marketplace.json · Manifest · 16 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. {
  2. "$schema": "https://code.claude.com/schemas/marketplace.json",
  3. "name": "klag",
  4. "description": "The Klag Kafka consumer lag exporter: install it, connect its MCP endpoint, and triage consumer lag from your agent.",
  5. "owner": {
  6. "name": "themoah",
  7. "url": "https://klag.dev"
  8. },
  9. "plugins": [
  10. {
  11. "name": "klag",
  12. "source": "./plugin",
  13. "description": "Install, connect and diagnose Klag — the Kafka consumer lag exporter."
  14. }
  15. ]
  16. }
README.md · README · 251 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. # Klag
  2. [![License][license-badge]][license] [![Release][release-badge]][release] [![CI][ci-badge]][ci]
  3. [![Docker Pulls][docker-badge]][docker] [![Artifact Hub][artifacthub-badge]][artifacthub] [![Ask DeepWiki][deepwiki-badge]][deepwiki]
  4. [license]: https://github.com/themoah/klag/blob/main/LICENSE
  5. [license-badge]: https://img.shields.io/badge/License-Apache%202.0-blue.svg
  6. [release]: https://github.com/themoah/klag/releases/latest
  7. [release-badge]: https://img.shields.io/github/v/release/themoah/klag?logo=github
  8. [ci]: https://github.com/themoah/klag/actions/workflows/build-and-push.yml
  9. [ci-badge]: https://github.com/themoah/klag/actions/workflows/build-and-push.yml/badge.svg
  10. [docker]: https://hub.docker.com/r/themoah/klag
  11. [docker-badge]: https://img.shields.io/docker/pulls/themoah/klag?logo=docker
  12. [artifacthub]: https://artifacthub.io/packages/helm/klag/klag
  13. [artifacthub-badge]: https://img.shields.io/endpoint?url=https://artifacthub.io/badge/repository/klag
  14. [deepwiki]: https://deepwiki.com/themoah/klag
  15. [deepwiki-badge]: https://deepwiki.com/badge.svg
  16. **Kafka Consumer Lag Exporter** — Know when your consumers fall behind, before it becomes a problem.
  17. Inspired by [kafka-lag-exporter](https://github.com/seglo/kafka-lag-exporter) (archived 2024). Built with Vert.x and Micrometer.
  18. Migrating from kafka-lag-exporter? See the **[migration guide](https://klag.dev/migration)** (metric/label/config mapping + drop-in dashboard tips).
  19. > ### 📖 Documentation lives at **[klag.dev](https://klag.dev)**
  20. >
  21. > Full guides — configuration, Kafka ACLs, Helm/Strimzi deployment, integrations,
  22. > metrics reference, and development — are at **[klag.dev](https://klag.dev)**.
  23. > AI agents: a machine-readable corpus is at [klag.dev/llms.txt](https://klag.dev/llms.txt).
  24. > The docs source lives in [`website/`](website/) (Astro + Starlight).
  25. > ### 🤖 Install with your AI agent
  26. >
  27. > In Claude Code: `/plugin marketplace add themoah/klag` → `/plugin install klag@klag` → `/klag:install`.
  28. > It detects Docker/Kubernetes/ArgoCD/Strimzi, deploys, verifies metrics, and can wire up the MCP
  29. > endpoint (`/klag:connect`, `/klag:diagnose`). Copilot, Codex, Cursor, OpenCode and others:
  30. > **[klag.dev/agent-setup](https://klag.dev/agent-setup)**.
  31. > **Scales to large clusters:** Monitors thousands of consumer groups in ~50MB heap. Request batching with configurable delays prevents overwhelming brokers — fetch offsets for 500+ groups without spiking cluster CPU.
  32. ## Why Klag?
  33. Consumer lag is the gap between what Kafka has produced and what your consumers have processed. Left unmonitored, growing lag leads to stale downstream data, memory pressure as consumers struggle to catch up, and silent failures when groups die without alerts. Klag continuously monitors all consumer groups and exposes metrics to your observability stack.
  34. ## Key Features
  35. | Feature | Why It Matters |
  36. |-----------------------------------|---------------------------------------------------------------------------|
  37. | **Lag velocity** | Know if lag is growing or shrinking — catch problems before they escalate |
  38. | **Time-based lag estimation** | See lag in seconds/minutes, not just message counts |
  39. | **Hot partition detection** | Find partitions with uneven load causing bottlenecks |
  40. | **Consumer group state tracking** | Alert on Rebalancing, Dead, or Empty states |
  41. | **Data loss prevention** | Alert before lag exceeds retention and data is lost |
  42. | **Request batching** | Safely monitor large clusters without overwhelming brokers |
  43. | **AI-native** | Opt-in read-only [MCP endpoint](https://klag.dev/ai/mcp/) for SRE/dev agents |
  44. **Sinks:** Prometheus, Datadog, OTLP (Grafana Cloud, New Relic, etc.), StatsD/DogStatsD. See [integrations](https://klag.dev/integrations/prometheus/).
  45. ## How Klag compares
  46. Klag, [Burrow](https://github.com/linkedin/Burrow), and [KMinion](https://github.com/redpanda-data/kminion) all monitor Kafka consumer lag. They differ in what they measure and where they send it.
  47. | Feature | Klag | Burrow | KMinion |
  48. |---|:---:|:---:|:---:|
  49. | Lag in messages | ✅ | ✅ | ✅ |
  50. | Consumer group state | ✅ | ✅ | ✅ |
  51. | Lag velocity (growing/shrinking) | ✅ | ⚠️ status only | ❌ |
  52. | Time-based lag + time-to-catch-up | ✅ | ❌ | ❌ |
  53. | Hot partition detection | ✅ | ❌ | ❌ |
  54. | Data loss / retention alerting | ✅ | ❌ | ❌ |
  55. | Lag status rules + notifiers | ❌ | ✅ | ❌ |
  56. | Prometheus / Datadog / OTLP native | ✅ | ⚠️ exporter / ❌ / ❌ | ✅ / ❌ / ❌ |
  57. | AI agent endpoint (MCP) | ✅ | ❌ | ❌ |
  58. | Read-only ACLs (DESCRIBE only) | ✅ | ✅ | ⚠️ writes for e2e |
  59. ## Quick Start
  60. ```bash
  61. docker run -e KAFKA_BOOTSTRAP_SERVERS=kafka:9092 \
  62. -e METRICS_REPORTER=prometheus \
  63. -p 8888:8888 \
  64. themoah/klag:latest
  65. ```
  66. Metrics available at `http://localhost:8888/metrics`.
  67. A GraalVM native image (`themoah/klag:native`) starts in ~70-100 ms using ~44 MB RSS,
  68. versus ~500 ms / ~119 MB for the JVM image — same config, endpoints, and metrics. See
  69. [Native Image](https://klag.dev/deployment/native-image/).
  70. ### Helm
  71. ```bash
  72. helm repo add klag https://themoah.github.io/klag
  73. helm repo update
  74. helm install klag klag/klag --set kafka.bootstrapServers="kafka-broker:9092"
  75. ```
  76. Published to [Artifact Hub](https://artifacthub.io/packages/helm/klag/klag). Full chart
  77. config, SASL, and Strimzi: [Kubernetes deployment](https://klag.dev/deployment/kubernetes/)
  78. and [charts/klag/README.md](charts/klag/README.md).
  79. ## Metrics
  80. | Metric | Description |
  81. |--------|----------------------------------------------------------------------|
  82. | `klag.consumer.lag` | Current lag per partition (also `.sum`, `.max`, `.min`) |
  83. | `klag.consumer.lag.velocity` | Rate of change — positive means falling behind |
  84. | `klag.consumer.lag.ms` | Lag in ms from Kafka log timestamps |
  85. | `klag.consumer.lag.time_to_close_seconds` | Estimated seconds until lag reaches zero |
  86. | `klag.consumer.lag.retention_percent` | Lag as % of available messages (data loss alerting) |
  87. | `klag.consumer.group.state` | Group health: stable, preparing/completing_rebalance (assigning/reconciling on KIP-848 groups), dead, empty |
  88. | `klag.hot_partition[.lag]` | Partitions with statistically abnormal throughput |
  89. | `klag.topic.size_skew` | Topic retained-size imbalance (max/mean × 100; opt-in `DATA_SKEW_ENABLED`) |
  90. | `klag.partition.under_replicated` | Partitions with fewer in-sync replicas than configured (ISR gap) |
  91. Full metrics reference and the pre-built [Grafana dashboard](https://klag.dev/integrations/grafana-dashboard/)
  92. are documented at [klag.dev/metrics](https://klag.dev/metrics/overview/). The dashboard is on
  93. [Grafana.com (ID 25379)](https://grafana.com/grafana/dashboards/25379-klag-kafka-lag-monitoring/) —
  94. import by ID or use `dashboard/demo-dashboard.json`.
  95. [![Grafana Dashboard](dashboard/grafana.png)](dashboard/demo-dashboard.json)
  96. [Blogpost: Introducing Klag](https://medium.com/p/introducing-klag-the-kafka-lag-exporter-i-always-wanted-d919bdb64a7a)
  97. ## Configuration
  98. Configure via `src/main/resources/application.properties` or environment variables. The
  99. most common:
  100. | Variable | Default | Description |
  101. |----------|---------|-------------|
  102. | `KAFKA_BOOTSTRAP_SERVERS` | `localhost:9092` | Kafka broker addresses |
  103. | `HTTP_PORT` | `8888` | HTTP server port (health, metrics, MCP) |
  104. | `METRICS_REPORTER` | `none` | `prometheus`, `datadog`, `otlp`, or `statsd` |
  105. | `METRICS_INTERVAL_MS` | `60000` | How often to collect metrics |
  106. | `METRICS_GROUP_FILTER` | `*` | Comma-separated glob patterns. A group is included if it matches any segment (e.g. `ingest*,categorize*`). |
  107. | `METRICS_GROUP_EXCLUDE` | _(empty)_ | Comma-separated glob patterns to exclude even if included by the filter (e.g. `debug-*,canary-*,*-shadow`). |
  108. Any `Env`-backed variable can also be set as a JVM system property — `-DNAME` or its
  109. dotted-lowercase form `-Dname.dotted` (e.g. `HTTP_PORT` → `-Dhttp.port=8881`) — useful
  110. when running the jar or native binary directly. Env vars take precedence. This works on
  111. the GraalVM native binary too (runtime `-D` flags). See the
  112. [configuration reference](website/src/content/docs/configuration/reference.md) for the
  113. complete list of `Env`-backed settings.
  114. See [CLAUDE.md](CLAUDE.md) for the complete configuration reference.
  115. ---
  116. ## Broker Compatibility
  117. Klag works with Apache Kafka **2.1 through 4.x** brokers. The lower bound comes from
  118. `kafka-clients` 4.x, which dropped support for brokers older than 2.1. CI e2e-tests
  119. against 4.1 and 4.2 (`scripts/e2e-strimzi-matrix.sh`); older brokers are covered by unit
  120. tests of the fallback paths, since Strimzi no longer ships images that old.
  121. ### Running against Kafka 2.x
  122. When klag first talks to a 2.x broker, you'll see a single WARN log line — emitted **once per process**, not per topic and not per scrape:
  123. ```
  124. MAX_TIMESTAMP listOffsets unsupported by broker (likely pre-Kafka 3.0);
  125. falling back to LATEST for logEndTimestamp. Logged once per process;
  126. further occurrences at DEBUG. Cause: ...
  127. ```
  128. **This is expected, and safe to ignore.** All klag metrics remain accurate. Subsequent per-topic fallback events are emitted at DEBUG; flip `LOG_LEVEL_KLAG=DEBUG` if you want to see them.
  129. <details>
  130. <summary>Why this happens, and why the fallback exists at all</summary>
  131. Klag asks the broker for partition end-offsets using `OffsetSpec.MAX_TIMESTAMP`, an API added in Kafka 3.0 ([KIP-734](https://cwiki.apache.org/confluence/display/KAFKA/KIP-734%3A+Improve+AdminClient.listOffsets+to+return+timestamp+and+offset+for+the+record+with+the+largest+timestamp)). 2.x brokers don't support it and respond with `UnsupportedVersionException`. Klag catches that and falls back to `OffsetSpec.LATEST`, which every Kafka version supports. The fallback returns the same partition end-offset that drives every lag metric, so accuracy is preserved.
  132. **Why the fallback is necessary.** Without it, every collection cycle fails against a 2.x broker and klag exports no lag metrics at all — one consumer group on a 2.x cluster was enough. It used to be worse: the first scrape ran synchronously during startup and its failure propagated to the process launcher, which exited with code 1 (CrashLoopBackOff under Kubernetes). The first cycle is now recovered so an unreachable or incompatible broker degrades instead of bricking startup, but that only buys a running process — the fallback is what makes the metrics correct.
  133. **Note for future contributors.** On 2.x brokers, both `logEndTimestamp` and `maxTimestampOffset` fall back to the LATEST offset's timestamp/offset, so time-based lag interpolation degrades gracefully (the anchor becomes the broker-side append time of the last record rather than the highest-timestamp record).
  134. </details>
  135. ---
  136. ## Kafka ACL Permissions
  137. Klag requires **read-only** access to monitor consumer lag. It uses only the Kafka Admin Client API with DESCRIBE permissions—no write or alter access needed.
  138. ### Required Permissions
  139. | Resource | Name | Permission | Operations |
  140. |----------|------|------------|------------|
  141. | CLUSTER | kafka-cluster | DESCRIBE | Health check, list consumer groups |
  142. | TOPIC | `*` or prefixed | DESCRIBE | Get partition info and offsets |
  143. | GROUP | `*` or prefixed | DESCRIBE | Get group state and committed offsets |
  144. ### Self-Managed Kafka
  145. <details>
  146. <summary>Monitor all groups and topics</summary>
  147. ```bash
  148. # Cluster permissions (required)
  149. kafka-acls --bootstrap-server <broker> \
  150. --add --allow-principal User:<klag-user> \
  151. --operation Describe --cluster
  152. # All topics
  153. kafka-acls --bootstrap-server <broker> \
  154. --add --allow-principal User:<klag-user> \
  155. --operation Describe --topic '*'
  156. # All consumer groups
  157. kafka-acls --bootstrap-server <broker> \
  158. --add --allow-principal User:<klag-user> \
  159. --operation Describe --group '*'
  160. ```
  161. </details>
  162. Klag needs **read-only** Kafka access (Admin Client DESCRIBE on cluster, topics, groups).
  163. Full configuration reference and ACL setup (self-managed + Confluent Cloud) are at
  164. [klag.dev/configuration](https://klag.dev/configuration/reference/) and
  165. [klag.dev/kafka/acl-permissions](https://klag.dev/kafka/acl-permissions/).
  166. ## Development
  167. Requires Java 21.
  168. ```bash
  169. ./gradlew clean test # Run tests
  170. ./gradlew clean assemble # Build fat JAR
  171. ./gradlew clean run # Run with hot-reload
  172. ```
  173. End-to-end tests (k3d + real Kafka, Strimzi matrix) live in `scripts/`. See
  174. [Build from Source](https://klag.dev/development/build/) and
  175. [Contributing](https://klag.dev/development/contributing/).
  176. New to the project? Pick a [`good first issue`](https://github.com/themoah/klag/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22)
  177. or [`hacktoberfest`](https://github.com/themoah/klag/issues?q=is%3Aissue+is%3Aopen+label%3Ahacktoberfest) ticket and follow [CONTRIBUTING.md](CONTRIBUTING.md).
  178. **Using Klag in production?** Soft ask — [tell us (opt-in)](https://github.com/themoah/klag/issues/96) or see [ADOPTERS.md](ADOPTERS.md).
  179. ---
  180. [![vert.x](https://img.shields.io/badge/vert.x-4.5.34-purple.svg)](https://vertx.io)
  181. Some parts of the code were written with Claude
  182. <img src="https://raw.githubusercontent.com/lobehub/lobe-icons/refs/heads/master/packages/static-png/dark/claude-color.png" width="56" height="56" alt="Claude">
  183. </content>
  184. </invoke>

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.