Skip to content
Back to search

redpanda-connect-plugins

MarketplaceWorks with Claude CodeGitHub

Plugins for Redpanda Connect

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/redpanda-data/connect with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/redpanda-data/connect'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/redpanda-data--connect/badge.svg)](/listings/redpanda-data--connect)

HTML

<a href="/listings/redpanda-data--connect"><img src="/api/listings/redpanda-data--connect/badge.svg" alt="findsafeskills safety rating" /></a>

Install

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Install

Linux:

curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-amd64.zip
unzip rpk-linux-amd64.zip -d ~/.local/bin/

macOS (Homebrew):

brew install redpanda-data/tap/redpanda

Docker:

docker pull docker.redpanda.com/redpandadata/connect

See the [getting started guide][getting-started] for more options.

Run

rpk connect run ./config.yaml

With Docker:

# From a config file
docker run --rm -v /path/to/your/config.yaml:/connect.yaml docker.redpanda.com/redpandadata/connect run

# With inline overrides
docker run --rm -p 4195:4195 docker.redpanda.com/redpandadata/connect run \
  -s "input.type=http_server" \
  -s "output.type=kafka" \
  -s "output.kafka.addresses=kafka-server:9092" \
  -s "output.kafka.topic=redpanda_topic"

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/marketplace.json · Manifest · 17 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. {
  2. "name": "redpanda-connect-plugins",
  3. "version": "0.1.0",
  4. "description": "Plugins for Redpanda Connect",
  5. "owner": {
  6. "name": "Redpanda Data",
  7. "url": "https://redpanda.com"
  8. },
  9. "plugins": [
  10. {
  11. "name": "redpanda-connect",
  12. "description": "YAML config and Bloblang authoring for Redpanda Connect",
  13. "source": "./.claude-plugin/plugins/redpanda-connect",
  14. "category": "development"
  15. }
  16. ]
  17. }
README.md · README · 228 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. # Redpanda Connect
  2. [![Build Status][actions-badge]][actions-url]
  3. [![Apache V2 API][godoc-badge]][godoc-url-apache]
  4. [![Enterprise API][godoc-badge]][godoc-url-enterprise]
  5. Redpanda Connect is a stream processor that moves data between a wide range of [sources][inputs] and [sinks][outputs], with support for [hydration, enrichment, transformation, and filtering][processors] along the way.
  6. That includes a rich set of change-data-capture (CDC) connectors — for Postgres, MySQL, MongoDB, Oracle, MSSQL, and more — so database changes can flow through your pipelines as first-class events.
  7. It uses [Bloblang][bloblang-about] for mapping, runs as a single static binary or container image, and is easy to operate and monitor.
  8. ## Highlights
  9. - **Declarative pipelines** — a stream topology fits in a single YAML file.
  10. - **At-least-once delivery by default** — in-process transactions, no disk state required.
  11. - **A large connector catalog** — cloud services, message brokers, databases, HTTP, and more.
  12. - **First-class CDC** — change-data-capture connectors for Postgres, MySQL, MongoDB, Oracle, and MSSQL.
  13. - **Bloblang** — a mapping language designed for stream data.
  14. - **Cloud-friendly** — stateless and horizontally scalable, with metrics and tracing built in.
  15. ## Example
  16. Stream Postgres changes into Apache Iceberg tables on S3, one Iceberg table per source table:
  17. ```yaml
  18. input:
  19. postgres_cdc:
  20. dsn: postgres://user:pass@db.example.com:5432/app?sslmode=require
  21. schema: public
  22. tables: [ orders, customers ]
  23. stream_snapshot: true
  24. output:
  25. iceberg:
  26. catalog:
  27. url: https://glue.us-east-1.amazonaws.com/iceberg
  28. warehouse: "123456789012"
  29. auth:
  30. aws_sigv4:
  31. region: us-east-1
  32. service: glue
  33. namespace: cdc
  34. table: ${! meta("table") }
  35. storage:
  36. aws_s3:
  37. bucket: my-iceberg-warehouse
  38. region: us-east-1
  39. schema_evolution:
  40. enabled: true
  41. table_location: s3://my-iceberg-warehouse/cdc/
  42. ```
  43. ## Quickstart
  44. ### Install
  45. Linux:
  46. ```shell
  47. curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-amd64.zip
  48. unzip rpk-linux-amd64.zip -d ~/.local/bin/
  49. ```
  50. macOS (Homebrew):
  51. ```shell
  52. brew install redpanda-data/tap/redpanda
  53. ```
  54. Docker:
  55. ```shell
  56. docker pull docker.redpanda.com/redpandadata/connect
  57. ```
  58. See the [getting started guide][getting-started] for more options.
  59. ### Run
  60. ```shell
  61. rpk connect run ./config.yaml
  62. ```
  63. With Docker:
  64. ```shell
  65. # From a config file
  66. docker run --rm -v /path/to/your/config.yaml:/connect.yaml docker.redpanda.com/redpandadata/connect run
  67. # With inline overrides
  68. docker run --rm -p 4195:4195 docker.redpanda.com/redpandadata/connect run \
  69. -s "input.type=http_server" \
  70. -s "output.type=kafka" \
  71. -s "output.kafka.addresses=kafka-server:9092" \
  72. -s "output.kafka.topic=redpanda_topic"
  73. ```
  74. ## Connectors
  75. The catalog includes AWS (DynamoDB, Kinesis, S3, SQS, SNS), Azure (Blob, Queue, Table), GCP (Pub/Sub, Cloud Storage, BigQuery), Kafka, NATS (JetStream, Streaming), NSQ, MQTT, AMQP 0.91 (RabbitMQ), AMQP 1, Redis, Cassandra, Elasticsearch, HDFS, HTTP (server, client, websockets), MongoDB, and SQL (MySQL, PostgreSQL, ClickHouse, MSSQL) — and a lot more in the [components documentation][about-categories].
  76. ## Delivery guarantees
  77. Delivery guarantees [can be a tricky subject](https://youtu.be/QmpBOCvY8mY). Redpanda Connect processes and acknowledges messages using an in-process transaction model with no disk-persisted state, so when it's connecting at-least-once sources and sinks it can guarantee at-least-once delivery — even through crashes, disk corruption, or other server faults.
  78. That's the default, with no caveats, which keeps deployment and scaling straightforward.
  79. ## Observability
  80. ### Health checks
  81. Two HTTP endpoints are exposed for orchestration probes:
  82. - `/ping` — liveness probe; always returns 200.
  83. - `/ready` — readiness probe; returns 200 once both input and output are connected, otherwise 503.
  84. ### Metrics
  85. Redpanda Connect [exposes metrics][metrics] to Statsd, Prometheus, a JSON HTTP endpoint, and [other backends][metrics].
  86. ### Tracing
  87. OpenTelemetry traces are [emitted natively][tracers], so you can visualize what's happening inside a pipeline end-to-end.
  88. ## Configuration
  89. Redpanda Connect ships with tooling for configuration discovery, debugging, and organization — see the [configuration guide][config-doc].
  90. ## Documentation
  91. - [General documentation][general-docs]
  92. - [Bloblang language guide][bloblang-about]
  93. - [Public Go APIs](https://pkg.go.dev/github.com/redpanda-data/benthos/v4/public/service) for building custom plugins
  94. ## Build from source
  95. Requires a [currently supported Go version](https://go.dev/dl/):
  96. ```shell
  97. git clone git@github.com:redpanda-data/connect
  98. cd connect
  99. task build:all
  100. ```
  101. ### Plugins with external dependencies
  102. Components that link against external C libraries (for example `zmq4`) aren't included by default. To pull them in, set the `x_benthos_extra` build tag:
  103. ```shell
  104. # With go
  105. go install -tags "x_benthos_extra" github.com/redpanda-data/connect/v4/cmd/redpanda-connect@latest
  106. # With task
  107. TAGS=x_benthos_extra task build:all
  108. ```
  109. This tag may change or be split into more granular tags in future releases. If the required system libraries aren't installed, the build will fail with an error like `ld: library not found for -lzmq`.
  110. ### Docker image
  111. A multi-stage `Dockerfile` builds a minimal scratch-based image:
  112. ```shell
  113. task docker:all
  114. ```
  115. ```shell
  116. docker run --rm \
  117. -v /path/to/your/config.yaml:/config.yaml \
  118. -v /tmp/data:/data \
  119. -p 4195:4195 \
  120. docker.redpanda.com/redpandadata/connect run /config.yaml
  121. ```
  122. ## Custom plugins
  123. Writing your own plugins in Go is straightforward — check out the [API docs][godoc-url] and the [example plugin repository][plugin-repo] for reference implementations.
  124. ## Development
  125. Redpanda Connect uses [golangci-lint][golangci-lint] for linting and `gofumpt` for formatting. You can configure your editor to use `gofumpt` automatically — instructions are [here](https://github.com/mvdan/gofumpt#installation).
  126. ```shell
  127. task fmt # format the codebase
  128. task lint # lint the codebase
  129. task test # unit and template tests
  130. ```
  131. ## Contributing
  132. Contributions are welcome. Before opening a pull request, please make sure it has been:
  133. - Unit tested with `task test`
  134. - Linted with `task lint`
  135. - Formatted with `task fmt`
  136. Most integration tests spin up Docker containers, so they're skipped by `task test`. You can run them individually with:
  137. ```shell
  138. go test -run "^Test.*Integration.*$" ./internal/impl/<connector directory>/...
  139. ```
  140. [inputs]: https://docs.redpanda.com/connect/components/inputs/about
  141. [about-categories]: https://docs.redpanda.com/connect/components/about/
  142. [processors]: https://docs.redpanda.com/connect/components/processors/about
  143. [outputs]: https://docs.redpanda.com/connect/components/outputs/about
  144. [metrics]: https://docs.redpanda.com/connect/components/metrics/about
  145. [tracers]: https://docs.redpanda.com/connect/components/tracers/about
  146. [config-interp]: https://docs.redpanda.com/connect/configuration/interpolation
  147. [streams-api]: https://docs.redpanda.com/connect/guides/streams_mode/streams_api
  148. [streams-mode]: https://docs.redpanda.com/connect/guides/streams_mode/about
  149. [general-docs]: https://docs.redpanda.com/connect/home/
  150. [bloblang-about]: https://docs.redpanda.com/connect/guides/bloblang/about
  151. [config-doc]: https://docs.redpanda.com/connect/configuration/about
  152. [releases]: https://github.com/redpanda-data/connect/releases
  153. [plugin-repo]: https://github.com/redpanda-data/redpanda-connect-plugin-example
  154. [getting-started]: https://docs.redpanda.com/connect/get-started/quickstarts/rpk/
  155. [godoc-badge]: https://pkg.go.dev/badge/github.com/redpanda-data/benthos/v4/public
  156. [godoc-url]: https://pkg.go.dev/github.com/redpanda-data/benthos/v4/public
  157. [godoc-url-apache]: https://pkg.go.dev/github.com/redpanda-data/connect/public/bundle/free/v4
  158. [godoc-url-enterprise]: https://pkg.go.dev/github.com/redpanda-data/connect/public/bundle/enterprise/v4
  159. [actions-badge]: https://github.com/redpanda-data/connect/actions/workflows/test.yml/badge.svg
  160. [actions-url]: https://github.com/redpanda-data/connect/actions/workflows/test.yml
  161. [golangci-lint]: https://golangci-lint.run/
  162. [jaeger]: https://www.jaegertracing.io/

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.