Kafka Connect

The idea in one sentence

Kafka Connect is a ready-made framework for moving data into and out of Kafka with configuration instead of code — you point a source connector at a database or file to pull data in, or a sink connector at a search index or warehouse to push data out, and Connect handles scaling, offsets, and restarts for you.

Source and sink connectors

A vast amount of “Kafka work” is not business logic at all — it is plumbing: read rows from Postgres into a topic, or write a topic into Elasticsearch. Writing that plumbing by hand means re-solving offset tracking, retries, and parallelism every time. Connect packages it as reusable connectors:

Source connector — external system → Kafka (a database, a file, a change-data-capture stream).
Sink connector — Kafka → external system (a database, a search index, a data warehouse, object storage).

Because a connector is just config, hundreds already exist. You rarely write one; you configure one.

// A JDBC source connector: poll a Postgres table into a topic
{
  "name": "orders-source",
  "config": {
    "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
    "connection.url": "jdbc:postgresql://db:5432/shop",
    "table.whitelist": "orders",          // which table to pull
    "mode": "incrementing",               // track new rows by a column
    "incrementing.column.name": "id",
    "topic.prefix": "pg-",                // -> topic "pg-orders"
    "tasks.max": "3"                       // up to 3 parallel tasks
  }
}

Workers, tasks, and converters

Three moving parts do the work:

Worker — the JVM process that runs connectors. One or more workers form a Connect cluster.
Task — the unit of parallelism. A connector splits its job into up to tasks.max tasks, which workers spread across the cluster (e.g. one task per table or per partition range).
Converter — translates between Connect’s internal record format and the bytes on the topic. This is where Schema Registry plugs in: an AvroConverter serializes with a schema id, a JsonConverter writes plain JSON.

flowchart LR
  db["Postgres"] -->|"source connector"| worker["Connect worker + tasks"]
  worker -->|"converter"| topic["Topic: pg-orders"]
  topic -->|"converter"| worker2["Connect worker + tasks"]
  worker2 -->|"sink connector"| es["Elasticsearch"]

Connect moves data between external systems and Kafka

Distributed vs standalone

Connect runs in two modes:

Standalone — a single worker, config in a local file. Simple, no fault tolerance. Good for a laptop demo or a one-off file ingest.
Distributed — multiple workers coordinating like a consumer group, with connector configs and offsets stored in Kafka topics. If a worker dies, its tasks rebalance onto the survivors. This is what you run in production, and you manage it over a REST API:

# Deploy the connector above to a distributed Connect cluster
curl -X POST http://localhost:8083/connectors \
  -H "Content-Type: application/json" \
  -d @orders-source.json

When to use Connect instead of code

Reach for Connect when the job is moving data with no custom logic — ingest a table, mirror a topic to S3, feed a search index. You get offset management, restarts, and scaling for free. Write a producer or consumer instead when you need real business logic: enrichment against another service, non-trivial branching, or transformations that outgrow a simple converter. If you find yourself wanting to transform and join streams, that is the next lesson’s job (Kafka Streams), not Connect’s.