How to Run Apache Kafka with ksqlDB

docker run \
    --add-host=dockerhost:host-gateway \
    -p 0.0.0.0:9092:9092 \
    -e ALLOW_PLAINTEXT_LISTENER=yes \
    -e KAFKA_CFG_ZOOKEEPER_CONNECT=dockerhost:2181 \
    -e KAFKA_CFG_LISTENERS=PLAINTEXT://:9092 \
    -e KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://dockerhost:9092 \
    bitnami/kafka:latest

Change KAFKA_CFG_ZOOKEEPER_CONNECT=dockerhost:2181 to your zookeeper address.

Important – To connect to your Apache Kafka broker from the host, add the following DNS entry to your /etc/hosts:

dockerhost    127.0.0.1

• Running a ksqlDB server
• Launching a ksqlDB shell

--add-host=dockerhost:host-gateway creates a custom DNS name inside the container called dockerhost and attaches the IP of your localhost.

Docker containers can’t access services running on your computer by simply using localhost. Instead, you can use host.docker.internal on MacOSX or Windows. An alternative that also works for every platform, including Linux, is to manually add an extra host entry — in our case, we attach the host-gateway IP to a new DNS name that we call dockerhost.

There are plenty of guides out there on how to run Apache Kafka, if you are looking for a docker-compose that runs every service for you, you can look at the Confluent

• Extendable – open to easily attach new services that run via docker or from the host machine without docker.
• Independent – connect to anything on its own without the need for anything extra like docker networks or running every service in tandem using docker-compose.
• Multiplatform – run using the same command for Linux, Mac, and Windows
• Versions – easy to switch between different versions for testing.
• Easy to enable and disable – if a particular service from the stack is not needed
• Localhost accessible – every service needs to be accessible from the host machine as if it was another service

We discarded using --host network because it doesn’t expose the ports running inside the container to the host on Mac and Windows. From the official docker documentation:

The host networking driver only works on Linux hosts, and is not supported on Docker Desktop for Mac, Docker Desktop for Windows, or Docker EE for Windows Server.

If you still want to run it via this option, these are the commands for zookeeper:

docker run \
    -e ALLOW_ANONYMOUS_LOGIN=yes \
    -p 0.0.0.0:2181:2181 \
    bitnami/zookeeper:latest

and for Apache Kafka:

docker run \
    --network host \
    -e ALLOW_PLAINTEXT_LISTENER=yes \
    -e KAFKA_CFG_ZOOKEEPER_CONNECT=localhost:2181 \
    -e KAFKA_CFG_LISTENERS=PLAINTEXT://:9092 \
    -e KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092 \
    bitnami/kafka:latest

We wanted something easy to switch services on and off independently, without having to rely on a docker-compose.yml file stored somewhere.