Kafka and MySQL Integration on Kubernetes

Step 1: Download and Build Debezium MySQL Connector

sudo curl https://repo1.maven.org/maven2/io/debezium/debezium-connector-mysql/1.0.0.Final/debezium-connector-mysql-1.0.0.Final-plugin.tar.gz | tar xvz

FROM strimzi/kafka:0.20.1-kafka-2.5.0
USER root:root
RUN mkdir -p /opt/kafka/plugins/debezium
COPY ./debezium-connector-mysql/ /opt/kafka/plugins/debezium/
USER 1001

docker build . -t henriksin1/connect-debezium
docker push henriksin1/connect-debezium

Step 2: Create Kubernetes Namespace

kubectl apply -f 0-namespace.yaml

Step 3: Deploy Strimzi Operators

Install Strimzi operators using either OLM or Helm:

Using OLM

curl -L https://github.com/operator-framework/operator-lifecycle-manager/releases/download/v0.26.0/install.sh -o install.sh
chmod +x install.sh
./install.sh v0.26.0

Using Helm

helm repo add strimzi https://strimzi.io/charts
helm install my-strimzi-kafka-operator strimzi/strimzi-kafka-operator --version 0.38.0 -n kafka

kubectl create -f https://operatorhub.io/install/strimzi-kafka-operator.yaml

Operators Included:

• Cluster Operator: Manages Kafka clusters, automating setup of Kafka brokers.
• Topic Operator: Manages creation, configuration, and deletion of topics.
• User Operator: Manages Kafka users requiring access to Kafka brokers.

Step 4: Deploy Secrets

kubectl -n kafka create secret generic debezium-secret --from-file=secrets.properties

Create the file secrets.properties.

Step 5: Deploy RBAC Access Control

kubectl apply -f rbac-debezium-role.yaml

Step 6: Deploy Service Account

kubectl apply -f serviceaccount.yaml

Step 7: Deploy RBAC Cluster Binding

kubectl apply -f rbac-cluster-binding.yaml

Step 8: Deploy Kafka Cluster

kubectl apply -f kafka-kluster.yaml

Confirm the cluster is running:

kubectl wait kafka/my-cluster --for=condition=Ready --timeout=300s -n kafka

Exec into the Kafka cluster to run topics:

kubectl -n kafka exec my-cluster-kafka-0 -c kafka -i -t -- bin/kafka-console-producer.sh --bootstrap-server my-cluster-kafka-bootstrap:9092 --topic my-topic

Open another terminal to confirm the topics are consumed:

kubectl -n kafka exec my-cluster-kafka-0 -c kafka -i -t -- bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my-topic --from-beginning

Step 9: Deploy MySQL Cluster Database

kubectl apply -f 6-sql.yaml

Confirm the database is running:

kubectl describe service mysql -n kafka

Connect with your local MySQL Workbench.

Step 10: Run Kafka Connector

Deploy Kafka Connect:

kubectl apply -f kafka-connect.yaml

Confirm all pods are running:

kubectl get all -n kafka

Run:

kubectl logs debezium-connect-cluster-connect-0 -n kafka

Step 11: Create Kafka Connector File

kubectl apply -f debezium-connector.yaml

Run:

kubectl describe kafkaconnector debezium-connector-mysql -n kafka

Now it's showtime:

kubectl -n kafka exec my-cluster-kafka-0 -c kafka -i -t -- bin/kafka-topics.sh --bootstrap-server localhost:9092 --list

kubectl -n kafka exec my-cluster-kafka-0 -c kafka -i -t -- bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic mysql.inventory.customers

Exec into the database:

kubectl exec -it mysql-6597659cb8-j9rk4 -n kafka -- sh

mysql -h mysql -u root -p

Password:

use inventory;
show tables;

kubectl -n kafka exec my-cluster-kafka-0 -c kafka -i -t -- bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic mysql --from-beginning

Pictures and documentation on connecting to the SQL database serving as a microservice.

Feel free to adjust and enhance the configuration as needed.