Skip to content

Latest commit

 

History

History
45 lines (30 loc) · 1.02 KB

README.md

File metadata and controls

45 lines (30 loc) · 1.02 KB
Raw

Smartcity Realtime Pipeline

Architecture

architecture_diagram

Development

  • Use locally
docker build --rm --compress -t poc/smartcity-kafka-spark:latest .
docker compose -f ./docker-compose.yml -p poc-smartcity-kafka-spark up -d
# Detached help to not to run in terminal
  • Submit a job (execute)
docker exec -it poc-smartcity-kafka-spark-spark-master-1 spark-submit --master spark://spark-master:7077 --packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.5.1,org.apache.hadoop:hadoop-aws:3.3.6,com.amazonaws:aws-java-sdk:1.12.666 jobs/spark-city.py
  • Send messages to topics (execute)
python jobs/main.py
  • Check topics in broker
kafka-topics --list --bootstrap-server broker:29092
  • Check messages delivered to topic
kafka-console-consumer --topic vehicle_data --bootstrap-server broker:9092 --from-beginning
  • Delete topics
kafka-topics --delete --topic vehicle_data --bootstrap-server broker:9092