⚠️ THIS IS A PUBLIC REPO ⚠️

💀 Don't push private/confidantial code! 💀

Hadoop cluster - lite

Build a lightweight hadoop cluster with below mentioned hadoop services. These versions match what Hortonworks HDP 2.3.4 distribution has.

forked from: https://github.com/anair-it/hadoop-docker-lite

Dockerhub: https://hub.docker.com/r/anoopnair/

Versions

• Alpine Linux: latest
• Debian: 8.3
• Java: Open JDK 7
• hadoop : 2.7.1
• Zookeeper: 3.4.6
• Kafka: 0.9.0.1
• Kafka manager: latest

Note: Hadoop is built on Debian Linux and all others on Alpine Linux

Prerequsities

• Install Docker
• Install Docker compose

The Docker project is tested on Ubuntu host. There is no guarantee that this will work on Windows.

Building all images

• Pull from docker hub OR
• Build each image individually

This will individually build the above mentioned images. This will take sometime and is required to be done only once.

• Get zookeeper IP address using

docker inspect --format='{{.NetworkSettings.IPAddress}}' zookeeper

• Get hbase IP address using

docker inspect --format='{{.NetworkSettings.IPAddress}}' hbase

• Add zookeeper and hbase entries in the host /etc/hosts file

Verification

• docker images to see all images created with proper tag version
• docker logs -f _CONTAINER_NAME_ to follow logs for a specific container

If there is inconsistency in the image creation process, run the affected build script only. The build script is in each component directory.

Starting

./start.sh (background process) OR

docker-compose up (foreground process)

Stopping

docker-compose stop

Web UI

• Kafka manager
• Hadoop cluster

READMEs

Please go through below READMEs to get detailed information about each image

• Alpine base README
• Debian base README
• Hadoop README
• Zookeeper README
• Kafka README

Reference

• https://hub.docker.com/r/wurstmeister/kafka/
• https://hub.docker.com/r/sheepkiller/kafka-manager/
• https://hub.docker.com/r/sequenceiq/hadoop-ubuntu/
• http://sookocheff.com/post/docker/containerizing-zookeeper-a-guided-tour/
• https://docs.docker.com/engine/userguide/intro/
• https://docs.docker.com/compose/