Fix initialization of dockerized services

[oguzhanunlu]

Background story

During development of 0.6.0, I used a cronjob running docker-compose restart at reboot and RC instances weren't launching properly on GCP but launching fine on AWS. After some discussions, I created an init script and made Ansible place it under /etc/init.d/, however I haven't made Ansible execute service snowplow_mini start command to launch the services, I made Ansible run docker-compose up -d command.

Although both AWS and GCP RC instances launched fine after this change, this init script should be used to launch the service, instead of direct execution of docker-compose command.

Why is it a bug?

Per docker-compose docs,

Compose supports declaring default environment variables in an environment file named .env placed in the folder where the docker-compose command is executed (current working directory).

Although related Ansible task executes docker command in the same directory as .env, launch an instance and check /var/log/snowplow_mini.err;

The ES_JVM_SIZE variable is not set. Defaulting to a blank string.
The SP_JVM_SIZE variable is not set. Defaulting to a blank string.

Why does it work then?

Since first launch of docker services happen through direct execution of docker-compose up -d with correct environment variables from .env, containers continue with these correct env values even you restart services through control plane or reboot the box.

What to do?

1 - Launch docker services via init script instead of directly executing docker-compose up -d command.
2 - After handling 1, check if env vars are fetched correctly, fix if it is not the case.

I noticed this thanks to @jbeemster .

[alexanderdean]

How fragile is this?

[jbeemster]

If any of the containers were deleted it would break the env var settings and possibly prevent the docker-compose file from even being valid. As long as no one can get onto the server to break this it should be fine but its not an ideal scenario - especially for production!

[dcow]

My AWS instance seems to be stuck in a state where snowplow is not started on boot. The log shows the same items along with:

...
Couldn't connect to Docker daemon at http+docker://localhost - is it running?

If it's at a non-standard location, specify the URL with the DOCKER_HOST environment variable.

Running docker-compose up -d yields

ubuntu@ip-xxx-xx-xx-x:~/snowplow$ docker-compose up -d
ERROR: Couldn't connect to Docker daemon at http+docker://localhost - is it running?

Is there a way to fix a borked instance?

[dcow]

Things weren't coming up due to #191