Using the AECID tools with docker compose

A very easy and comfortable way to start aminer-akafka and logdata-anomaly-miner is using docker-compose. First download the logdata-anomaly-miner repository from github:

$ git clone https://github.com/ait-aecid/logdata-anomaly-miner
$ cd logdata-anomaly-miner

Now configure the kafka-bootstrap-server and topics in docker-compose.yml:

version: "3"

services:
    akafka:
        image: aitaecid/akafka:latest
        environment:
                KAFKA_TOPICS: '["aminer"]'
                KAFKA_BOOTSTRAP_SERVERS: localhost:9092
        volumes:
            - '$PWD/akafka:/var/lib/akafka'
    aminer:
        build:
            context: .
        volumes:
            - '$PWD/akafka:/var/lib/akafka'
            - '$PWD/aminercfg:/etc/aminer'
            - '$PWD/persistency:/var/lib/aminer'
            - '$PWD/logs:/logs'
        depends_on:
                - akafka

Before running docker-compose it is necessary to bootstrap the initial configuration. This can be done by executing the preparation-script as follows:

$ scripts/prep-docker-compose.sh

The initial logdata-anomaly-miner configuration can parse apache access.logs and is located at aminercfg/config.yml. In order to see the aminer-output it is recommended to define the output_file_path on the Eventhandler in aminer/config.yml as follows:

EventHandlers:
        - id: "stpe"
          json: true # optional default: false
          type: "StreamPrinterEventHandler"
          output_file_path: "/var/lib/aminer/log/aminer.out"

To start the kafka-importer and logdata-anomaly-miner the following command can be used:

$ docker-compose up -d

The following command can be used to monitor of the output of logdata-anomaly-miner:

$ tail -f persistency/log/aminer.out
    "LogLinesCount": 1
  },
  "AnalysisComponent": {
    "AnalysisComponentIdentifier": null,
    "AnalysisComponentType": "VerboseUnparsedAtomHandler",
    "AnalysisComponentName": null,
    "Message": "Unparsed atom received",
    "PersistenceFileName": null
  }
}

For further information about configuring the logdata-anomaly-miner please read https://github.com/ait-aecid/logdata-anomaly-miner/wiki/Getting-started-(tutorial) and https://github.com/ait-aecid/logdata-anomaly-miner/wiki/Deployment-with-Docker