Add open options settings for test configs

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,24 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: ''
+assignees: jennyowen, bfeshti
+
+---
+
+## Guidelines
+
+Please note that GitHub issues are only meant for bug reports/feature requests. 
+If you have questions on how to use Neo4j, please ask on [Neo4j Community](https://community.neo4j.com/) or [StackOverflow](http://stackoverflow.com/questions/tagged/neo4j) instead of creating an issue here.
+
+To help us understand your issue, please specify important details, primarily:
+
+- **Steps to reproduce**. *Not including reproduction steps will mean your bug will take considerably longer to investigate and fix. Please don't skip this*.
+- Expected behaviour
+- Actual behaviour
+- Neo4j image tag being used, eg `neo4j:latest`, `neo4j:enterprise-3.5` etc
+- The output of the `docker version` command
+- Operating system: (for example Windows 95/Ubuntu 16.04)
+
+Additionally, include (as appropriate) log-files, stacktraces, and other debug output.

.gitignore

@@ -4,3 +4,10 @@
 /out/
 /in/
 /devenv.local
+/local-mounts/
+/target/
+
+# simlinks to dev scripts in https://github.com/neo-technology/teamcity-witchcraft
+download_tool.py
+docker_tests_get_installers.py
+__pycache__/

DEVELOPMENT.md

@@ -8,26 +8,129 @@ other Linuxes. Pull requests welcomed for other platforms.
 ## OSX only
 
 1. install GNU Make (>=4.0)
-1. install the Docker Toolbox
+1. install the Docker Toolbox. See: https://docs.docker.com/install/
 
-## Debian
+## Linux
 
-1. install `uuid-runtime`
+1. install the Docker Toolbox. See https://docs.docker.com/install/
 
-## All platforms
+# Building the Image
 
-1. download the Neo4j Community unix tarball
-1. copy `devenv.local.template` as `devenv.local`; fill in the
-   directory to which you have downloaded the tarball and its version
+The build will create two images (one for Enterprise and one for Community) for a single version of Neo4j. 
 
-# Build process
+The make script will automatically download the source files needed to build the images. 
+You just need to specify the **full** Neo4j version including major, minor and patch numbers. For example:
 
-## OSX only
+```bash
+NEO4JVERSION=3.5.11 make clean build
+```
+
+If you want to build an alpha/beta release, this will still work:
+
+```bash
+NEO4JVERSION=3.5.0-alpha01 make clean build
+```
+
+When the make script is complete, the image name will be written to file in `tmp/.image-id-community` and `tmp/.image-id-enterprise`:
+
+```bash
+$ cat tmp/.image-id-community
+test/19564
+
+$ cat tmp/.image-id-enterprise
+test/13909
+```
+
+## Building ARM64 based images
+
+There is a separate make script for building ARM64 based images. 
+
+Like with `amd64` images, you must still specify the **full** Neo4j version including major, minor and patch numbers. For example:
+
+```bash
+NEO4JVERSION=3.5.11 make -f make-arm64.mk clean tag-arm
+```
+
+
+## If the Neo4j Version is not Publicly Available
+
+The make script cannot automatically download unreleased source files, so you need to manually download them before building the images.
+
+1. Assuming you cloned this repository to `$NEO4J_DOCKER_ROOT`, 
+download the community and enterprise unix tar.gz files and copy them to `$NEO4J_DOCKER_ROOT/in`.
+1. Run the make script setting `NEO4JVERSION` to the version number in the files downloaded into the `in/` folder.
+
+For example: 
+
+```bash
+$ cd $NEO4J_DOCKER_ROOT
+$ ls $NEO4J_DOCKER_ROOT/in
+  neo4j-community-4.0.0-alpha05-unix.tar.gz  neo4j-enterprise-4.0.0-alpha05-unix.tar.gz
+
+$ NEO4JVERSION=4.0.0-alpha05 make clean build
+``` 
+
+
+# Running the Tests
+
+The tests are written in java, and require Maven plus jdk 11 for Neo4j version 4.0 onwards or jdk 8 for earlier Neo4j versions.
+
+The tests require some information about the image before they can test it. 
+These can be passed as an environment variable or a command line parameter when invoking maven:
+
+
+| Env Variable    | Maven parameter | Description                                                |
+|-----------------|-----------------|------------------------------------------------------------|
+| `NEO4JVERSION`  | `-Dversion`     | the Neo4j version of the image                             |
+| `NEO4J_IMAGE`   | `-Dimage`       | the tag of the image to test                               |
+| `NEO4J_EDITION` | `-Dedition`     | Either `community` or `enterprise` depending on the image. |
+
+<!-- prettified with http://www.tablesgenerator.com/markdown_tables -->
+
+## Using Maven
+The Makefile can run the entire test suite.
+1. Make sure `java --version` is java 11 or java 8 as necessary.
+2. `NEO4JVERSION=<VERSION> make test` This is a make target that will run these commands:
+```bash
+mvn test -Dimage=$(cat tmp/.image-id-community) -Dedition=community -Dversion=${NEO4JVERSION}
+mvn test -Dimage=$(cat tmp/.image-id-enterprise) -Dedition=enterprise -Dversion=${NEO4JVERSION}
+```
+
+## In Intellij
+
+1. Make sure the project SDK is java 11 or java 8 as necessary.
+1. Edit the [pom.xml file](../master/pom.xml) to replace  `${env.NEO4JVERSION}` with the `NEO4JVERSION` you used to build the image.
+*(Yes this is terrible, and we need to think of an alternative to this)*. 
+
+    For example:
+    ```xml
+    <neo4j.version>${env.NEO4JVERSION}</neo4j.version>
+    ```
+    becomes
+    ```xml
+    <neo4j.version>4.0.0-alpha05</neo4j.version>
+    ```
+1. Install the [EnvFile](https://plugins.jetbrains.com/plugin/7861-envfile) Intellij plugin.
+2. Under Run Configurations edit the Template JUnit configuration:
+   1. Select the "EnvFile" tab
+   2. Make sure "Enable EnvFile" is checked.
+   3. Click the `+` then click to add a `.env` file.
+   4. In the file selection box select `./tmp/devenv-enterprise.env` or `./tmp/devenv-community.env` depending on which one you want to test.
+   5. Rebuilding the Neo4j image will regenerate the `.env` files, so you don't need to worry about keeping the environment up to date.
+
+
+### If the Neo4j Version is not Publicly Available
+
+1. Clone the Neo4j github repository and checkout the branch you want. 
+2. Make sure `java --version` returns java 11 if you're building Neo4j 4.0+, or java 8 if building an earlier branch.
+1. Run `mvn install` plus whatever maven build flags you like. This should install the latest neo4j jars into the maven cache.
+1. Follow instructions for [running tests in Intellij](#in-intellij), 
+use the `NEO4JVERSION` that is in the pom file of your Neo4j repository clone.
+
+### cannot find symbol `com.sun.security.auth.module.UnixSystem`
 
-1. create a docker-machine VM
-1. export the docker-machine environment for docker
+This can happen if you switch from java 8 to java 11 and then try to rebuild the tests in Intellij.
 
-## All platforms
+Check that the `java.version` property in the [pom.xml file](../master/pom.xml) is set to 11 instead of 1.8.
+DO NOT commit this set to 11 (yes this is a terrible solution).
 
-1. `. devenv`
-1. `make`

Makefile

@@ -1,106 +1,68 @@
-SHELL := bash
-.ONESHELL:
-.SHELLFLAGS := -eu -o pipefail -c
-.DELETE_ON_ERROR:
-.SECONDEXPANSION:
-.SECONDARY:
+include make-common.mk
 
-ifeq ($(origin .RECIPEPREFIX), undefined)
-  $(error This Make does not support .RECIPEPREFIX. Please use GNU Make 4.0 or later)
-endif
-.RECIPEPREFIX = >
+NEO4J_BASE_IMAGE?="openjdk:11-jdk-slim"
 
-ifndef NEO4J_VERSION
-  $(error NEO4J_VERSION is not set)
-endif
+# Use make test TESTS='<pattern>' to run specific tests
+# e.g. `make test TESTS='TestCausalCluster'` or `make test TESTS='*Cluster*'`
+# the value of variable is passed to the maven test property. For more info see https://maven.apache.org/surefire/maven-surefire-plugin/examples/single-test.html
+# by default this is empty which means all tests will be run
+TESTS?=""
 
-tarball = neo4j-$(1)-$(2)-unix.tar.gz
-dist_site := http://dist.neo4j.org
-series := $(shell echo "$(NEO4J_VERSION)" | sed -E 's/^([0-9]+\.[0-9]+)\..*/\1/')
-
-all: out/enterprise/.sentinel out/community/.sentinel
+all: test
 .PHONY: all
 
-test: test-community test-enterprise
+test: test-enterprise test-community
 .PHONY: test
 
-out/%/.sentinel: tmp/image-%/.sentinel tmp/.tests-pass-%
-> mkdir -p $(@D)
-> cp -r $(<D)/* $(@D)
-> touch $@
+test-enterprise: build-enterprise
+> mvn test -Dimage=$$(cat tmp/.image-id-enterprise) -Dadminimage=$$(cat tmp/.image-id-neo4j-admin-enterprise) -Dedition=enterprise -Dversion=$(NEO4JVERSION) -Dtest=$(TESTS)
+.PHONY: test-enterprise
 
-tmp/.tests-pass-%: tmp/.image-id-% $(shell find test -name 'test-*') $(shell find test -name '*.yml') $(shell find test -name '*.sh')
-> mkdir -p $(@D)
-> image_id=$$(cat $<)
-> for test in $(filter test/test-%,$^); do
->   echo "Running $${test}"
->   "$${test}" "$${image_id}" "${series}" "$*"
-> done
-> touch $@
+test-community: build-community
+> mvn test -Dimage=$$(cat tmp/.image-id-community) -Dadminimage=$$(cat tmp/.image-id-neo4j-admin-community) -Dedition=community -Dversion=$(NEO4JVERSION) -Dtest=$(TESTS)
+.PHONY: test-community
+
+# just build the images, don't test or package
+build: build-community build-enterprise
+.PHONY: build
+
+build-community: tmp/.image-id-community tmp/.image-id-neo4j-admin-community tmp/devenv-community.env
+.PHONY: build-community
+
+build-enterprise: tmp/.image-id-enterprise tmp/.image-id-neo4j-admin-enterprise tmp/devenv-enterprise.env
+.PHONY: build-enterprise
+
+tmp/devenv-%.env:  tmp/.image-id-% tmp/.image-id-neo4j-admin-%
+> echo "NEO4JVERSION=$(NEO4JVERSION)" > ${@}
+> echo "NEO4J_IMAGE=$$(cat tmp/.image-id-${*})" >> ${@}
+> echo "NEO4JADMIN_IMAGE=$$(cat tmp/.image-id-neo4j-admin-${*})" >> ${@}
+> echo "NEO4J_EDITION=${*}" >> ${@}
+
+# create release images and loadable images
+package: package-community package-enterprise
+.PHONY: package
+
+package-community: tmp/.image-id-community tmp/.image-id-neo4j-admin-community out/community/.sentinel
+> mkdir -p out
+> docker tag $$(cat $<) neo4j:$(NEO4JVERSION)
+> docker save neo4j:$(NEO4JVERSION) > out/neo4j-community-$(NEO4JVERSION)-docker-loadable.tar
 
+package-enterprise: tmp/.image-id-enterprise tmp/.image-id-neo4j-admin-enterprise out/enterprise/.sentinel
+> mkdir -p out
+> docker tag $$(cat $<) neo4j:$(NEO4JVERSION)-enterprise
+> docker save neo4j:$(NEO4JVERSION)-enterprise > out/neo4j-enterprise-$(NEO4JVERSION)-docker-loadable.tar
+
+# create image from local build context
 tmp/.image-id-%: tmp/local-context-%/.sentinel
 > mkdir -p $(@D)
 > image=test/$$RANDOM
 > docker build --tag=$$image \
-    --build-arg="NEO4J_URI=file:///tmp/$(call tarball,$*,$(NEO4J_VERSION))" \
+    --build-arg="NEO4J_URI=file:///tmp/$(call tarball,$*,$(NEO4JVERSION))" \
     $(<D)
 > echo -n $$image >$@
 
-tmp/local-context-%/.sentinel: tmp/image-%/.sentinel in/$(call tarball,%,$(NEO4J_VERSION))
-> rm -rf $(@D)
+# copy the releaseable version of the image to the output folder.
+out/%/.sentinel: tmp/image-%/.sentinel
 > mkdir -p $(@D)
 > cp -r $(<D)/* $(@D)
-> cp $(filter %.tar.gz,$^) $(@D)/local-package
-> touch $@
-
-tmp/image-%/.sentinel: src/$(series)/Dockerfile src/$(series)/docker-entrypoint.sh \
-                       in/$(call tarball,%,$(NEO4J_VERSION))
-> mkdir -p $(@D)
-> cp $(filter %/docker-entrypoint.sh,$^) $(@D)/docker-entrypoint.sh
-> sha=$$(shasum --algorithm=256 $(filter %.tar.gz,$^) | cut -d' ' -f1)
-> <$(filter %/Dockerfile,$^) sed \
-    -e "s|%%NEO4J_SHA%%|$${sha}|" \
-    -e "s|%%NEO4J_TARBALL%%|$(call tarball,$*,$(NEO4J_VERSION))|" \
-    -e "s|%%NEO4J_EDITION%%|$*|" \
-    -e "s|%%NEO4J_DIST_SITE%%|$(dist_site)|" \
-    >$(@D)/Dockerfile
-> mkdir -p $(@D)/local-package
-> touch $(@D)/local-package/.sentinel
-> touch $@
-
-run = trapping-sigint \
-    docker run --publish 7474:7474 --publish 7687:7687 \
-    --env=NEO4J_ACCEPT_LICENSE_AGREEMENT=yes \
-    --env=NEO4J_AUTH=neo4j/foo --rm $$(cat $1)
-build-enterprise: tmp/.image-id-enterprise
-> @echo "Neo4j $(NEO4J_VERSION)-enterprise available as: $$(cat $<)"
-build-community: tmp/.image-id-community
-> @echo "Neo4j $(NEO4J_VERSION)-community available as: $$(cat $<)"
-run-enterprise: tmp/.image-id-enterprise
-> $(call run,$<)
-run-community: tmp/.image-id-community
-> $(call run,$<)
-test-enterprise: tmp/.tests-pass-enterprise
-test-community: tmp/.tests-pass-community
-.PHONY: run-enterprise run-community build-enterprise build-community test-enterprise test-community
-
-fetch_tarball = curl --fail --silent --show-error --location --remote-name \
-    $(dist_site)/$(call tarball,$(1),$(NEO4J_VERSION))
-
-cache: in/neo4j-%-$(NEO4J_VERSION)-unix.tar.gz
-.PHONY: cache
-
-in/neo4j-community-$(NEO4J_VERSION)-unix.tar.gz:
-> mkdir -p in
-> cd in
-> $(call fetch_tarball,community)
-
-in/neo4j-enterprise-$(NEO4J_VERSION)-unix.tar.gz:
-> mkdir -p in
-> cd in
-> $(call fetch_tarball,enterprise)
-
-clean:
-> rm -rf tmp
-> rm -rf out
-.PHONY: clean
+> touch $@