Test upgrading from the latest published minor version

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,24 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: ''
+assignees: jennyowen, bfeshti
+
+---
+
+## Guidelines
+
+Please note that GitHub issues are only meant for bug reports/feature requests. 
+If you have questions on how to use Neo4j, please ask on [Neo4j Community](https://community.neo4j.com/) or [StackOverflow](http://stackoverflow.com/questions/tagged/neo4j) instead of creating an issue here.
+
+To help us understand your issue, please specify important details, primarily:
+
+- **Steps to reproduce**. *Not including reproduction steps will mean your bug will take considerably longer to investigate and fix. Please don't skip this*.
+- Expected behaviour
+- Actual behaviour
+- Neo4j image tag being used, eg `neo4j:latest`, `neo4j:enterprise-3.5` etc
+- The output of the `docker version` command
+- Operating system: (for example Windows 95/Ubuntu 16.04)
+
+Additionally, include (as appropriate) log-files, stacktraces, and other debug output.

.gitignore

@@ -4,3 +4,10 @@
 /out/
 /in/
 /devenv.local
+/local-mounts/
+/target/
+
+# simlinks to dev scripts in https://github.com/neo-technology/teamcity-witchcraft
+download_tool.py
+docker_tests_get_installers.py
+__pycache__/

DEVELOPMENT.md

@@ -8,26 +8,119 @@ other Linuxes. Pull requests welcomed for other platforms.
 ## OSX only
 
 1. install GNU Make (>=4.0)
-1. install the Docker Toolbox
+1. install the Docker Toolbox. See: https://docs.docker.com/install/
 
-## Debian
+## Linux
 
-1. install `uuid-runtime`
+1. install the Docker Toolbox. See https://docs.docker.com/install/
 
-## All platforms
+# Building the Image
 
-1. download the Neo4j Community unix tarball
-1. copy `devenv.local.template` as `devenv.local`; fill in the
-   directory to which you have downloaded the tarball and its version
+The build will create two images (one for Enterprise and one for Community) for a single version of Neo4j. 
 
-# Build process
+The make script will automatically download the source files needed to build the images. 
+You just need to specify the **full** Neo4j version including major, minor and patch numbers For example:
 
-## OSX only
+```bash
+NEO4JVERSION=3.5.11 make clean build
+```
+
+If you want to build an alpha/beta release, this will still work:
+
+```$bash
+NEO4JVERSION=3.5.0-alpha01 make clean build
+```
+
+When the make script is complete, the image name will be written to file in `tmp/.image-id-community` and `tmp/.image-id-enterprise`:
+
+```bash
+$ cat tmp/.image-id-community
+test/19564
+
+$ cat tmp/.image-id-enterprise
+test/13909
+```
+
+
+## If the Neo4j Version is not Publicly Available
+
+The make script cannot automatically download unreleased source files, so you need to manually download them before building the images.
+
+1. Assuming you cloned this repository to `$NEO4J_DOCKER_ROOT`, 
+download the community and enterprise unix tar.gz files and copy them to `$NEO4J_DOCKER_ROOT/in`.
+1. Run the make script setting `NEO4JVERSION` to the version number in the files downloaded into the `in/` folder.
+
+For example: 
+
+```bash
+$ cd $NEO4J_DOCKER_ROOT
+$ ls $NEO4J_DOCKER_ROOT/in
+  neo4j-community-4.0.0-alpha05-unix.tar.gz  neo4j-enterprise-4.0.0-alpha05-unix.tar.gz
+
+$ NEO4JVERSION=4.0.0-alpha05 make clean build
+``` 
+
+
+# Running the Tests
+
+The tests are written in java, and require Maven plus jdk 11 for Neo4j version 4.0 onwards or jdk 8 for earlier Neo4j versions.
+
+The tests require some information about the image before they can test it. 
+These can be passed as an environment variable or a command line parameter when invoking maven:
+
+
+| Env Variable    | Maven parameter | Description                                                |
+|-----------------|-----------------|------------------------------------------------------------|
+| `NEO4JVERSION`  | `-Dversion`     | the Neo4j version of the image                             |
+| `NEO4J_IMAGE`   | `-Dimage`       | the tag of the image to test                               |
+| `NEO4J_EDITION` | `-Dedition`     | Either `community` or `enterprise` depending on the image. |
+
+<!-- prettified with http://www.tablesgenerator.com/markdown_tables -->
+
+## Using Maven
+The Makefile can run the entire test suite.
+1. Make sure `java --version` is java 11 or java 8 as necessary.
+2. `NEO4JVERSION=<VERSION> make test` This is a make target that will run these commands:
+```bash
+mvn test -Dimage=$(cat tmp/.image-id-community) -Dedition=community -Dversion=${NEO4JVERSION}
+mvn test -Dimage=$(cat tmp/.image-id-enterprise) -Dedition=enterprise -Dversion=${NEO4JVERSION}
+```
+
+## In Intellij
+
+1. Make sure the project SDK is java 11 or java 8 as necessary.
+1. Edit the [pom.xml file](../master/pom.xml) to replace  `${env.NEO4JVERSION}` with the `NEO4JVERSION` you used to build the image.
+*(Yes this is terrible, and we need to think of an alternative to this)*. 
+
+    For example:
+    ```xml
+    <neo4j.version>${env.NEO4JVERSION}</neo4j.version>
+    ```
+    becomes
+    ```xml
+    <neo4j.version>4.0.0-alpha05</neo4j.version>
+    ```
+1. Install the [EnvFile](https://plugins.jetbrains.com/plugin/7861-envfile) Intellij plugin.
+2. Under Run Configurations edit the Template JUnit configuration:
+   1. Select the "EnvFile" tab
+   2. Make sure "Enable EnvFile" is checked.
+   3. Click the `+` then click to add a `.env` file.
+   4. In the file selection box select `./tmp/devenv-enterprise.env` or `./tmp/devenv-community.env` depending on which one you want to test.
+   5. Rebuilding the Neo4j image will regenerate the `.env` files, so you don't need to worry about keeping the environment up to date.
+
+
+### If the Neo4j Version is not Publicly Available
+
+1. Clone the Neo4j github repository and checkout the branch you want. 
+2. Make sure `java --version` returns java 11 if you're building Neo4j 4.0+, or java 8 if building an earlier branch.
+1. Run `mvn install` plus whatever maven build flags you like. This should install the latest neo4j jars into the maven cache.
+1. Follow instructions for [running tests in Intellij](#in-intellij), 
+use the `NEO4JVERSION` that is in the pom file of your Neo4j repository clone.
+
+### cannot find symbol `com.sun.security.auth.module.UnixSystem`
 
-1. create a docker-machine VM
-1. export the docker-machine environment for docker
+This can happen if you switch from java 8 to java 11 and then try to rebuild the tests in Intellij.
 
-## All platforms
+Check that the `java.version` property in the [pom.xml file](../master/pom.xml) is set to 11 instead of 1.8.
+DO NOT commit this set to 11 (yes this is a terrible solution).
 
-1. `. devenv`
-1. `make`

Makefile

@@ -10,92 +10,110 @@ ifeq ($(origin .RECIPEPREFIX), undefined)
 endif
 .RECIPEPREFIX = >
 
-ifndef NEO4J_VERSION
-  $(error NEO4J_VERSION is not set)
+ifndef NEO4JVERSION
+  $(error NEO4JVERSION is not set)
 endif
 
 tarball = neo4j-$(1)-$(2)-unix.tar.gz
-dist_site := http://dist.neo4j.org
-series := $(shell echo "$(NEO4J_VERSION)" | sed -E 's/^([0-9]+\.[0-9]+)\..*/\1/')
+dist_site := https://dist.neo4j.org
+series := $(shell echo "$(NEO4JVERSION)" | sed -E 's/^([0-9]+\.[0-9]+)\..*/\1/')
 
-all: out/enterprise/.sentinel out/community/.sentinel
+# Use make test TESTS='<pattern>' to run specific tests
+# e.g. `make test TESTS='TestCausalCluster'` or `make test TESTS='*Cluster*'`
+# the value of variable is passed to the maven test property. For more info see https://maven.apache.org/surefire/maven-surefire-plugin/examples/single-test.html
+# by default this is empty which means all tests will be run
+TESTS?=""
+
+all: test
 .PHONY: all
 
-test: test-community test-enterprise
+test: test-enterprise test-community
 .PHONY: test
 
-out/%/.sentinel: tmp/image-%/.sentinel tmp/.tests-pass-%
+test-enterprise: tmp/.image-id-enterprise
+> mvn test -Dimage=$$(cat $<) -Dedition=enterprise -Dversion=$(NEO4JVERSION) -Dtest=$(TESTS)
+.PHONY: test-enterprise
+
+test-community: tmp/.image-id-community
+> mvn test -Dimage=$$(cat $<) -Dedition=community -Dversion=$(NEO4JVERSION) -Dtest=$(TESTS)
+.PHONY: test-community
+
+# just build the images, don't test or package
+build: tmp/.image-id-community tmp/.image-id-enterprise
+.PHONY: build
+
+# create release images and loadable images
+package: package-community package-enterprise
+.PHONY: package
+
+package-community: tmp/.image-id-community out/community/.sentinel
+> mkdir -p out
+> docker tag $$(cat $<) neo4j:$(NEO4JVERSION)
+> docker save neo4j:$(NEO4JVERSION) > out/neo4j-community-$(NEO4JVERSION)-docker-loadable.tar
+
+package-enterprise: tmp/.image-id-enterprise out/enterprise/.sentinel
+> mkdir -p out
+> docker tag $$(cat $<) neo4j:$(NEO4JVERSION)-enterprise
+> docker save neo4j:$(NEO4JVERSION)-enterprise > out/neo4j-enterprise-$(NEO4JVERSION)-docker-loadable.tar
+
+out/%/.sentinel: tmp/image-%/.sentinel
 > mkdir -p $(@D)
 > cp -r $(<D)/* $(@D)
 > touch $@
 
-tmp/.tests-pass-%: tmp/.image-id-% $(shell find test -name 'test-*') $(shell find test -name '*.yml') $(shell find test -name '*.sh')
-> mkdir -p $(@D)
-> image_id=$$(cat $<)
-> for test in $(filter test/test-%,$^); do
->   echo "Running $${test}"
->   "$${test}" "$${image_id}" "${series}" "$*"
-> done
-> touch $@
+# building the image
 
 tmp/.image-id-%: tmp/local-context-%/.sentinel
 > mkdir -p $(@D)
 > image=test/$$RANDOM
 > docker build --tag=$$image \
-    --build-arg="NEO4J_URI=file:///tmp/$(call tarball,$*,$(NEO4J_VERSION))" \
+    --build-arg="NEO4J_URI=file:///tmp/$(call tarball,$*,$(NEO4JVERSION))" \
     $(<D)
 > echo -n $$image >$@
+> echo "NEO4JVERSION=$(NEO4JVERSION)" > tmp/devenv-${*}.env
+> echo "NEO4J_IMAGE=$$image" >> tmp/devenv-${*}.env
+> echo "NEO4J_EDITION=${*}" >> tmp/devenv-${*}.env
 
-tmp/local-context-%/.sentinel: tmp/image-%/.sentinel in/$(call tarball,%,$(NEO4J_VERSION))
+tmp/neo4jlabs-plugins.json: ./neo4jlabs-plugins.json
+> mkdir -p $(@D)
+> cp $< $@
+
+tmp/local-context-%/.sentinel: tmp/image-%/.sentinel in/$(call tarball,%,$(NEO4JVERSION)) tmp/neo4jlabs-plugins.json
 > rm -rf $(@D)
 > mkdir -p $(@D)
 > cp -r $(<D)/* $(@D)
 > cp $(filter %.tar.gz,$^) $(@D)/local-package
+> cp $(filter %.json,$^) $(@D)/local-package
 > touch $@
 
-tmp/image-%/.sentinel: src/$(series)/Dockerfile src/$(series)/docker-entrypoint.sh \
-                       in/$(call tarball,%,$(NEO4J_VERSION))
+tmp/image-%/.sentinel: docker-image-src/$(series)/Dockerfile docker-image-src/$(series)/docker-entrypoint.sh \
+                       in/$(call tarball,%,$(NEO4JVERSION)) tmp/neo4jlabs-plugins.json
 > mkdir -p $(@D)
 > cp $(filter %/docker-entrypoint.sh,$^) $(@D)/docker-entrypoint.sh
 > sha=$$(shasum --algorithm=256 $(filter %.tar.gz,$^) | cut -d' ' -f1)
 > <$(filter %/Dockerfile,$^) sed \
     -e "s|%%NEO4J_SHA%%|$${sha}|" \
-    -e "s|%%NEO4J_TARBALL%%|$(call tarball,$*,$(NEO4J_VERSION))|" \
+    -e "s|%%NEO4J_TARBALL%%|$(call tarball,$*,$(NEO4JVERSION))|" \
     -e "s|%%NEO4J_EDITION%%|$*|" \
     -e "s|%%NEO4J_DIST_SITE%%|$(dist_site)|" \
     >$(@D)/Dockerfile
 > mkdir -p $(@D)/local-package
+> cp $(filter %.json,$^) $(@D)/local-package
 > touch $(@D)/local-package/.sentinel
 > touch $@
 
-run = trapping-sigint \
-    docker run --publish 7474:7474 --publish 7687:7687 \
-    --env=NEO4J_ACCEPT_LICENSE_AGREEMENT=yes \
-    --env=NEO4J_AUTH=neo4j/foo --rm $$(cat $1)
-build-enterprise: tmp/.image-id-enterprise
-> @echo "Neo4j $(NEO4J_VERSION)-enterprise available as: $$(cat $<)"
-build-community: tmp/.image-id-community
-> @echo "Neo4j $(NEO4J_VERSION)-community available as: $$(cat $<)"
-run-enterprise: tmp/.image-id-enterprise
-> $(call run,$<)
-run-community: tmp/.image-id-community
-> $(call run,$<)
-test-enterprise: tmp/.tests-pass-enterprise
-test-community: tmp/.tests-pass-community
-.PHONY: run-enterprise run-community build-enterprise build-community test-enterprise test-community
-
 fetch_tarball = curl --fail --silent --show-error --location --remote-name \
-    $(dist_site)/$(call tarball,$(1),$(NEO4J_VERSION))
+    $(dist_site)/$(call tarball,$(1),$(NEO4JVERSION))
 
-cache: in/neo4j-%-$(NEO4J_VERSION)-unix.tar.gz
+cache: in/neo4j-%-$(NEO4JVERSION)-unix.tar.gz
 .PHONY: cache
 
-in/neo4j-community-$(NEO4J_VERSION)-unix.tar.gz:
+in/neo4j-community-$(NEO4JVERSION)-unix.tar.gz:
 > mkdir -p in
 > cd in
 > $(call fetch_tarball,community)
 
-in/neo4j-enterprise-$(NEO4J_VERSION)-unix.tar.gz:
+in/neo4j-enterprise-$(NEO4JVERSION)-unix.tar.gz:
 > mkdir -p in
 > cd in
 > $(call fetch_tarball,enterprise)

README.md

@@ -1,34 +1,41 @@
 *NOTE:* Supported images are available in the [official image library](https://hub.docker.com/_/neo4j/) on Docker Hub.
 Please use those for production use.
 
-# Using the Neo4j Docker image
+# Using the Neo4j Docker Image
 
-## Neo4j 2.3
+Documentation for the Neo4j image can be found [here](https://neo4j.com/docs/operations-manual/current/deployment/single-instance/docker/).
 
-Documentation for the Neo4j 2.3 image can be found [here](https://neo4j.com/developer/docker-23/).
-
-You can start a Neo4j 2.3 container like this:
+You can start a Neo4j container like this:
 
 ```
 docker run \
-    --publish=7474:7474 \
+    --publish=7474:7474 --publish=7687:7687 \
     --volume=$HOME/neo4j/data:/data \
-    neo4j:2.3
+    --volume=$HOME/neo4j/logs:/logs \
+    neo4j:latest
 ```
 
-## Neo4j 3.0
-
-Documentation for the Neo4j 3.0 image can be found [here](http://neo4j.com/docs/operations-manual/current/deployment/single-instance/docker/).
-
-You can start a Neo4j 3.0 container like this:
+To start a Neo4j Enterprise Edition container, you can run:
 
 ```
 docker run \
     --publish=7474:7474 --publish=7687:7687 \
+    --env=NEO4J_ACCEPT_LICENSE_AGREEMENT=yes \
     --volume=$HOME/neo4j/data:/data \
-    neo4j:3.0
+    --volume=$HOME/neo4j/logs:/logs \
+    neo4j:enterprise
 ```
 
+Mounting the `/data` and `/logs` folder is optional, 
+but it means that data can persist between closing and reopening Neo4j containers.
+
+
+# Building and Developing the Neo4j Docker Image
+
+See [DEVELOPMENT.md](DEVELOPMENT.md)
+
 # Getting support and contributing
 
-Please create issues and pull requests in the Github repository.
+For bug reports and feature requests, please create issues and pull requests against this Github repository.
+
+If you need guidance with using Neo4j you can ask questions here: https://community.neo4j.com/