Improve Docker support

[Makeshift]

Hi,

This PR adds some Docker best-practice changes to:

• Improve caching (for faster builds)
• Reduce image size (Only a tiny bit - I'm pretty sure it could be a lot smaller without using the Playright base image, but I didn't get around to it)
• Have Travis build the image from the code in the commit rather than latest main branch
• Have Travis also build an arm64 variant (I wanted to run it on my Pi). Note I haven't tested the Travis build as I'm not familiar with it.

For some reason I couldn't get the config to read from the current working directory within the container, so I added an env var to set the src location within the container, defaulting to . (current working directory) if not set, which I believe should work. My python isn't great, so please feel free to let me know if I made a mistake.

[Makeshift]

I was curious: Removing Playwright as a dependency (and the plugin that depends on it) brings the container size down to 141mb (from ~2gb), but obviously removes support for steamdb which is probably unwanted 😛

[lupohan44]

You forgot to change each plugin's config path as well

[lupohan44]

For the playwright part, I don't think it is neccessary to use docker if steamdb plugin is not needed. The only reason for this docker is to support for playwright's complicated requirements.

I will help test CI build.

[Makeshift]

You forgot to change each plugin's config path as well

Oops! Weirdly it still worked for me, and I'm not sure why. Python was being very strange with the current working directory - I'll pull it up in a debugger and see if I did something silly.

For the playwright part, I don't think it is neccessary to use docker if steamdb plugin is not needed. The only reason for this docker is to support for playwright's complicated requirements.

That's fair. I run all my services in docker just for convenience, which is why I modified it to work a bit more generically rather than being focused around playwright. Would you accept a second Dockerfile (maybe built with a different tag?) that removes the steamdb plugin and playwright dependency, since you don't recommend using it anyway (according to your readme)?

I will help test CI build.

Thanks!

Cheers for bearing with me - I'm not super familiar with python or Travis, but thought the changes I made to get this running on my pi might be useful to others.

[lupohan44]

Would you accept a second Dockerfile (maybe built with a different tag?) that removes the steamdb plugin and playwright dependency, since you don't recommend using it anyway (according to your readme)?

Of course, that is a good idea. You can use other base image for that.

[Makeshift]

I added an env var that tells Gameshub to either use its src directory or the current working directory for config files, and I think I updated all references.

I think that should work. I've tested both Dockerfiles and running the app outside of Docker and get the same output on both ARM+AMD64 (with no errors), but it's a bit hard to tell if it's working since the application is pretty quiet when there's nothing to do.

[lupohan44]

There are a few issues.

1. When I tested the image with CI, it appeared that something went wrong while building docker. Can you please help check it out? (Testing branch: test) Error Log
2. In all plugins, you should not change config_example_path = os.path.join(os.path.split(os.path.realpath(__file__))[0], "config.example.json5") to config_example_path = os.path.join(GAMESHUB_SRC_DIR, os.path.split(os.path.realpath(__file__))[0], "config.example.json5"). This is
   deliberately using the current file's folder.
3. (Not a big issue.) It is not necessary to use all requirements.txt files from all plugins. The requirements.txt file at the top level contains all the required packages. Though no harm will be done,

[lupohan44]

Both docker build error can be recreated in my Ubuntu VM.

#0 58.43 /tmp/apt-dpkg-install-Q5Ssf3/43-python3-wheel_0.34.2-1_all.deb
#0 58.43 /tmp/apt-dpkg-install-Q5Ssf3/44-python3-pip_20.0.2-5ubuntu1.6_all.deb
#0 58.62 E: Sub-process /usr/bin/dpkg returned an error code (1)

Dockerfile:3

2 |
3 | >>> RUN apt-get update
4 | >>> && apt-get --no-install-recommends install -y python3-dev python3-pip build-essential
5 | >>> && ln -s /usr/bin/pip3 /usr/bin/pip3.8
6 | >>> && rm -rf /var/lib/apt/lists/*
7 |

ERROR: failed to solve: process "/dev/.buildkit_qemu_emulator /bin/sh -c apt-get update && apt-get --no-install-recommends install -y python3-dev python3-pip build-essential && ln -s /usr/bin/pip3 /usr/bin/pip3.8 && rm -rf /var/lib/apt/lists/*" did not complete successfully: exit code: 100

=> CANCELED [linux/amd64 build-deps 6/6] RUN rm -rf plugins/gameshub.official.scraper.steamdb && find . -name 'requ 41.0s
=> [linux/arm64 build-deps 3/6] WORKDIR /tmp/python_requirements 0.1s
=> [linux/arm64 build-deps 4/6] COPY ./requirements-no_plugin.txt . 0.0s
=> [linux/arm64 build-deps 5/6] COPY plugins plugins 0.0s
=> ERROR [linux/arm64 build-deps 6/6] RUN rm -rf plugins/gameshub.official.scraper.steamdb && find . -name 'require 21.5s

[linux/arm64 build-deps 6/6] RUN rm -rf plugins/gameshub.official.scraper.steamdb && find . -name 'requirements*.txt' -print0 | while read -d $'\0' file; do pip3 install --user --no-cache-dir -r $file || exit 1; done:
#0 20.97 ERROR: Exception:
#0 20.97 Traceback (most recent call last):
#0 20.97 File "/usr/local/lib/python3.10/site-packages/pip/_internal/cli/base_command.py", line 160, in exc_logging_wrapper
#0 20.97 status = run_func(*args)
#0 20.97 File "/usr/local/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 247, in wrapper
#0 20.97 return func(self, options, args)
#0 20.97 File "/usr/local/lib/python3.10/site-packages/pip/_internal/commands/install.py", line 326, in run
#0 20.97 session = self.get_default_session(options)
#0 20.97 File "/usr/local/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 98, in get_default_session
#0 20.97 self._session = self.enter_context(self._build_session(options))
#0 20.97 File "/usr/local/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 125, in _build_session
#0 20.97 session = PipSession(
#0 20.97 File "/usr/local/lib/python3.10/site-packages/pip/_internal/network/session.py", line 343, in init
#0 20.97 self.headers["User-Agent"] = user_agent()
#0 20.97 File "/usr/local/lib/python3.10/site-packages/pip/_internal/network/session.py", line 140, in user_agent
#0 20.97 linux_distribution = distro.name(), distro.version(), distro.codename()
#0 20.97 File "/usr/local/lib/python3.10/site-packages/pip/_vendor/distro/distro.py", line 345, in version
#0 20.97 return _distro.version(pretty, best)
#0 20.97 File "/usr/local/lib/python3.10/site-packages/pip/_vendor/distro/distro.py", line 874, in version
#0 20.97 self.uname_attr("release"),
#0 20.97 File "/usr/local/lib/python3.10/site-packages/pip/_vendor/distro/distro.py", line 1059, in uname_attr
#0 20.97 return self._uname_info.get(attribute, "")
#0 20.97 File "/usr/local/lib/python3.10/functools.py", line 981, in get
#0 20.97 val = self.func(instance)
#0 20.97 File "/usr/local/lib/python3.10/site-packages/pip/_vendor/distro/distro.py", line 1173, in _uname_info
#0 20.97 stdout = subprocess.check_output(cmd, stderr=subprocess.DEVNULL)
#0 20.97 File "/usr/local/lib/python3.10/subprocess.py", line 421, in check_output
#0 20.97 return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
#0 20.97 File "/usr/local/lib/python3.10/subprocess.py", line 526, in run
#0 20.97 raise CalledProcessError(retcode, process.args,
#0 20.97 subprocess.CalledProcessError: Command '('uname', '-rs')' returned non-zero exit status 1.
#0 20.99 Traceback (most recent call last):
#0 20.99 File "/usr/local/bin/pip3", line 8, in
#0 20.99 sys.exit(main())
#0 21.00 File "/usr/local/lib/python3.10/site-packages/pip/_internal/cli/main.py", line 70, in main
#0 21.00 return command.main(cmd_args)
#0 21.00 File "/usr/local/lib/python3.10/site-packages/pip/_internal/cli/base_command.py", line 101, in main
#0 21.00 return self._main(args)
#0 21.00 File "/usr/local/lib/python3.10/site-packages/pip/_internal/cli/base_command.py", line 216, in _main
#0 21.00 self.handle_pip_version_check(options)
#0 21.00 File "/usr/local/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 179, in handle_pip_version_check
#0 21.00 session = self._build_session(
#0 21.00 File "/usr/local/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 125, in _build_session
#0 21.00 session = PipSession(
#0 21.00 File "/usr/local/lib/python3.10/site-packages/pip/_internal/network/session.py", line 343, in init
#0 21.00 self.headers["User-Agent"] = user_agent()
#0 21.01 File "/usr/local/lib/python3.10/site-packages/pip/_internal/network/session.py", line 140, in user_agent
#0 21.01 linux_distribution = distro.name(), distro.version(), distro.codename()
#0 21.01 File "/usr/local/lib/python3.10/site-packages/pip/_vendor/distro/distro.py", line 345, in version
#0 21.01 return _distro.version(pretty, best)
#0 21.01 File "/usr/local/lib/python3.10/site-packages/pip/_vendor/distro/distro.py", line 874, in version
#0 21.01 self.uname_attr("release"),
#0 21.01 File "/usr/local/lib/python3.10/site-packages/pip/_vendor/distro/distro.py", line 1059, in uname_attr
#0 21.01 return self._uname_info.get(attribute, "")
#0 21.01 File "/usr/local/lib/python3.10/functools.py", line 981, in get
#0 21.02 val = self.func(instance)
#0 21.02 File "/usr/local/lib/python3.10/site-packages/pip/_vendor/distro/distro.py", line 1173, in _uname_info
#0 21.02 stdout = subprocess.check_output(cmd, stderr=subprocess.DEVNULL)
#0 21.02 File "/usr/local/lib/python3.10/subprocess.py", line 421, in check_output
#0 21.02 return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
#0 21.02 File "/usr/local/lib/python3.10/subprocess.py", line 526, in run
#0 21.02 raise CalledProcessError(retcode, process.args,
#0 21.02 subprocess.CalledProcessError: Command '('uname', '-rs')' returned non-zero exit status 1.

---

WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
Dockerfile.alpine:11

10 |
11 | >>> RUN rm -rf plugins/gameshub.official.scraper.steamdb
12 | >>> && find . -name 'requirements*.txt' -print0 | while read -d $'\0' file; do pip3 install --user --no-cache-dir -r $file || exit 1; done
13 |

ERROR: failed to solve: process "/dev/.buildkit_qemu_emulator /bin/sh -c rm -rf plugins/gameshub.official.scraper.steamdb && find . -name 'requirements*.txt' -print0 | while read -d $'\0' file; do pip3 install --user --no-cache-dir -r $file || exit 1; done" did not complete successfully: exit code: 1

[lupohan44]

Hi, any updates?