Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GitOps - Stacks managed by Git #471

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

Felioh
Copy link

@Felioh Felioh commented Mar 30, 2024

⚠️⚠️⚠️ Since we do not accept all types of pull requests and do not want to waste your time. Please be sure that you have read pull request rules:
https://github.com/louislam/dockge/blob/master/CONTRIBUTING.md

Tick the checkbox if you understand [x]:

  • I have read and understand the pull request rules.

Description

This is an initial draft of a GitOps integration, related to #36 . Including:

  • specifying a git repository (via URL) upon creation of a stack
    • private repositories can be used when including an access token in the URL
    • a stack is intended to always follow a branch
  • updating the stack automatically
    • by enabling a scheduled check for remote updates (currently 10min) - this is disabled by default
    • by triggering the update via webhook for a specific stack

Since I have not discussed this before, I first would like to know if this is a feature that you (@louislam) would like to see included in this project. If you do, I am happy to apply any wanted changes.

I have some stuff in mind that could be really cool here, but that I would like to discuss before.
In the future allowing for local changes (displaying the editor in the edit view) and adding an option to commit/push could be implemented. (local changes should not be a problem with the way it is implemented right now, and should just be overwritten by any merge-conflicts with the remote).

I have tried to follow conventions, but I have never used ts, js, or vue before so if I can improve anything, please let me know :).

Type of change

Please delete any options that are not relevant.

  • User interface (UI)
  • New feature (non-breaking change which adds functionality)
  • This change requires a documentation update

Checklist

  • My code follows the style guidelines of this project
  • I ran ESLint and other linters for modified files
  • I have performed a self-review of my own code and tested it
  • I have commented my code, particularly in hard-to-understand areas
    (including JSDoc for methods)
  • My changes generate no new warnings
  • My code needed automated testing. I have added them (this is optional task)

Screenshots (if any)

Please do not use any external image service. Instead, just paste in or drag and drop the image here, and it will be uploaded automatically.
image
image
image
image

@Felioh Felioh marked this pull request as ready for review April 1, 2024 11:32
Copy link

@nathan815 nathan815 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will be a great feature addition!

How about ability to specify repo subdirectory for a stack? My use case is I have a "stacks" directory in a repo instead of a repo per stack with the docker-compose at root.


let exitCode = await Terminal.exec(server, socket, terminalName, "git", [ "clone", "-b", branch, gitUrl, stackName ], server.stacksDir);
if (exitCode !== 0) {
throw new Error("Failed to clone git repo");

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would including the exit code in this error message be helpful?

stderr would be nice, but looks like Terminal.exec only exposes the exit code so I suppose that isn't possible without other changes.

@Felioh
Copy link
Author

Felioh commented Apr 29, 2024

How about ability to specify repo subdirectory for a stack? My use case is I have a "stacks" directory in a repo instead of a repo per stack with the docker-compose at root.

I think this is related to #214, and actually exactly my use case too.
I am not sure how to implement this, maybe you have some ideas. Currently we check if a directory in the root contains a files of acceptedComposeFileNames. I would relax that to a maximum depth of 3 (or 2?).
It could also be implemented by specifying the subpath on creation, but that makes things more complex, because we do not really want to save any additional information for a stack.

If you like I can do that, since I am interested in this too, but I think it should be a separate PR.

@Felioh Felioh requested a review from nathan815 April 29, 2024 19:12
@buldezir
Copy link

awesome proposal!

maybe its possible to play with git clone --depth 1 or similar? cause repos could be quite heavy, and for deploy we usually need only compose file

@Charghoul
Copy link

this looks amazing! just what i've been waiting for. when can it be merged? :)

@zongkx
Copy link

zongkx commented Jun 19, 2024

when to merged?

@Plarpoon
Copy link

That looks amazing and it looks like all checks have passed too! When is this beauty going to be merged?

@cyril59310
Copy link
Contributor

Even if you are eager for this feature, avoid asking when it will be merged as it spams unnecessarily.
Additionally, for now, Louislam is focusing on version 2.0 of his other project, Uptime Kuma. He will return to this project afterward.

Copy link

@mh166 mh166 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How would one enter credentials for the git repos if they are not public? I did not find anything in the PR about it. I'm not saying you need to implement credential handling (would be way out of scope, I think). However, maybe you can add some documentation about it to the wiki to make it easier and prevent upcoming issues about it.

<div class="form-check form-switch my-3">
<input id="git-auto-update" v-model="settings.gitAutoUpdate" class="form-check-input" type="checkbox">
<label class="form-check-label">
{{ $t("enableAutoUpdate") }}
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it make sense to display the current update interval here? So that people know what to expect from "periodical checks". Or maybe you could even offer an option to set a custom interval. Either would improve the user experience. What do you think?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How would one enter credentials for the git repos if they are not public? I did not find anything in the PR about it. I'm not saying you need to implement credential handling (would be way out of scope, I think). However, maybe you can add some documentation about it to the wiki to make it easier and prevent upcoming issues about it.

Currently, private repositories can be used by including an access-token in the url https://username:token@github.com/username/repo.git. This is not ideal and I would like to improve this. However I dont think it is in the spirit of this project to add credential handling. Possibly we could have dockge generate its own private/public key pair and require the user to add the public key to their github profile. That way the repo could be cloned via ssh and the stacks could still be managed without dockge by any user that is authorized.

Would it make sense to display the current update interval here? So that people know what to expect from "periodical checks". Or maybe you could even offer an option to set a custom interval. Either would improve the user experience. What do you think?

I agree, I would like to add an option to configure the update interval in the settings (next to enabling the periodic checks)

However, this PR was intended as a POC. To this point I am not certain @louislam even wants this functionality in his project so I wont invest any more time in this feature until I get a reaction.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I appreciate your replies. Both of them absolutely make sense and I agree with them. Like you said, this is a POC - just wanted to make sure you are aware of these two points. 😊

Thanks for the effort and time you put into this! 🥳 Hope that it will be released eventually. :)

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as i had the same question as @mh166, i will document my findings here.

problem

we are using a private gitlab instance with a private docker registry. both have their own token we use for deployments.

our workaround

as @Felioh mentioned, one can put the token into the repo url. but there is another way (which also allows to change tokens later):

git can pull its config from the environment (see docs). git also has a means of filling the credentials from somewhere else (see docs).

we provide these environment variables to the dockge container to enable private repository access

services:
  dockge:
    environment:
      - GITLAB_TOKEN=thisisatoken
      - GIT_CONFIG_COUNT=2
      - GIT_CONFIG_KEY_0=credential.https://git.example.com.username
      - GIT_CONFIG_VALUE_0=dockge_deploy
      - GIT_CONFIG_KEY_1=credential.https://git.example.com.helper
      - GIT_CONFIG_VALUE_1=!f()·{·test·"$$1"·=·get·&&·echo·"password=$${GITLAB_TOKEN}";·};·f

last thing to conquer is the access token to enable use of the private registry.
to do that, we specify a docker config containing the auth to the dockge-container:

services:
  dockge:
    configs:
      - source: docker_json
        target: /root/.docker/config.json

configs:
  docker_json:
    content: |
      {
        "auths": {
          "registry.example.com:5050": {
            "auth": "LONGrandomAUTHstring"
          }
        }
      }

HINT: you can not just put the access token into the "auth"-field above. instead: use docker login once and put credentials in there. then copy the relevant parts from ~/.docker/config.json.

result

we are now able to use @Felioh's dockge version without having to manually execute commands or configs within a fresh dockge-instance and without opening repos or registries to the public. as an added benefit, we are easily able to change the gitlab token and just recreate our dockge container.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for sharing your findings @elgarfo , however, I think I should point out that the 'randomAuthString'

"auth": "LONGrandomAUTHstring"

is not random but rather just username:password base64 encoded. Therefore you should be careful using this.

create(app: Express, server: DockgeServer): ExpressRouter {
const router = express.Router();

router.get("/webhook/update/:stackname", async (req, res, _next) => {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So if i know or find out the stack name I can update it a few hundred times a day, just for fun..? 😉 Usually webhooks are kind of "masked" by using a random string.

@tobi1449
Copy link

I'm looking forward to being able to use this :)
Two thoughts:

  • You're using branches here to allow multiple compose files in the same repo. Portainer for example also allows specifying the path to the compose file you want to use directly. With that you could have multiple in a single repo & branch and enable reuse of other compose files, .env files, etc.
  • Having the ability to specify a preprocessor like dotenvx that is run on the compose file right after the pull from git would also be great

@rogueai
Copy link

rogueai commented Oct 30, 2024

Hi, are there any updates on this? I'm really looking forward to be able to do what's proposed here, without having to deal with something like Portainer, which is a bit overkill for my use case...

I can help with testing, if that might help. Thanks!

@Felioh
Copy link
Author

Felioh commented Nov 13, 2024

This was supposed to be kind of a proof of concept and I would like to work further on this. However, before investing any more time, I want to ensure this feature is wanted.

@Plarpoon
Copy link

It surely is wanted by me!

@smkamranqadri
Copy link

Defiantly wanted and highly appreciated if it get merged too.

@Felioh Felioh mentioned this pull request Nov 17, 2024
7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.