Views on tag deletion prior to release

[matthewfeickert]

This is a transfer and summary of points brought up on PR #2183. So tagging people who were part of that discussion RE: Git tags: @webknjaz, @kratsg, @henryiii. Thanks again to @webknjaz for the time he's contributed to discussing these ideas with us!

There @webknjaz had mentioned

Though, I also don't create git tags / GH releases before the publishing since the build/publish may fail, causing the need to cleanup stuff.

and

It's easy for you [to delete a tag], sometimes. But by the time you do it, it may propagate into external systems (the ones that watch releases), local git repos of different people, forks made before you have a chance to clean up etc. These external side effects are harder to deal with, not talking about the convention that the tags should be immutable

As I mentioned in #2183 (comment), while I understand the idea that the tag is immutable (because it is) I don't see the problem with deleting a tag if there is a problem that was caught when it was made. Deleting a tag is not ideal, and should be rare, but I would prefer to delete a tag that has existed in a repository that I maintain for less than 10 minutes before a release is made then to treat a pre-release tag as an unchangeable component of Git history that can't be rewritten through deletion to avoid disruption in development and user experience (here I'm imagining a worse case scenario where a tag has been pushed to some branch through human error where it should never have landed).

I also want to ensure the provenance of the distribution by requiring that a tag with an associated GitHub release necessarily cause the creation and deployment of a distribution to PyPI from that tag. I should also mention that after a distribution is published then I very much want to treat that tag as undeletable as I want to keep provenance.

Also, what is an example of an external system that watches git tags? I do not know of any services that I use that do this. On the contrary, Zenodo is looking for a GitHub Release (as @kratsg pointed out in #2183 (comment)) and conda-forge is looking for a PyPI release (which for pyhf means a GitHub Release was made). I fully believe that there are services that are looking specifically for tags but if I as a maintainer am not using them, and I am not providing infrastructure as a service, then should I be constrained in how I want to manage a project given a service that exists that I don't support?

The current (pre-PR #2183) workflow of how pyhf publishes things might be relevant for how I'm thinking about things and can be found here #2183 (comment).

I'm by all admissions not an expert's expert here, so if there are things that I'm missing from security, or workflow best practices, or just overlooked problems I'd be quite interested to learn about them. At the moment not being able to rewrite Git history for a good reason feels like imposing SVN-like restrictions on my maintainer workflow for dealing with errors in deployment that never should happen unless gross human error has occurred. Can people explain more to me?

Though @webknjaz did mention

Not sure if this was clear but I did[n't] suggest using tags or releases as triggers. Rather vise versa — in my projects I use workflow_dispatch to kick things off and tags/releases are results of a successful workflow run. FWIW I'm not calling for immediate changes or how things are done here, just sharing what I got myself.

so I should probably be looking at workflows that he manages to understand this approach, though the provenance seems a bit off if the artifact is build before the tag is created.

In the event that this needs to be said, I very much welcome feedback from the broader Python community here, so if you were not tagged I would still like to hear from you.

[tacaswell]

I view the git tag is the point of truth that everything else in the release chain derives from (as the GH release is derived from the tag, the sdist is strictly derived from the tag, and any down stream packaging (like wheels, conda packages, linux packages, ...) are derived from the sdist).

If you create a tag "locally" either on your own machine or in a CI job but do not push it I think it is fair to change your mind and remake the tag (or just abort the whole process) because no one else has had a chance to see it.

However, once the tag is pushed to the canonical repository you have irrevocably used that name for than commit. If something goes wrong somewhere down stream from creating the tag I think the options are to either fix those downstream things in what ever way you have to or to fix them upstream and make a new tag.

At the end of the day incrementing a patch release number is super cheap compared to violating "tags never change".

Similarly, "a tag is missing artifacts" is free, but "the tag does not match the artifacts that have the same name", even if rare, can have a huge cost!

[matthewfeickert]

Thanks for taking the time to share your experiences @tacaswell.

I view the git tag is the point of truth that everything else in the release chain derives from

I agree with that very much, which is why I would want all tags to mean something.

Similarly, "a tag is missing artifacts" is free

How do you communicate to users that this tag has a problem and should never be used?

but "the tag does not match the artifacts that have the same name", even if rare, has can have a huge cost!

I also agree with this. But if there are no public artifacts then what is the cost of deleting a tag that has some defect to it, correcting that defect, and then restarting the process after things are fixed?

However, once the tag is pushed to the canonical repository you have irrevocably used that name for that commit.

I'm curious about the social conventions for this (as you can certainly actually do this) as this all makes plenty of sense to me once there are public artifacts that have been distributed but before hand this just seems like a choice. Maybe the idea of a social norm in the software community should be regarded as strongly though just to make sure that everything flows.

... because no one else has had a chance to see it.

Are we primarily concerned with humans here? Or are there automated workflows that I am not considering that then break very hard?

My assumption is that this discussion is going to go the way of my discussions with @henryiii and Hynek where I was very confused about their views on dependency bounds but now feel very strongly in favor of their views (Twitter, PR #1382). I am just trying to understand what I'm missing and how to get there at this point.

If people have specific examples of why not to delete a tag when there are no artifacts associated to it that would be helpful. I will also accept the idea that people just fundamentally don't think you should delete anything in Git and though you can revert a commit you can't revert a tag.

[tacaswell]

How do you communicate to users that this tag has a problem and should never be used?

If they are only getting artifacts from the system you communicate it should not be used by there being no artifacts ;) This effectively what yanking does (you only get it if you had it and know to ask for it...) but preepmtively (but in this case no one had it so the set of people who should be able to get it is empty so you don't have to build them).

Are we primarily concerned with humans here? Or are there automated workflows that I am not considering that then break very hard?

Either human or machine. You can not discount that someone has git pull in a cron job, detects new tags and does something about it.

But if there are no public artifacts

The tag is a public artifact (in my view). By pushing it to a public server you have lit the solid fuel rocket 🚀 .

I'm much more in favor of re-writing the history of public branches (we did this on Matplotlib to remove a full venv from the history) than of deleting a tag than of making a new tag with the same name.

[webknjaz]

I agree with what @tacaswell wrote in the root comment. He basically relays what I wanted to (and perhaps failed?)

How do you communicate to users that this tag has a problem and should never be used?

How about yanking a PyPI release?
Also, with a workflow of not having even pushed the tag when publishing to PyPI fails, you don't really need to do anything. Though the version might be burnt in TestPyPI, it's unlikely to happen.
Before publishing, you may have a local tag that only exists in CI to facilitate setuptools-scm, and after, it'd be published to GH unconditionally.

[webknjaz]

Maybe the idea of a social norm in the software community should be regarded as strongly though just to make sure that everything flows.

I think that the immutability of public things is a tribal knowledge in some areas and might be documented in the other ones. I was unable to find an article documenting this.

[webknjaz]

I will also accept the idea that people just fundamentally don't think you should delete anything in Git and though you can revert a commit you can't revert a tag.

In general, I wouldn't delete (or force-push for that matter) public tags or branches. Doing so immediately, within minutes is a gray area for me.
So yeah, I think it's a rather fundamental belief.