Broken dependency evaluation in pip 24.3.1. Pip will not recognize installed prerelease versions

[maarre]

Description

I want to be able to test multiple prerelease versions together without having to shoehorn installations.

I really don't want to put prerelease versions in my pyproject.toml.

Current version of pip will only allow prerelease dependencies using --no-deps flag.

$ python --version
Python 3.11.2
$ python -m pip --version
pip 24.3.1 from xxxxx\Python311-64\Lib\site-packages\pip (python 3.11)
$ pip list | grep spdbtools
spdbtools                 0.7.1a20241118121953
$ python -m pip install dist/postgresqlpatches-0.4.0a20241118163330-py3-none-any.whl
Looking in indexes: https://python.repo.sfa.se/repository/all/simple
Processing c:\fkapps\repo\git\ios\python-postgresql-patches\dist\postgresqlpatches-0.4.0a20241118163330-py3-none-any.whl
Requirement already satisfied: psycopg2<3,>=2.9.5 in xxx\python311-64\lib\site-packages (from postgresqlpatches==0.4.0a20241118163330) (2.9.9)
INFO: pip is looking at multiple versions of postgresqlpatches to determine which version is compatible with other requirements. This could take a while.
ERROR: Could not find a version that satisfies the requirement spdbtools<1.0,>=0.7.1 (from postgresqlpatches) (from versions: 0.4.1, 0.5.0, 0.5.1, 0.5.2, 0.6.2, 0.6.3, 0.6.4)
ERROR: No matching distribution found for spdbtools<1.0,>=0.7.1
$ python -m pip install dist/postgresqlpatches-0.4.0a20241118163330-py3-none-any.whl --pre
Looking in indexes: https://python.repo.sfa.se/repository/all/simple
Processing c:\fkapps\repo\git\ios\python-postgresql-patches\dist\postgresqlpatches-0.4.0a20241118163330-py3-none-any.whl
Requirement already satisfied: psycopg2<3,>=2.9.5 in c:\fkapps\python311-64\lib\site-packages (from postgresqlpatches==0.4.0a20241118163330) (2.9.9)
INFO: pip is looking at multiple versions of postgresqlpatches to determine which version is compatible with other requirements. This could take a while.
ERROR: Could not find a version that satisfies the requirement spdbtools<1.0,>=0.7.1 (from postgresqlpatches) (from versions: 0.4.1, 0.5.0, 0.5.1, 0.5.2, 0.6.2, 0.6.3, 0.6.4)
ERROR: No matching distribution found for spdbtools<1.0,>=0.7.1
$ python -m pip install dist/postgresqlpatches-0.4.0a20241118163330-py3-none-any.whl --no-deps
Looking in indexes: https://python.repo.sfa.se/repository/all/simple
Processing xxxx\git\ios\python-postgresql-patches\dist\postgresqlpatches-0.4.0a20241118163330-py3-none-any.whl
Installing collected packages: postgresqlpatches
Successfully installed postgresqlpatches-0.4.0a20241118163330

Expected behavior

If there is an alpha, beta or release candidate installed i want pip to accept this installation when doing the dependency evaluation. The installations should succeed without having to use --no-deps switch.

pip version

24.3.1

Python version

3.11.2

OS

Windows, Linux

How to Reproduce

1 Create a package with an alpha version.´
2 Build the first package
3 Install the first package
4 Create another package with a dependency to the first package
5 The pyproject.toml file should reference the version of the first package without the alpha specifier.
6 Build the second package
7 Install the second package

Output

No response

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pfmoore]

Doesn’t the --pre option do exactly this?

[maarre]

$ python -m pip install dist/postgresqlpatches-0.4.0a20241118163330-py3-none-any.whl --pre
Looking in indexes: https://python.repo.sfa.se/repository/all/simple
Processing c:\fkapps\repo\git\ios\python-postgresql-patches\dist\postgresqlpatches-0.4.0a20241118163330-py3-none-any.whl
Requirement already satisfied: psycopg2<3,>=2.9.5 in c:\fkapps\python311-64\lib\site-packages (from postgresqlpatches==0.4.0a20241118163330) (2.9.9)
INFO: pip is looking at multiple versions of postgresqlpatches to determine which version is compatible with other requirements. This could take a while.
ERROR: Could not find a version that satisfies the requirement spdbtools<1.0,>=0.7.1 (from postgresqlpatches) (from versions: 0.4.1, 0.5.0, 0.5.1, 0.5.2, 0.6.2, 0.6.3, 0.6.4)
ERROR: No matching distribution found for spdbtools<1.0,>=0.7.1

[pfmoore]

Hang on - your requirement is spdbtools<1.0,>=0.7.1, but you have 0.7.1a20241118121953. According to the version specification:

Within a numeric release (1.0, 2.7.3), the following suffixes are permitted and MUST be ordered as shown:
.devN, aN, bN, rcN, <no suffix>, .postN

So your alpha release does not satisfy the constraint >=0.7.1.

Pip is behaving according to spec here.

[notatallshaw]

Yeah, it's a part of the spec that I see a lot of people confused by, but 1.0a1 is not satisfied by >=1.0 even with pre release available.

You must specify >0.7.0 and enable pre releases (--pre).

[potiuk]

Pip is behaving according to spec here.

Yes. I looked very closely at those in the past. It's according to the specification. And specification is good. But still it does not make it easy for the workflow described by @maarre.

You must specify >0.7.0 and enable pre releases (--pre).

Yes, but this is a bit problematic becasue pip does not support per-package --pre release - you can only enable --pre for all resolution, not for individual packages.

By adding --pre you allow pre-releases of ALL packages - which means that you will install pre-releases that you might absolutely not want - if you have many packages as dependencies and you have it specified as >= and your resolution normally returns latest available version of the packages, you are almost guaranteed to break something by using --pre - because someone is testing something and pushed a broken pre-release to pypI. Been there, done that (on both sides).

And it does make it very tricky when you have a workflow of releasing two packages at the same time and want to make rc candidates of package A that might become final candidate and you want it to depend on the new version of the package B - that you also want to release as an rc candidate at the same time.

Say: you releae A==0.1.rc1 that should (when final) depend on B>=0.2 - but you release the B==0.2.rc1 at the same time.

We have this very problem very often in Airflow - when it often happens that at the same we want to relese a new common package with new feature (say airflow-providers-common-sql) and another package that depends on the new features added in airflow-providers-common-sql (say airflow-providers-postgres==1.0.0rc1 that depends on features added in airflow-providers-common-sql==2.1.0rc1). The final dependency will have to be apache-alrflow-providers-postgres==1.0.0 should depend on apache-airlfow-providers-common.sql>=2.1.0. But if our postgres 1.0.0rc1 provider has common-sql>=2.1.0 - that will not work.

And this is problematic because you effectively have to modify your dependencies in either git repo or relased packages between the RC and final version. That's not good workflow - for security and relase process especially - modifying code between rc and final version is problematic especially if you already started to develop new version and you release from main (it basically forces you to create branches or dynamic dependency modification while you are building the package).

Our solution in airflow (really a workaround) that we found working for us is to dynamically modify the rc packages with rc dependencies for all our rc packages. In the example above when we generate apache-airflow-providers-postgres==1.0.0rc1 we check all the required dependencies it has and if we find any apache-airflow-providers-*>=X.Y.Z - we dynamically change them to apache-airlfow-provider-....>=X.Y.Zrc1 (so in this case apache-airflow-providers-common-sql>=2.1.0 will be dynamically modified to apache-airflow-providers-common-sql>=2.1.0rc1. Since 2.1.0rc1 is the only one that satisfies the requirement, it will be used even without --pre flag. This is a special case where using --pre is not even needed to install pre-release version.

THis way in our pyproject.toml/hatch_build.py we keep the final dependency (apache-airflow-providers-common-sql>=2.1.0) and we do not have to worry about modifying it between rc and final version.

It's a workable solution and works good for us in Airflow - so maybe you can adapt it as well @maarre - however it requires dynamic generation of requirements and quite some automation of your release process.

Another, better solution that maybe we might see in the future is to be able to specify selectively which packages should be treated differently, say if we could specify which packages could be treated differently with pre-releases we could use somethign like --allow-pre-releases-of "apache-airflow-providers-common-sql,apache-airflow-providers-common-io" - and for those packages automatically treat >=0.7.2 as (>=0.7.2.dev0).

Another option is what uv does - where you can selectively override specific dependencies when you install packages https://docs.astral.sh/uv/concepts/resolution/#dependency-overrides - that would effectively serve even more use cases (though I think this one gives far too much freedom to the users, and it's a little bit of footgun for uv when implemented this way). And I would not recommend it.

[notatallshaw]

By adding --pre you allow pre-releases of ALL packages - which means that you will install pre-releases that you might absolutely not want

There's a workaround to get per package prerelease, create a constraints file prereleases.txt:

package>=0.0dev0

Then you can add -c prereleases.txt, the prerelease specifier now allows prereleases for just that package.

[potiuk]

Then you can add -c prereleases.txt, the prerelease specifier now allows prereleases for just that package.

This is only half of a solution. You'd still need to modify dynamically your RC package requirement. In the example above, I want postgres 1.0.0rc1 depend on common-sql>=2.1.0. Adding constraints file common-sql>=2.1.0.rc1 will not change anything because common-sql>=2.1.0 will exclude common-sqll 2.1.0.rc1 (because rc1 is lower than 2.1.0).

So the only solution I see now is to dynamically modify rc1 packages to modify their dependencies >=X.Y.Z.rc1 - all while RC packages are being built.

Again - I want to avoid to have to bump >=X.Y.Z.rc1 to >=X.YZ in the repository between RC and final version, so if I "naively" build RC package from the same sources as final it will always have >=X.YZ.

The flag that could solve it will have to dp two things:

• allow pre-release flags
• allow to install X.Y.Z.rc1/.dev0 etc. packages even if the requirement is >=X.Y.Z. Basically it means violating the ordering of pre-releases for X.YZ if >=X.Y.Z is specified for that package as requirement.

[notatallshaw]

As I read this it appears to me what you're asking for is a "light" override of the requirements. It's an interesting idea, but I don't see if getting any kind of traction with pip maintainers unless it became a standard. As maintainers have strongly expressed not wanting to help user install "broken" requirements.

There has been discussion of override interfaces before, e.g. #8076 (comment) where I proposed an exactly equivalent interface to uv's --override, back in 2022.

[potiuk]

As I read this it appears to me what you're asking for is a "light" override of the requirements. It's an interesting idea, but I don't see if getting any kind of traction with pip maintainers unless it became a standard. As maintainers have strongly expressed not wanting to help user install "broken" requirements.

Yes. but with a twist. I do not want (and I agree with maintainers here) to allow for broken dependencies. That would be very bad.

What I would see as a possible solution is to handle specific case where you want to treat pre-release candidates version comparision differently for the packages that have not yet been released (only pre-released). Simply recognising the fact that there is a use case where >=1.2 should also include 1.2.rc when there is no 1.2 released yet (maybe controlled by a flag).

[notatallshaw]

Simply recognising the fact that there is a use case where >=1.2 should also include 1.2.rc when there is no 1.2 released yet (maybe controlled by a flag).

That's a spec change, so the discussion must happen on the packaging forum, as it happens there is a discussion that is going on right now that has devolved into including that very question: https://discuss.python.org/t/proposal-intersect-and-disjoint-operations-for-python-version-specifiers/71888/20

I'm hoping though that discussion gets moved to it's own dedicated thread, I'll post back here if it does.

[pfmoore]

I'm hoping though that discussion gets moved to it's own dedicated thread

It will, and I intend to propose a clarified version of the spec. However, I will say right now that changing the fact that 1.2rc1 is before 1.2 (and hence doesn't satisfy >=1.2) is firmly off the table. There's no lack of clarity in the current spec over this behaviour, and I intend to strictly separate "clarifications" from "rules changes". Someone else would have to pick that up, after the current discussion has run its course.