Skip to content

Commit

Permalink
Merge pull request #456 from greenc-FNAL/maintenance/merge-from-upstr…
Browse files Browse the repository at this point in the history 
…eam-2024-10-16

Merge from upstream 2024-10-16
  • Loading branch information
@greenc-FNAL
greenc-FNAL authored Oct 16, 2024
2 parents f5cf781 + 2ce08c7 commit fc40bdd
Show file tree
Hide file tree
Showing 53 changed files with 739 additions and 313 deletions.
19 changes: 10 additions & 9 deletions .github/workflows/unit_tests.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -154,26 +154,27 @@ jobs:
fetch-depth: 0
- uses: actions/setup-python@f677139bbe7f9c59b41e40162b753c062f5d49a3
with:
python-version: '3.11'
python-version: '3.13'
- name: Install System packages
run: |
sudo apt-get -y update
sudo apt-get -y install coreutils cvs gfortran graphviz gnupg2 mercurial ninja-build kcov
sudo apt-get -y install coreutils gfortran graphviz gnupg2
- name: Install Python packages
run: |
pip install --upgrade pip setuptools pytest coverage[toml] pytest-cov clingo pytest-xdist
pip install --upgrade pip setuptools pytest coverage[toml] pytest-cov clingo
pip install --upgrade flake8 "isort>=4.3.5" "mypy>=0.900" "click" "black"
- name: Setup git configuration
run: |
# Need this for the git tests to succeed.
git --version
. .github/workflows/bin/setup_git.sh
- name: Run unit tests (full suite with coverage)
env:
COVERAGE: true
COVERAGE_FILE: coverage/.coverage-clingo-cffi
run: |
share/spack/qa/run-unit-tests
. share/spack/setup-env.sh
spack bootstrap disable spack-install
spack bootstrap disable github-actions-v0.4
spack bootstrap disable github-actions-v0.5
spack bootstrap status
spack solve zlib
spack unit-test --verbose --cov --cov-config=pyproject.toml --cov-report=xml:coverage.xml lib/spack/spack/test/concretize.py
- uses: actions/upload-artifact@b4b15b8c7c6ac21ea08fcf65892d2ee8f75cf882
with:
name: coverage-clingo-cffi
Expand Down
227 changes: 107 additions & 120 deletions lib/spack/docs/developer_guide.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -712,27 +712,27 @@ Release branches
^^^^^^^^^^^^^^^^

There are currently two types of Spack releases: :ref:`major releases
<major-releases>` (``0.17.0``, ``0.18.0``, etc.) and :ref:`point releases
<point-releases>` (``0.17.1``, ``0.17.2``, ``0.17.3``, etc.). Here is a
<major-releases>` (``0.21.0``, ``0.22.0``, etc.) and :ref:`patch releases
<patch-releases>` (``0.22.1``, ``0.22.2``, ``0.22.3``, etc.). Here is a
diagram of how Spack release branches work::

o branch: develop (latest version, v0.19.0.dev0)
o branch: develop (latest version, v0.23.0.dev0)
|
o
| o branch: releases/v0.18, tag: v0.18.1
| o branch: releases/v0.22, tag: v0.22.1
o |
| o tag: v0.18.0
| o tag: v0.22.0
o |
| o
|/
o
|
o
| o branch: releases/v0.17, tag: v0.17.2
| o branch: releases/v0.21, tag: v0.21.2
o |
| o tag: v0.17.1
| o tag: v0.21.1
o |
| o tag: v0.17.0
| o tag: v0.21.0
o |
| o
|/
Expand All @@ -743,8 +743,8 @@ requests target ``develop``. The ``develop`` branch will report that its
version is that of the next **major** release with a ``.dev0`` suffix.

Each Spack release series also has a corresponding branch, e.g.
``releases/v0.18`` has ``0.18.x`` versions of Spack, and
``releases/v0.17`` has ``0.17.x`` versions. A major release is the first
``releases/v0.22`` has ``v0.22.x`` versions of Spack, and
``releases/v0.21`` has ``v0.21.x`` versions. A major release is the first
tagged version on a release branch. Minor releases are back-ported from
develop onto release branches. This is typically done by cherry-picking
bugfix commits off of ``develop``.
Expand Down Expand Up @@ -774,53 +774,62 @@ for more details.
Scheduling work for releases
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

We schedule work for releases by creating `GitHub projects
<https://github.com/spack/spack/projects>`_. At any time, there may be
several open release projects. For example, below are two releases (from
some past version of the page linked above):
We schedule work for **major releases** through `milestones
<https://github.com/spack/spack/milestones>`_ and `GitHub Projects
<https://github.com/spack/spack/projects>`_, while **patch releases** use `labels
<https://github.com/spack/spack/labels>`_.

.. image:: images/projects.png
There is only one milestone open at a time. Its name corresponds to the next major version, for
example ``v0.23``. Important issues and pull requests should be assigned to this milestone by
core developers, so that they are not forgotten at the time of release. The milestone is closed
when the release is made, and a new milestone is created for the next major release.

This image shows one release in progress for ``0.15.1`` and another for
``0.16.0``. Each of these releases has a project board containing issues
and pull requests. GitHub shows a status bar with completed work in
green, work in progress in purple, and work not started yet in gray, so
it's fairly easy to see progress.
Bug reports in GitHub issues are automatically labelled ``bug`` and ``triage``. Spack developers
assign one of the labels ``impact-low``, ``impact-medium`` or ``impact-high``. This will make the
issue appear in the `Triaged bugs <https://github.com/orgs/spack/projects/6>`_ project board.
Important issues should be assigned to the next milestone as well, so they appear at the top of
the project board.

Spack's project boards are not firm commitments so we move work between
releases frequently. If we need to make a release and some tasks are not
yet done, we will simply move them to the next minor or major release, rather
than delaying the release to complete them.
Spack's milestones are not firm commitments so we move work between releases frequently. If we
need to make a release and some tasks are not yet done, we will simply move them to the next major
release milestone, rather than delaying the release to complete them.

For more on using GitHub project boards, see `GitHub's documentation
<https://docs.github.com/en/github/managing-your-work-on-github/about-project-boards>`_.
^^^^^^^^^^^^^^^^^^^^^
Backporting bug fixes
^^^^^^^^^^^^^^^^^^^^^

When a bug is fixed in the ``develop`` branch, it is often necessary to backport the fix to one
(or more) of the ``release/vX.Y`` branches. Only the release manager is responsible for doing
backports, but Spack maintainers are responsible for labelling pull requests (and issues if no bug
fix is available yet) with ``vX.Y.Z`` labels. The label should correspond to the next patch version
that the bug fix should be backported to.

Backports are done publicly by the release manager using a pull request named ``Backports vX.Y.Z``.
This pull request is opened from the ``backports/vX.Y.Z`` branch, targets the ``releases/vX.Y``
branch and contains a (growing) list of cherry-picked commits from the ``develop`` branch.
Typically there are one or two backport pull requests open at any given time.

.. _major-releases:

^^^^^^^^^^^^^^^^^^^^^
Making major releases
^^^^^^^^^^^^^^^^^^^^^

Assuming a project board has already been created and all required work
completed, the steps to make the major release are:

#. Create two new project boards:
Assuming all required work from the milestone is completed, the steps to make the major release
are:

* One for the next major release
* One for the next point release
#. `Create a new milestone <https://github.com/spack/spack/milestones>`_ for the next major
release.

#. Move any optional tasks that are not done to one of the new project boards.
#. `Create a new label <https://github.com/spack/spack/labels>`_ for the next patch release.

In general, small bugfixes should go to the next point release. Major
features, refactors, and changes that could affect concretization should
go in the next major release.
#. Move any optional tasks that are not done to the next milestone.

#. Create a branch for the release, based on ``develop``:

.. code-block:: console
$ git checkout -b releases/v0.15 develop
$ git checkout -b releases/v0.23 develop
For a version ``vX.Y.Z``, the branch's name should be
``releases/vX.Y``. That is, you should create a ``releases/vX.Y``
Expand Down Expand Up @@ -856,8 +865,8 @@ completed, the steps to make the major release are:

Create a pull request targeting the ``develop`` branch, bumping the major
version in ``lib/spack/spack/__init__.py`` with a ``dev0`` release segment.
For instance when you have just released ``v0.15.0``, set the version
to ``(0, 16, 0, 'dev0')`` on ``develop``.
For instance when you have just released ``v0.23.0``, set the version
to ``(0, 24, 0, 'dev0')`` on ``develop``.

#. Follow the steps in :ref:`publishing-releases`.

Expand All @@ -866,82 +875,52 @@ completed, the steps to make the major release are:
#. Follow the steps in :ref:`announcing-releases`.


.. _point-releases:
.. _patch-releases:

^^^^^^^^^^^^^^^^^^^^^
Making point releases
Making patch releases
^^^^^^^^^^^^^^^^^^^^^

Assuming a project board has already been created and all required work
completed, the steps to make the point release are:

#. Create a new project board for the next point release.

#. Move any optional tasks that are not done to the next project board.

#. Check out the release branch (it should already exist).

For the ``X.Y.Z`` release, the release branch is called ``releases/vX.Y``.
For ``v0.15.1``, you would check out ``releases/v0.15``:

.. code-block:: console
$ git checkout releases/v0.15
#. If a pull request to the release branch named ``Backports vX.Y.Z`` is not already
in the project, create it. This pull request ought to be created as early as
possible when working on a release project, so that we can build the release
commits incrementally, and identify potential conflicts at an early stage.

#. Cherry-pick each pull request in the ``Done`` column of the release
project board onto the ``Backports vX.Y.Z`` pull request.

This is **usually** fairly simple since we squash the commits from the
vast majority of pull requests. That means there is only one commit
per pull request to cherry-pick. For example, `this pull request
<https://github.com/spack/spack/pull/15777>`_ has three commits, but
they were squashed into a single commit on merge. You can see the
commit that was created here:

.. image:: images/pr-commit.png

You can easily cherry pick it like this (assuming you already have the
release branch checked out):
To make the patch release process both efficient and transparent, we use a *backports pull request*
which contains cherry-picked commits from the ``develop`` branch. The majority of the work is to
cherry-pick the bug fixes, which ideally should be done as soon as they land on ``develop``:
this ensures cherry-picking happens in order, and makes conflicts easier to resolve since the
changes are fresh in the mind of the developer.

.. code-block:: console
The backports pull request is always titled ``Backports vX.Y.Z`` and is labelled ``backports``. It
is opened from a branch named ``backports/vX.Y.Z`` and targets the ``releases/vX.Y`` branch.

$ git cherry-pick 7e46da7
Whenever a pull request labelled ``vX.Y.Z`` is merged, cherry-pick the associated squashed commit
on ``develop`` to the ``backports/vX.Y.Z`` branch. For pull requests that were rebased (or not
squashed), cherry-pick each associated commit individually. Never force push to the
``backports/vX.Y.Z`` branch.

For pull requests that were rebased (or not squashed), you'll need to
cherry-pick each associated commit individually.
.. warning::

.. warning::
Sometimes you may **still** get merge conflicts even if you have
cherry-picked all the commits in order. This generally means there
is some other intervening pull request that the one you're trying
to pick depends on. In these cases, you'll need to make a judgment
call regarding those pull requests. Consider the number of affected
files and/or the resulting differences.

It is important to cherry-pick commits in the order they happened,
otherwise you can get conflicts while cherry-picking. When
cherry-picking look at the merge date,
**not** the number of the pull request or the date it was opened.
1. If the changes are small, you might just cherry-pick it.

Sometimes you may **still** get merge conflicts even if you have
cherry-picked all the commits in order. This generally means there
is some other intervening pull request that the one you're trying
to pick depends on. In these cases, you'll need to make a judgment
call regarding those pull requests. Consider the number of affected
files and or the resulting differences.
2. If the changes are large, then you may decide that this fix is not
worth including in a patch release, in which case you should remove
the label from the pull request. Remember that large, manual backports
are seldom the right choice for a patch release.

1. If the dependency changes are small, you might just cherry-pick it,
too. If you do this, add the task to the release board.
When all commits are cherry-picked in the ``backports/vX.Y.Z`` branch, make the patch
release as follows:

2. If the changes are large, then you may decide that this fix is not
worth including in a point release, in which case you should remove
the task from the release project.
#. `Create a new label <https://github.com/spack/spack/labels>`_ ``vX.Y.{Z+1}`` for the next patch
release.

3. You can always decide to manually back-port the fix to the release
branch if neither of the above options makes sense, but this can
require a lot of work. It's seldom the right choice.
#. Replace the label ``vX.Y.Z`` with ``vX.Y.{Z+1}`` for all PRs and issues that are not done.

#. When all the commits from the project board are cherry-picked into
the ``Backports vX.Y.Z`` pull request, you can push a commit to:
#. Manually push a single commit with commit message ``Set version to vX.Y.Z`` to the
``backports/vX.Y.Z`` branch, that both bumps the Spack version number and updates the changelog:

1. Bump the version in ``lib/spack/spack/__init__.py``.
2. Update ``CHANGELOG.md`` with a list of the changes.
Expand All @@ -950,20 +929,22 @@ completed, the steps to make the point release are:
release branch. See `the changelog from 0.14.1
<https://github.com/spack/spack/commit/ff0abb9838121522321df2a054d18e54b566b44a>`_.

#. Merge the ``Backports vX.Y.Z`` PR with the **Rebase and merge** strategy. This
is needed to keep track in the release branch of all the commits that were
cherry-picked.

#. Make sure CI passes on the release branch, including:
#. Make sure CI passes on the **backports pull request**, including:

* Regular unit tests
* Build tests
* The E4S pipeline at `gitlab.spack.io <https://gitlab.spack.io>`_

If CI does not pass, you'll need to figure out why, and make changes
to the release branch until it does. You can make more commits, modify
or remove cherry-picked commits, or cherry-pick **more** from
``develop`` to make this happen.
#. Merge the ``Backports vX.Y.Z`` PR with the **Rebase and merge** strategy. This
is needed to keep track in the release branch of all the commits that were
cherry-picked.

#. Make sure CI passes on the last commit of the **release branch**.

#. In the rare case you need to include additional commits in the patch release after the backports
PR is merged, it is best to delete the last commit ``Set version to vX.Y.Z`` from the release
branch with a single force push, open a new backports PR named ``Backports vX.Y.Z (2)``, and
repeat the process. Avoid repeated force pushes to the release branch.

#. Follow the steps in :ref:`publishing-releases`.

Expand Down Expand Up @@ -1038,25 +1019,31 @@ Updating `releases/latest`

If the new release is the **highest** Spack release yet, you should
also tag it as ``releases/latest``. For example, suppose the highest
release is currently ``0.15.3``:
release is currently ``0.22.3``:

* If you are releasing ``0.15.4`` or ``0.16.0``, then you should tag
it with ``releases/latest``, as these are higher than ``0.15.3``.
* If you are releasing ``0.22.4`` or ``0.23.0``, then you should tag
it with ``releases/latest``, as these are higher than ``0.22.3``.

* If you are making a new release of an **older** major version of
Spack, e.g. ``0.14.4``, then you should not tag it as
Spack, e.g. ``0.21.4``, then you should not tag it as
``releases/latest`` (as there are newer major versions).

To tag ``releases/latest``, do this:
To do so, first fetch the latest tag created on GitHub, since you may not have it locally:

.. code-block:: console
$ git fetch --force git@github.com:spack/spack vX.Y.Z
Then tag ``vX.Y.Z`` as ``releases/latest`` and push the individual tag to GitHub.

.. code-block:: console
$ git checkout releases/vX.Y # vX.Y is the new release's branch
$ git tag --force releases/latest
$ git push --force --tags
$ git tag --force releases/latest vX.Y.Z
$ git push --force git@github.com:spack/spack releases/latest
The ``--force`` argument to ``git tag`` makes ``git`` overwrite the existing
``releases/latest`` tag with the new one.
The ``--force`` argument to ``git tag`` makes ``git`` overwrite the existing ``releases/latest``
tag with the new one. Do **not** use the ``--tags`` flag when pushing, since this will push *all*
local tags.


.. _announcing-releases:
Expand Down
10 changes: 7 additions & 3 deletions lib/spack/docs/environments.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -425,9 +425,13 @@ Developing Packages in a Spack Environment

The ``spack develop`` command allows one to develop Spack packages in
an environment. It requires a spec containing a concrete version, and
will configure Spack to install the package from local source. By
default, it will also clone the package to a subdirectory in the
environment. This package will have a special variant ``dev_path``
will configure Spack to install the package from local source.
If a version is not provided from the command line interface then spack
will automatically pick the highest version the package has defined.
This means any infinity versions (``develop``, ``main``, ``stable``) will be
preferred in this selection process.
By default, ``spack develop`` will also clone the package to a subdirectory in the
environment for the local source. This package will have a special variant ``dev_path``
set, and Spack will ensure the package and its dependents are rebuilt
any time the environment is installed if the package's local source
code has been modified. Spack's native implementation to check for modifications
Expand Down
Binary file removed lib/spack/docs/images/pr-commit.png
View file
Binary file not shown.
Binary file removed lib/spack/docs/images/projects.png
View file
Binary file not shown.
Loading

0 comments on commit fc40bdd

Please sign in to comment.