Skip to content

Commit

Permalink
doc: Mention the problem of conflicting asset names more explicitly
Browse files Browse the repository at this point in the history
* State the problem more explicitly, also considering concurrent uploads
* Move private assets (which are a solution to avoid the problem for jobs
  with chained dependencies) into their own section
    * Avoid mentioning private assets directly at the beginning to simplify
      the wording
* Avoid confusion with reference to variable expansion (it does *not*
  relate to private assets specifically)
* See https://progress.opensuse.org/issues/109319
  • Loading branch information
Martchus committed Apr 7, 2022
1 parent 10b0626 commit a6b4ca2
Showing 1 changed file with 37 additions and 21 deletions.
58 changes: 37 additions & 21 deletions docs/UsersGuide.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -1149,29 +1149,45 @@ uncompress it to `foo2.iso`, store it in `/var/lib/openqa/share/factory/iso` and
and uncompressed as.

=== Specifying assets created by a job ===
`HDD_n` assets can be 'stored' or 'published' by a job, and `UEFI_PFLASH_VARS` assets can be
'published'. These both mean that if the job completes successfully, the resulting state of those
disk assets will be sent back to the web-UI and made available as an `hdd`-type asset. To 'store'
an asset, you can specify e.g. `STORE_HDD_1`. To 'publish' it, you can specify e.g.
`PUBLISH_HDD_1` or `PUBLISH_PFLASH_VARS`. If you specify `PUBLISH_HDD_1=updated.qcow2`, the
`HDD_1` disk image as it exists at the end of the test will be uploaded back to the web-UI and
stored under the name `updated.qcow2`; any other job can then specify `HDD_1=updated.qcow2` to use
this published image as its `HDD_1`. To force publishing assets even in case
of a failed job one can try the `FORCE_PUBLISH_HDD_` variable.

The difference between 'storing' and 'publishing' is that when 'storing' an asset, it will be
altered in some way (currently, by prepending the job ID to the filename) to associate it with
the particular job that produced it. That means that many jobs can 'store' an asset under "the
same name" without conflicting. Of course, that would seem to make it hard for other jobs to use
the 'stored' image - but for "chained" jobs, the reverse operation is done transparently. This
all means that a 'parent' job template can specify `STORE_HDD_1=somename.qcow2` and its 'child'
job template(s) can specify `HDD_1=somename.qcow2`, and everything will work, without multiple
runs of the same jobs overwriting the asset. For more on "chained" jobs, see the documentation
of <<WritingTests.asciidoc#_job_dependencies,job dependencies>>.

When using this mechanism you will often also want to use the
Jobs can upload assets to the web-UI so other jobs can used them as `HDD_n` and
`UEFI_PFLASH_VARS` assets as described in the previous section.

To declare an asset to be uploaded, you can use the job settings `PUBLISH_HDD_n`
and `PUBLISH_PFLASH_VARS`. For instance, if you specify
`PUBLISH_HDD_1=updated.qcow2`, the `HDD_1` disk image as it exists at the end of
the test will be uploaded back to the web-UI and stored under the name
`updated.qcow2`. Any other job can then specify `HDD_1=updated.qcow2` to use
this published image as its `HDD_1`.

IMPORTANT: Assets that are already existing will be overridden. If the same
asset is uploaded by multiple jobs concurrently this will lead to file
corruption. So be sure to use unique names or use private assets as explained
in the subsection below.

NOTE: Note that assets are by default only uploaded if the job completes
successfully. To force publishing assets even in case of a failed job one can
specify the `FORCE_PUBLISH_HDD_` variable.

NOTE: When using this mechanism you will often also want to use the
<<UsersGuide.asciidoc#_variable_expansion,variable expansion>> mechanism.

==== Private assets ====
There is a mechanism to alter an asset's file name automatically to associate
it with the particular job that produced it (currently, by prepending the job ID
to the filename). To make use of it, use `STORE_HDD_n` (instead of
`PUBLISH_HDD_n`). Those assets can then be consumed by chained jobs. For
instance, if a parent job uploads an asset via `STORE_HDD_1=somename.qcow2`, its
children can use it via `HDD_1=somename.qcow2` without having to worry about
naming conflicts.

IMPORTANT: This only works if the jobs uploading and consuming jobs have a
chained dependency. For more on "chained" jobs, see the documentation of
<<WritingTests.asciidoc#_job_dependencies,job dependencies>>.

NOTE: Access to private assets is not protected. Theoretically, jobs outside the
chain can still access the asset by explicitly prepending the ID of the creating
job.

[id="asset_cleanup"]
=== Asset cleanup ===
For more information on assets, start reading from the beginning of the
Expand Down

0 comments on commit a6b4ca2

Please sign in to comment.