Commits on Sep 13, 2024

1. feat: support Debian flat repos …

   Fixes issue GoogleContainerTools#56
   
   Follow-up and credit to @alexconrey (PR GoogleContainerTools#55), @ericlchen1 (PR GoogleContainerTools#64) and
   @benmccown (PR GoogleContainerTools#67) for their work on similar PRs that I've reviewed and
   drawn some inspiration to create "one 💍 PR to merge them all" 😅
   
   Problem:
   
   Debian has two types of repos: "canonical" and "flat". Each has a
   different sources.list syntax:
   
   "canonical":
   ```
   deb uri distribution [component1] [component2] [...]
   ```
   (see https://wiki.debian.org/DebianRepository/Format#Overview)
   
   flat:
   ```
   deb uri directory/
   ```
   (see https://wiki.debian.org/DebianRepository/Format#Flat_Repository_Format)
   
   A flat repository does not use the dists hierarchy of directories, and
   instead places meta index and indices directly into the archive root (or
   some part below it)
   
   Thus, the URL logic in _fetch_package_index() is incorrect for these
   repos and it always fails to fetch the Package index.
   
   Solution:
   
   Just use the Debian sources.list convention in the 'sources' section of
   the manifest to add canonical and flat repos. Depending on whether the
   channel has one directory that ends in '/' or a (dist, component, ...)
   structure the _fetch_package_index and other internal logic will
   know whether the source is a canonical or a flat repo.
   
   For example:
   ```
   version: 1
   
   sources:
     # canonical repo
     - channel: bullseye main contrib
       url: https://snapshot-cloudflare.debian.org/archive/debian/20240210T223313Z
     # flat repos, note the trailing '/' and the lack of distribution or components
     - channel: bullseye-cran40/
       url: https://cloud.r-project.org/bin/linux/debian
     - channel: ubuntu2404/x86_64/
       url: https://developer.download.nvidia.com/compute/cuda/repos
   
   archs:
     - amd64
   
   packages:
     - bash
     - r-mathlib
     - nvidia-container-toolkit-base
   ```
   
   Disregarding the "mixing" of Ubuntu and Debian repos for the purpose of
   the example, this manifest shows that you can mix canonical and flat
   repos and you can mix multiarch and single-arch flat repos and canonical
   repos.
   
   You will still have the same problems as before with packages that only
   exist for one architecture and/or repos that only support one
   architecture. In those cases, simply separate the repos and packages
   into their own manifests.
   
   NOTE:
   The NVIDIA CUDA repos don't follow Debian specs and have issues with the
   package filenames. This is addressed in a separate commit.

   

   jjmaestro committed Sep 13, 2024
   Configuration menu

   • View commit details
   • Copy full SHA for 885b28c
   • Browse repository at this point

   Copy the full SHA

   885b28c View commit details

   Browse the repository at this point in the history

2. fix: NVIDIA CUDA flat repos don't follow Debian repo spec …

   Although the Debian repo spec for 'Filename' (see
   https://wiki.debian.org/DebianRepository/Format#Filename) clearly says
   that 'Filename' should be relative to the base directory of the repo and
   should be in canonical form (i.e. without '.' or '..') there are cases
   where this is not honored.
   
   In those cases we try to work around this by assuming 'Filename' is
   relative to the sources.list directory/ so we combine them and normalize
   the new 'Filename' path.
   
   Note that, so far, only the NVIDIA CUDA repos needed this workaround so
   maybe this heuristic will break for other repos that don't conform to
   the Debian repo spec.

   

   jjmaestro committed Sep 13, 2024
   Configuration menu

   • View commit details
   • Copy full SHA for 8441ec3
   • Browse repository at this point

   Copy the full SHA

   8441ec3 View commit details

   Browse the repository at this point in the history