Fix issues with deb_index and Nvidia's apt repository #67

benmccown · 2024-07-18T16:21:57Z

I opened #66 detailing some of the problems we have run into when trying to use deb_index to install Nvidia's CUDA packages from their apt repository.

These fixes are my first pass at resolving issues that were present with the apt package implementation here in rules_distroless.

I did some research to try and find specification as to how Packages works with apt and apt-cache and whether there's a specific location that Packages should live, but I couldn't find anything detailing a correct implementation for a remote apt package repository. Based on this, it seems to me that Nvidia's package repository is formatted in a valid way and rules_distroless should ideally support it.

I'm fully expecting to need a code revision here, just wanted to share my first iteration to get feedback and see what maintainers want changed.

This reverts commit f26d909.

google-cla · 2024-07-18T16:22:03Z

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

benmccown

Left some comments explaining a few things.

benmccown · 2024-07-18T16:26:11Z

apt/private/index.bzl

@@ -85,17 +85,19 @@ def _deb_package_index_impl(rctx):
            ),
        )

+        deps = depset([


We had packages that shared duplicate transitive dependencies. The duplicate deps were a blocking issue, so I used a depset to dedupe packages.

Do you have an example the package(s) that broke? Were they packages in the NVIDIA CUDA repos?

@jjmaestro I had this in my sources:

- channel: nvidia cuda url: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64

And this in my packages:

# CUDA Dependencies - cuda-11-8 - cudnn9-cuda-11-8

I think those would replicate the issue.

@benmccown I'm checking the Packages index and there's no cuda-11-8 package, only cudnn9-cuda-11-8 :-/ I'll try to test with the latter and with other packages to see if I can repro it as well.

@benmccown OK, I looked more carefully and I was using the wrong Packages index / arch but that's also the problem for the repro, I can't get both of those packages for both amd64 and arm64 (they are only available for amd64 IIRC).

Have a look at jjmaestro/rules_distroless/tree/testing-cuda-packages, I have one commit there so far: jjmaestro@f216178, this is a test running on top of #97.

I've gotten it to work with just the cuda-11-8 package in arm64, and since that's the largest package by far, I think it's probably working but I'll change the example I'm working on to only amd64 and see if something breaks when adding both packages.

I've found yet-another-issue when testing with amd64... the failure I'm getting is because there's a Debian package that's already compressed as a tar.gz, see #98. I'll see if I can work around that issue and test things further but so far, I haven't seen duplicate transitive dependencies.

Alright, @benmccown please check jjmaestro/rules_distroless/tree/testing-cuda-packages.

I didn't get any errors / issues with duplicate transitive dependencies. I have it all working after porting #99 and also making sure I flatten the layers to avoid max depth exceeded (#36) and also, using pkg_tar to flatten, to avoid duplicates of file paths not supported Docker errors (e.g. bazelbuild/rules_docker#246).

Can you check that branch and confirm if that works for you?

Thanks!

benmccown · 2024-07-18T16:27:02Z

apt/private/lockfile.bzl

@@ -16,11 +16,14 @@ def _add_package(lock, package, arch):
    k = _package_key(package, arch)
    if k in lock.fast_package_lookup:
        return
+    filename = package["Filename"]
+    if filename.startswith("./"):


When Packages lives in the same directory/URL that the packages themselves live, the manifest has ./ as the leading file path for all the packages listed, however we don't want a ./ in the URL.

FYI, this is not what the spec says, check https://wiki.debian.org/DebianRepository/Format#Filename:

Filename

The mandatory Filename field shall list the path of the package archive relative to the base directory of the repository. The path should be in canonical form, that is, without any components denoting the current or parent directory ("." or ".."). It also should not make use of any protocol-specific components, such as URL-encoded parameters.

I've checked the R-Project flat repos and this doesn't happen / is not needed. However, this is something that happens in the NVIDIA CUDA flat repos . Thus, I think that this is a quirk of some specific flat repos that don't conform with the Debian repo spec.

Super interesting. I was looking for documentation on the manifest format for a debian repository and was having a hard time finding anything authoritative. Thanks for sharing the link.

benmccown · 2024-07-18T16:28:49Z

apt/private/package_index.bzl

+    # Split the rest of the URL by the '/' delimiter and take the first part
+    domain = rest.split("/")[0]
+
+    target_triple = "{domain}/{dist}/{comp}/{arch}".format(domain = domain, dist = dist, comp = comp, arch = arch)


We needed to add domain to the target_triple because dist/comp/arch was duplicated across repositories since Ubuntu and Nvidia's repositories were both focal/main/x86_64.

target_triple is probably no longer the right term to use. I can change it as you see fit.

benmccown · 2024-07-18T16:29:18Z

apt/private/package_index.bzl

-                break
+        urls = [
+            "{}/dists/{}/{}/binary-{}/Packages.{}".format(url, dist, comp, arch, file_type),
+            "{}/Packages.{}".format(url, file_type),


This line adds support for Nvidia's Packages location. We had to add gzip support as well since they didn't have an xz compressed archive.

benmccown · 2024-07-19T15:35:37Z

I'm fully expecting to need a code revision here, just wanted to share my first iteration to get feedback and see what maintainers want changed.

benmccown · 2024-07-19T15:36:52Z

@alexeagle @thesayyn Just tagging y'all as you look like recently active maintainers.

benmccown · 2024-08-21T16:39:21Z

@alexeagle @thesayyn bump. Any thoughts on what I'm trying to accomplish here? Nvidia's cuda packages (and corresponding apt repository) seem like common packages folks might need to be able to pull.

thesayyn · 2024-08-21T16:54:07Z

This is same as #64

@alexconrey

Fixes issue GoogleContainerTools#56 Follow-up and credit to @alexconrey (PR GoogleContainerTools#55), @ericlchen1 (PR GoogleContainerTools#64) and @benmccown (PR GoogleContainerTools#67) for their work on similar PRs that I've reviewed and drawn some inspiration to create "one 💍 PR to merge them all" 😅 Problem: Debian has two types of repos: "canonical" and "flat". Each has a different sources.list syntax: "canonical": ``` deb uri distribution [component1] [component2] [...] ``` (see https://wiki.debian.org/DebianRepository/Format#Overview) flat: ``` deb uri directory/ ``` (see https://wiki.debian.org/DebianRepository/Format#Flat_Repository_Format) A flat repository does not use the dists hierarchy of directories, and instead places meta index and indices directly into the archive root (or some part below it) Thus, the URL logic in _fetch_package_index() is incorrect for these repos and it always fails to fetch the Package index. Solution: Just use the Debian sources.list convention in the 'sources' section of the manifest to add canonical and flat repos. Depending on whether the channel has one directory that ends in '/' or a (dist, component, ...) structure the _fetch_package_index and other internal logic will know whether the source is a canonical or a flat repo. For example: ``` version: 1 sources: # canonical repo - channel: bullseye main contrib url: https://snapshot-cloudflare.debian.org/archive/debian/20240210T223313Z # flat repos, note the trailing '/' and the lack of distribution or components - channel: bullseye-cran40/ url: https://cloud.r-project.org/bin/linux/debian - channel: ubuntu2404/x86_64/ url: https://developer.download.nvidia.com/compute/cuda/repos archs: - amd64 packages: - bash - r-mathlib - nvidia-container-toolkit-base ``` Disregarding the "mixing" of Ubuntu and Debian repos for the purpose of the example, this manifest shows that you can mix canonical and flat repos and you can mix multiarch and single-arch flat repos and canonical repos. You will still have the same problems as before with packages that only exist for one architecture and/or repos that only support one architecture. In those cases, simply separate the repos and packages into their own manifests. NOTE: The NVIDIA CUDA repos don't follow Debian specs and have issues with the package filenames. This is addressed in a separate commit.

@alexconrey

Fixes issue GoogleContainerTools#56 Follow-up and credit to @alexconrey (PR GoogleContainerTools#55), @ericlchen1 (PR GoogleContainerTools#64) and @benmccown (PR GoogleContainerTools#67) for their work on similar PRs that I've reviewed and drawn some inspiration to create "one 💍 PR to merge them all" 😅 Problem: Debian has two types of repos: "canonical" and "flat". Each has a different sources.list syntax: "canonical": ``` deb uri distribution [component1] [component2] [...] ``` (see https://wiki.debian.org/DebianRepository/Format#Overview) flat: ``` deb uri directory/ ``` (see https://wiki.debian.org/DebianRepository/Format#Flat_Repository_Format) A flat repository does not use the dists hierarchy of directories, and instead places meta index and indices directly into the archive root (or some part below it) Thus, the URL logic in _fetch_package_index() is incorrect for these repos and it always fails to fetch the Package index. Solution: Just use the Debian sources.list convention in the 'sources' section of the manifest to add canonical and flat repos. Depending on whether the channel has one directory that ends in '/' or a (dist, component, ...) structure the _fetch_package_index and other internal logic will know whether the source is a canonical or a flat repo. For example: ``` version: 1 sources: # canonical repo - channel: bullseye main contrib url: https://snapshot-cloudflare.debian.org/archive/debian/20240210T223313Z # flat repos, note the trailing '/' and the lack of distribution or components - channel: bullseye-cran40/ url: https://cloud.r-project.org/bin/linux/debian - channel: ubuntu2404/x86_64/ url: https://developer.download.nvidia.com/compute/cuda/repos archs: - amd64 packages: - bash - r-mathlib - nvidia-container-toolkit-base ``` Disregarding the "mixing" of Ubuntu and Debian repos for the purpose of the example, this manifest shows that you can mix canonical and flat repos and you can mix multiarch and single-arch flat repos and canonical repos. You will still have the same problems as before with packages that only exist for one architecture and/or repos that only support one architecture. In those cases, simply separate the repos and packages into their own manifests. NOTE: The NVIDIA CUDA repos don't follow Debian specs and have issues with the package filenames. This is addressed in a separate commit.

jjmaestro · 2024-09-13T13:29:05Z

OK, so I finally did try to write a PR that adds support for multi-arch and single-arch flat repos and addresses the issue with the NVIDIA CUDA flat repos. Hopefully that PR superseeds the others and can be merged and 1 + 3 PRs closed 😄

@benmccown please have a look and let me know what you think and if #86 works for you. Happy to address any issues or concerns if there's something that breaks in your use-case!

@alexconrey

Fixes issue GoogleContainerTools#56 Follow-up and credit to @alexconrey (PR GoogleContainerTools#55), @ericlchen1 (PR GoogleContainerTools#64) and @benmccown (PR GoogleContainerTools#67) for their work on similar PRs that I've reviewed and drawn some inspiration to create "one 💍 PR to merge them all" 😅 Problem: Debian has two types of repos: "canonical" and "flat". Each has a different sources.list syntax: "canonical": ``` deb uri distribution [component1] [component2] [...] ``` (see https://wiki.debian.org/DebianRepository/Format#Overview) flat: ``` deb uri directory/ ``` (see https://wiki.debian.org/DebianRepository/Format#Flat_Repository_Format) A flat repository does not use the dists hierarchy of directories, and instead places meta index and indices directly into the archive root (or some part below it) Thus, the URL logic in _fetch_package_index() is incorrect for these repos and it always fails to fetch the Package index. Solution: Just use the Debian sources.list convention in the 'sources' section of the manifest to add canonical and flat repos. Depending on whether the channel has one directory that ends in '/' or a (dist, component, ...) structure the _fetch_package_index and other internal logic will know whether the source is a canonical or a flat repo. For example: ``` version: 1 sources: # canonical repo - channel: bullseye main contrib url: https://snapshot-cloudflare.debian.org/archive/debian/20240210T223313Z # flat repos, note the trailing '/' and the lack of distribution or components - channel: bullseye-cran40/ url: https://cloud.r-project.org/bin/linux/debian - channel: ubuntu2404/x86_64/ url: https://developer.download.nvidia.com/compute/cuda/repos archs: - amd64 packages: - bash - r-mathlib - nvidia-container-toolkit-base ``` Disregarding the "mixing" of Ubuntu and Debian repos for the purpose of the example, this manifest shows that you can mix canonical and flat repos and you can mix multiarch and single-arch flat repos and canonical repos. You will still have the same problems as before with packages that only exist for one architecture and/or repos that only support one architecture. In those cases, simply separate the repos and packages into their own manifests. NOTE: The NVIDIA CUDA repos don't follow Debian specs and have issues with the package filenames. This is addressed in a separate commit.

@alexconrey

Fixes issue GoogleContainerTools#56 Follow-up and credit to @alexconrey (PR GoogleContainerTools#55), @ericlchen1 (PR GoogleContainerTools#64) and @benmccown (PR GoogleContainerTools#67) for their work on similar PRs that I've reviewed and drawn some inspiration to create "one 💍 PR to merge them all" 😅 Problem: Debian has two types of repos: "canonical" and "flat". Each has a different sources.list syntax: "canonical": ``` deb uri distribution [component1] [component2] [...] ``` (see https://wiki.debian.org/DebianRepository/Format#Overview) flat: ``` deb uri directory/ ``` (see https://wiki.debian.org/DebianRepository/Format#Flat_Repository_Format) A flat repository does not use the dists hierarchy of directories, and instead places meta index and indices directly into the archive root (or some part below it) Thus, the URL logic in _fetch_package_index() is incorrect for these repos and it always fails to fetch the Package index. Solution: Just use the Debian sources.list convention in the 'sources' section of the manifest to add canonical and flat repos. Depending on whether the channel has one directory that ends in '/' or a (dist, component, ...) structure the _fetch_package_index and other internal logic will know whether the source is a canonical or a flat repo. For example: ``` version: 1 sources: # canonical repo - channel: bullseye main contrib url: https://snapshot-cloudflare.debian.org/archive/debian/20240210T223313Z # flat repos, note the trailing '/' and the lack of distribution or components - channel: bullseye-cran40/ url: https://cloud.r-project.org/bin/linux/debian - channel: ubuntu2404/x86_64/ url: https://developer.download.nvidia.com/compute/cuda/repos archs: - amd64 packages: - bash - r-mathlib - nvidia-container-toolkit-base ``` Disregarding the "mixing" of Ubuntu and Debian repos for the purpose of the example, this manifest shows that you can mix canonical and flat repos and you can mix multiarch and single-arch flat repos and canonical repos. You will still have the same problems as before with packages that only exist for one architecture and/or repos that only support one architecture. In those cases, simply separate the repos and packages into their own manifests. NOTE: The NVIDIA CUDA repos don't follow Debian specs and have issues with the package filenames. This is addressed in a separate commit.

@alexconrey

Fixes issue GoogleContainerTools#56 Follow-up and credit to @alexconrey (PR GoogleContainerTools#55), @ericlchen1 (PR GoogleContainerTools#64) and @benmccown (PR GoogleContainerTools#67) for their work on similar PRs that I've reviewed and drawn some inspiration to create "one 💍 PR to merge them all" 😅 Problem: Debian has two types of repos: "canonical" and "flat". Each has a different sources.list syntax: "canonical": ``` deb uri distribution [component1] [component2] [...] ``` (see https://wiki.debian.org/DebianRepository/Format#Overview) flat: ``` deb uri directory/ ``` (see https://wiki.debian.org/DebianRepository/Format#Flat_Repository_Format) A flat repository does not use the dists hierarchy of directories, and instead places meta index and indices directly into the archive root (or some part below it) Thus, the URL logic in _fetch_package_index() is incorrect for these repos and it always fails to fetch the Package index. Solution: Just use the Debian sources.list convention in the 'sources' section of the manifest to add canonical and flat repos. Depending on whether the channel has one directory that ends in '/' or a (dist, component, ...) structure the _fetch_package_index and other internal logic will know whether the source is a canonical or a flat repo. For example: ``` version: 1 sources: # canonical repo - channel: bullseye main contrib url: https://snapshot-cloudflare.debian.org/archive/debian/20240210T223313Z # flat repos, note the trailing '/' and the lack of distribution or components - channel: bullseye-cran40/ url: https://cloud.r-project.org/bin/linux/debian - channel: ubuntu2404/x86_64/ url: https://developer.download.nvidia.com/compute/cuda/repos archs: - amd64 packages: - bash - r-mathlib - nvidia-container-toolkit-base ``` Disregarding the "mixing" of Ubuntu and Debian repos for the purpose of the example, this manifest shows that you can mix canonical and flat repos and you can mix multiarch and single-arch flat repos and canonical repos. You will still have the same problems as before with packages that only exist for one architecture and/or repos that only support one architecture. In those cases, simply separate the repos and packages into their own manifests. NOTE: The NVIDIA CUDA repos don't follow Debian specs and have issues with the package filenames. This is addressed in a separate commit.

@alexconrey

Fixes issue GoogleContainerTools#56 Follow-up and credit to @alexconrey (PR GoogleContainerTools#55), @ericlchen1 (PR GoogleContainerTools#64) and @benmccown (PR GoogleContainerTools#67) for their work on similar PRs that I've reviewed and drawn some inspiration to create "one 💍 PR to merge them all" 😅 Problem: Debian has two types of repos: "canonical" and "flat". Each has a different sources.list syntax: "canonical": ``` deb uri distribution [component1] [component2] [...] ``` (see https://wiki.debian.org/DebianRepository/Format#Overview) flat: ``` deb uri directory/ ``` (see https://wiki.debian.org/DebianRepository/Format#Flat_Repository_Format) A flat repository does not use the dists hierarchy of directories, and instead places meta index and indices directly into the archive root (or some part below it) Thus, the URL logic in _fetch_package_index() is incorrect for these repos and it always fails to fetch the Package index. Solution: Just use the Debian sources.list convention in the 'sources' section of the manifest to add canonical and flat repos. Depending on whether the channel has one directory that ends in '/' or a (dist, component, ...) structure the _fetch_package_index and other internal logic will know whether the source is a canonical or a flat repo. For example: ``` version: 1 sources: # canonical repo - channel: bullseye main contrib url: https://snapshot-cloudflare.debian.org/archive/debian/20240210T223313Z # flat repos, note the trailing '/' and the lack of distribution or components - channel: bullseye-cran40/ url: https://cloud.r-project.org/bin/linux/debian - channel: ubuntu2404/x86_64/ url: https://developer.download.nvidia.com/compute/cuda/repos archs: - amd64 packages: - bash - r-mathlib - nvidia-container-toolkit-base ``` Disregarding the "mixing" of Ubuntu and Debian repos for the purpose of the example, this manifest shows that you can mix canonical and flat repos and you can mix multiarch and single-arch flat repos and canonical repos. You will still have the same problems as before with packages that only exist for one architecture and/or repos that only support one architecture. In those cases, simply separate the repos and packages into their own manifests. NOTE: The NVIDIA CUDA repos don't follow Debian specs and have issues with the package filenames. This is addressed in a separate commit.

See GoogleContainerTools#67 (comment)

@alexconrey

Fixes issue GoogleContainerTools#56 Follow-up and credit to @alexconrey (PR GoogleContainerTools#55), @ericlchen1 (PR GoogleContainerTools#64) and @benmccown (PR GoogleContainerTools#67) for their work on similar PRs that I've reviewed and drawn some inspiration to create "one 💍 PR to merge them all" 😅 Problem: Debian has two types of repos: "canonical" and "flat". Each has a different sources.list syntax: "canonical": ``` deb uri distribution [component1] [component2] [...] ``` (see https://wiki.debian.org/DebianRepository/Format#Overview) flat: ``` deb uri directory/ ``` (see https://wiki.debian.org/DebianRepository/Format#Flat_Repository_Format) A flat repository does not use the dists hierarchy of directories, and instead places meta index and indices directly into the archive root (or some part below it) Thus, the URL logic in _fetch_package_index() is incorrect for these repos and it always fails to fetch the Package index. Solution: Just use the Debian sources.list convention in the 'sources' section of the manifest to add canonical and flat repos. Depending on whether the channel has one directory that ends in '/' or a (dist, component, ...) structure the _fetch_package_index and other internal logic will know whether the source is a canonical or a flat repo. For example: ``` version: 1 sources: # canonical repo - channel: bullseye main contrib url: https://snapshot-cloudflare.debian.org/archive/debian/20240210T223313Z # flat repos, note the trailing '/' and the lack of distribution or components - channel: bullseye-cran40/ url: https://cloud.r-project.org/bin/linux/debian - channel: ubuntu2404/x86_64/ url: https://developer.download.nvidia.com/compute/cuda/repos archs: - amd64 packages: - bash - r-mathlib - nvidia-container-toolkit-base ``` Disregarding the "mixing" of Ubuntu and Debian repos for the purpose of the example, this manifest shows that you can mix canonical and flat repos and you can mix multiarch and single-arch flat repos and canonical repos. You will still have the same problems as before with packages that only exist for one architecture and/or repos that only support one architecture. In those cases, simply separate the repos and packages into their own manifests. NOTE: The NVIDIA CUDA repos don't follow Debian specs and have issues with the package filenames. This is addressed in a separate commit.

@alexconrey

Fixes issue GoogleContainerTools#56 Follow-up and credit to @alexconrey (PR GoogleContainerTools#55), @ericlchen1 (PR GoogleContainerTools#64) and @benmccown (PR GoogleContainerTools#67) for their work on similar PRs that I've reviewed and drawn some inspiration to create "one 💍 PR to merge them all" 😅 Problem: Debian has two types of repos: "canonical" and "flat". Each has a different sources.list syntax: "canonical": ``` deb uri distribution [component1] [component2] [...] ``` (see https://wiki.debian.org/DebianRepository/Format#Overview) flat: ``` deb uri directory/ ``` (see https://wiki.debian.org/DebianRepository/Format#Flat_Repository_Format) A flat repository does not use the dists hierarchy of directories, and instead places meta index and indices directly into the archive root (or some part below it) Thus, the URL logic in _fetch_package_index() is incorrect for these repos and it always fails to fetch the Package index. Solution: Just use the Debian sources.list convention in the 'sources' section of the manifest to add canonical and flat repos. Depending on whether the channel has one directory that ends in '/' or a (dist, component, ...) structure the _fetch_package_index and other internal logic will know whether the source is a canonical or a flat repo. For example: ``` version: 1 sources: # canonical repo - channel: bullseye main contrib url: https://snapshot-cloudflare.debian.org/archive/debian/20240210T223313Z # flat repos, note the trailing '/' and the lack of distribution or components - channel: bullseye-cran40/ url: https://cloud.r-project.org/bin/linux/debian - channel: ubuntu2404/x86_64/ url: https://developer.download.nvidia.com/compute/cuda/repos archs: - amd64 packages: - bash - r-mathlib - nvidia-container-toolkit-base ``` Disregarding the "mixing" of Ubuntu and Debian repos for the purpose of the example, this manifest shows that you can mix canonical and flat repos and you can mix multiarch and single-arch flat repos and canonical repos. You will still have the same problems as before with packages that only exist for one architecture and/or repos that only support one architecture. In those cases, simply separate the repos and packages into their own manifests. NOTE: The NVIDIA CUDA repos don't follow Debian specs and have issues with the package filenames. This is addressed in a separate commit.

See GoogleContainerTools#67 (comment)

@alexconrey

Fixes issue GoogleContainerTools#56 Follow-up and credit to @alexconrey (PR GoogleContainerTools#55), @ericlchen1 (PR GoogleContainerTools#64) and @benmccown (PR GoogleContainerTools#67) for their work on similar PRs that I've reviewed and drawn some inspiration to create "one 💍 PR to merge them all" 😅 Problem: Debian has two types of repos: "canonical" and "flat". Each has a different sources.list syntax: "canonical": ``` deb uri distribution [component1] [component2] [...] ``` (see https://wiki.debian.org/DebianRepository/Format#Overview) flat: ``` deb uri directory/ ``` (see https://wiki.debian.org/DebianRepository/Format#Flat_Repository_Format) A flat repository does not use the dists hierarchy of directories, and instead places meta index and indices directly into the archive root (or some part below it) Thus, the URL logic in _fetch_package_index() is incorrect for these repos and it always fails to fetch the Package index. Solution: Just use the Debian sources.list convention in the 'sources' section of the manifest to add canonical and flat repos. Depending on whether the channel has one directory that ends in '/' or a (dist, component, ...) structure the _fetch_package_index and other internal logic will know whether the source is a canonical or a flat repo. For example: ``` version: 1 sources: # canonical repo - channel: bullseye main contrib url: https://snapshot-cloudflare.debian.org/archive/debian/20240210T223313Z # flat repos, note the trailing '/' and the lack of distribution or components - channel: bullseye-cran40/ url: https://cloud.r-project.org/bin/linux/debian - channel: ubuntu2404/x86_64/ url: https://developer.download.nvidia.com/compute/cuda/repos archs: - amd64 packages: - bash - r-mathlib - nvidia-container-toolkit-base ``` Disregarding the "mixing" of Ubuntu and Debian repos for the purpose of the example, this manifest shows that you can mix canonical and flat repos and you can mix multiarch and single-arch flat repos and canonical repos. You will still have the same problems as before with packages that only exist for one architecture and/or repos that only support one architecture. In those cases, simply separate the repos and packages into their own manifests. NOTE: The NVIDIA CUDA repos don't follow Debian specs and have issues with the package filenames. This is addressed in a separate commit.

@alexconrey

Fixes issue GoogleContainerTools#56 Follow-up and credit to @alexconrey (PR GoogleContainerTools#55), @ericlchen1 (PR GoogleContainerTools#64) and @benmccown (PR GoogleContainerTools#67) for their work on similar PRs that I've reviewed and drawn some inspiration to create "one 💍 PR to merge them all" 😅 Problem: Debian has two types of repos: "canonical" and "flat". Each has a different sources.list syntax: "canonical": ``` deb uri distribution [component1] [component2] [...] ``` (see https://wiki.debian.org/DebianRepository/Format#Overview) flat: ``` deb uri directory/ ``` (see https://wiki.debian.org/DebianRepository/Format#Flat_Repository_Format) A flat repository does not use the dists hierarchy of directories, and instead places meta index and indices directly into the archive root (or some part below it) Thus, the URL logic in _fetch_package_index() is incorrect for these repos and it always fails to fetch the Package index. Solution: Just use the Debian sources.list convention in the 'sources' section of the manifest to add canonical and flat repos. Depending on whether the channel has one directory that ends in '/' or a (dist, component, ...) structure the _fetch_package_index and other internal logic will know whether the source is a canonical or a flat repo. For example: ``` version: 1 sources: # canonical repo - channel: bullseye main contrib url: https://snapshot-cloudflare.debian.org/archive/debian/20240210T223313Z # flat repos, note the trailing '/' and the lack of distribution or components - channel: bullseye-cran40/ url: https://cloud.r-project.org/bin/linux/debian - channel: ubuntu2404/x86_64/ url: https://developer.download.nvidia.com/compute/cuda/repos archs: - amd64 packages: - bash - r-mathlib - nvidia-container-toolkit-base ``` Disregarding the "mixing" of Ubuntu and Debian repos for the purpose of the example, this manifest shows that you can mix canonical and flat repos and you can mix multiarch and single-arch flat repos and canonical repos. You will still have the same problems as before with packages that only exist for one architecture and/or repos that only support one architecture. In those cases, simply separate the repos and packages into their own manifests. NOTE: The NVIDIA CUDA repos don't follow Debian specs and have issues with the package filenames. This is addressed in a separate commit.

See GoogleContainerTools#67 (comment)

@alexconrey

Fixes issue GoogleContainerTools#56 Follow-up and credit to @alexconrey (PR GoogleContainerTools#55), @ericlchen1 (PR GoogleContainerTools#64) and @benmccown (PR GoogleContainerTools#67) for their work on similar PRs that I've reviewed and drawn some inspiration to create "one 💍 PR to merge them all" 😅 Problem: Debian has two types of repos: "canonical" and "flat". Each has a different sources.list syntax: "canonical": ``` deb uri distribution [component1] [component2] [...] ``` (see https://wiki.debian.org/DebianRepository/Format#Overview) flat: ``` deb uri directory/ ``` (see https://wiki.debian.org/DebianRepository/Format#Flat_Repository_Format) A flat repository does not use the dists hierarchy of directories, and instead places meta index and indices directly into the archive root (or some part below it) Thus, the URL logic in _fetch_package_index() is incorrect for these repos and it always fails to fetch the Package index. Solution: Just use the Debian sources.list convention in the 'sources' section of the manifest to add canonical and flat repos. Depending on whether the channel has one directory that ends in '/' or a (dist, component, ...) structure the _fetch_package_index and other internal logic will know whether the source is a canonical or a flat repo. For example: ``` version: 1 sources: # canonical repo - channel: bullseye main contrib url: https://snapshot-cloudflare.debian.org/archive/debian/20240210T223313Z # flat repos, note the trailing '/' and the lack of distribution or components - channel: bullseye-cran40/ url: https://cloud.r-project.org/bin/linux/debian - channel: ubuntu2404/x86_64/ url: https://developer.download.nvidia.com/compute/cuda/repos archs: - amd64 packages: - bash - r-mathlib - nvidia-container-toolkit-base ``` Disregarding the "mixing" of Ubuntu and Debian repos for the purpose of the example, this manifest shows that you can mix canonical and flat repos and you can mix multiarch and single-arch flat repos and canonical repos. You will still have the same problems as before with packages that only exist for one architecture and/or repos that only support one architecture. In those cases, simply separate the repos and packages into their own manifests. NOTE: The NVIDIA CUDA repos don't follow Debian specs and have issues with the package filenames. This is addressed in a separate commit.

See GoogleContainerTools#67 (comment)

@alexconrey

Fixes issue GoogleContainerTools#56 Follow-up and credit to @alexconrey (PR GoogleContainerTools#55), @ericlchen1 (PR GoogleContainerTools#64) and @benmccown (PR GoogleContainerTools#67) for their work on similar PRs that I've reviewed and drawn some inspiration to create "one 💍 PR to merge them all" 😅 Problem: Debian has two types of repos: "canonical" and "flat". Each has a different sources.list syntax: "canonical": ``` deb uri distribution [component1] [component2] [...] ``` (see https://wiki.debian.org/DebianRepository/Format#Overview) flat: ``` deb uri directory/ ``` (see https://wiki.debian.org/DebianRepository/Format#Flat_Repository_Format) A flat repository does not use the dists hierarchy of directories, and instead places meta index and indices directly into the archive root (or some part below it) Thus, the URL logic in _fetch_package_index() is incorrect for these repos and it always fails to fetch the Package index. Solution: Just use the Debian sources.list convention in the 'sources' section of the manifest to add canonical and flat repos. Depending on whether the channel has one directory that ends in '/' or a (dist, component, ...) structure the _fetch_package_index and other internal logic will know whether the source is a canonical or a flat repo. For example: ``` version: 1 sources: # canonical repo - channel: bullseye main contrib url: https://snapshot-cloudflare.debian.org/archive/debian/20240210T223313Z # flat repos, note the trailing '/' and the lack of distribution or components - channel: bullseye-cran40/ url: https://cloud.r-project.org/bin/linux/debian - channel: ubuntu2404/x86_64/ url: https://developer.download.nvidia.com/compute/cuda/repos archs: - amd64 packages: - bash - r-mathlib - nvidia-container-toolkit-base ``` Disregarding the "mixing" of Ubuntu and Debian repos for the purpose of the example, this manifest shows that you can mix canonical and flat repos and you can mix multiarch and single-arch flat repos and canonical repos. You will still have the same problems as before with packages that only exist for one architecture and/or repos that only support one architecture. In those cases, simply separate the repos and packages into their own manifests. NOTE: The NVIDIA CUDA repos don't follow Debian specs and have issues with the package filenames. This is addressed in a separate commit.

See GoogleContainerTools#67 (comment)

@alexconrey

Fixes issue GoogleContainerTools#56 Follow-up and credit to @alexconrey (PR GoogleContainerTools#55), @ericlchen1 (PR GoogleContainerTools#64) and @benmccown (PR GoogleContainerTools#67) for their work on similar PRs that I've reviewed and drawn some inspiration to create "one 💍 PR to merge them all" 😅 Problem: Debian has two types of repos: "canonical" and "flat". Each has a different sources.list syntax: "canonical": ``` deb uri distribution [component1] [component2] [...] ``` (see https://wiki.debian.org/DebianRepository/Format#Overview) flat: ``` deb uri directory/ ``` (see https://wiki.debian.org/DebianRepository/Format#Flat_Repository_Format) A flat repository does not use the dists hierarchy of directories, and instead places meta index and indices directly into the archive root (or some part below it) Thus, the URL logic in _fetch_package_index() is incorrect for these repos and it always fails to fetch the Package index. Solution: Just use the Debian sources.list convention in the 'sources' section of the manifest to add canonical and flat repos. Depending on whether the channel has one directory that ends in '/' or a (dist, component, ...) structure the _fetch_package_index and other internal logic will know whether the source is a canonical or a flat repo. For example: ``` version: 1 sources: # canonical repo - channel: bullseye main contrib url: https://snapshot-cloudflare.debian.org/archive/debian/20240210T223313Z # flat repos, note the trailing '/' and the lack of distribution or components - channel: bullseye-cran40/ url: https://cloud.r-project.org/bin/linux/debian - channel: ubuntu2404/x86_64/ url: https://developer.download.nvidia.com/compute/cuda/repos archs: - amd64 packages: - bash - r-mathlib - nvidia-container-toolkit-base ``` Disregarding the "mixing" of Ubuntu and Debian repos for the purpose of the example, this manifest shows that you can mix canonical and flat repos and you can mix multiarch and single-arch flat repos and canonical repos. You will still have the same problems as before with packages that only exist for one architecture and/or repos that only support one architecture. In those cases, simply separate the repos and packages into their own manifests. NOTE: The NVIDIA CUDA repos don't follow Debian specs and have issues with the package filenames. This is addressed in a separate commit.

See GoogleContainerTools#67 (comment)

@alexconrey

Fixes issue GoogleContainerTools#56 Follow-up and credit to @alexconrey (PR GoogleContainerTools#55), @ericlchen1 (PR GoogleContainerTools#64) and @benmccown (PR GoogleContainerTools#67) for their work on similar PRs that I've reviewed and drawn some inspiration to create "one 💍 PR to merge them all" 😅 Problem: Debian has two types of repos: "canonical" and "flat". Each has a different sources.list syntax: "canonical": ``` deb uri distribution [component1] [component2] [...] ``` (see https://wiki.debian.org/DebianRepository/Format#Overview) flat: ``` deb uri directory/ ``` (see https://wiki.debian.org/DebianRepository/Format#Flat_Repository_Format) A flat repository does not use the dists hierarchy of directories, and instead places meta index and indices directly into the archive root (or some part below it) Thus, the URL logic in _fetch_package_index() is incorrect for these repos and it always fails to fetch the Package index. Solution: Just use the Debian sources.list convention in the 'sources' section of the manifest to add canonical and flat repos. Depending on whether the channel has one directory that ends in '/' or a (dist, component, ...) structure the _fetch_package_index and other internal logic will know whether the source is a canonical or a flat repo. For example: ``` version: 1 sources: # canonical repo - channel: bullseye main contrib url: https://snapshot-cloudflare.debian.org/archive/debian/20240210T223313Z # flat repos, note the trailing '/' and the lack of distribution or components - channel: bullseye-cran40/ url: https://cloud.r-project.org/bin/linux/debian - channel: ubuntu2404/x86_64/ url: https://developer.download.nvidia.com/compute/cuda/repos archs: - amd64 packages: - bash - r-mathlib - nvidia-container-toolkit-base ``` Disregarding the "mixing" of Ubuntu and Debian repos for the purpose of the example, this manifest shows that you can mix canonical and flat repos and you can mix multiarch and single-arch flat repos and canonical repos. You will still have the same problems as before with packages that only exist for one architecture and/or repos that only support one architecture. In those cases, simply separate the repos and packages into their own manifests. NOTE: The NVIDIA CUDA repos don't follow Debian specs and have issues with the package filenames. This is addressed in a separate commit.

See GoogleContainerTools#67 (comment)

@alexconrey

Fixes issue GoogleContainerTools#56 Follow-up and credit to @alexconrey (PR GoogleContainerTools#55), @ericlchen1 (PR GoogleContainerTools#64) and @benmccown (PR GoogleContainerTools#67) for their work on similar PRs that I've reviewed and drawn some inspiration to create "one 💍 PR to merge them all" 😅 Problem: Debian has two types of repos: "canonical" and "flat". Each has a different sources.list syntax: "canonical": ``` deb uri distribution [component1] [component2] [...] ``` (see https://wiki.debian.org/DebianRepository/Format#Overview) flat: ``` deb uri directory/ ``` (see https://wiki.debian.org/DebianRepository/Format#Flat_Repository_Format) A flat repository does not use the dists hierarchy of directories, and instead places meta index and indices directly into the archive root (or some part below it) Thus, the URL logic in _fetch_package_index() is incorrect for these repos and it always fails to fetch the Package index. Solution: Just use the Debian sources.list convention in the 'sources' section of the manifest to add canonical and flat repos. Depending on whether the channel has one directory that ends in '/' or a (dist, component, ...) structure the _fetch_package_index and other internal logic will know whether the source is a canonical or a flat repo. For example: ``` version: 1 sources: # canonical repo - channel: bullseye main contrib url: https://snapshot-cloudflare.debian.org/archive/debian/20240210T223313Z # flat repos, note the trailing '/' and the lack of distribution or components - channel: bullseye-cran40/ url: https://cloud.r-project.org/bin/linux/debian - channel: ubuntu2404/x86_64/ url: https://developer.download.nvidia.com/compute/cuda/repos archs: - amd64 packages: - bash - r-mathlib - nvidia-container-toolkit-base ``` Disregarding the "mixing" of Ubuntu and Debian repos for the purpose of the example, this manifest shows that you can mix canonical and flat repos and you can mix multiarch and single-arch flat repos and canonical repos. You will still have the same problems as before with packages that only exist for one architecture and/or repos that only support one architecture. In those cases, simply separate the repos and packages into their own manifests. NOTE: The NVIDIA CUDA repos don't follow Debian specs and have issues with the package filenames. This is addressed in a separate commit.

@alexconrey

Fixes issue GoogleContainerTools#56 Follow-up and credit to @alexconrey (PR GoogleContainerTools#55), @ericlchen1 (PR GoogleContainerTools#64) and @benmccown (PR GoogleContainerTools#67) for their work on similar PRs that I've reviewed and drawn some inspiration to create "one 💍 PR to merge them all" 😅 Problem: Debian has two types of repos: "canonical" and "flat". Each has a different sources.list syntax: "canonical": ``` deb uri distribution [component1] [component2] [...] ``` (see https://wiki.debian.org/DebianRepository/Format#Overview) flat: ``` deb uri directory/ ``` (see https://wiki.debian.org/DebianRepository/Format#Flat_Repository_Format) A flat repository does not use the dists hierarchy of directories, and instead places meta index and indices directly into the archive root (or some part below it) Thus, the URL logic in _fetch_package_index() is incorrect for these repos and it always fails to fetch the Package index. Solution: Just use the Debian sources.list convention in the 'sources' section of the manifest to add canonical and flat repos. Depending on whether the channel has one directory that ends in '/' or a (dist, component, ...) structure the _fetch_package_index and other internal logic will know whether the source is a canonical or a flat repo. For example: ``` version: 1 sources: # canonical repo - channel: bullseye main contrib url: https://snapshot-cloudflare.debian.org/archive/debian/20240210T223313Z # flat repos, note the trailing '/' and the lack of distribution or components - channel: bullseye-cran40/ url: https://cloud.r-project.org/bin/linux/debian - channel: ubuntu2404/x86_64/ url: https://developer.download.nvidia.com/compute/cuda/repos archs: - amd64 packages: - bash - r-mathlib - nvidia-container-toolkit-base ``` Disregarding the "mixing" of Ubuntu and Debian repos for the purpose of the example, this manifest shows that you can mix canonical and flat repos and you can mix multiarch and single-arch flat repos and canonical repos. You will still have the same problems as before with packages that only exist for one architecture and/or repos that only support one architecture. In those cases, simply separate the repos and packages into their own manifests. NOTE: The NVIDIA CUDA repos don't follow Debian specs and have issues with the package filenames. This is addressed in a separate commit.

benmccown added 4 commits May 29, 2024 16:40

add support for third party repos

1cce1b1

debugging

f26d909

add domain to target_triple

d7a7c16

Revert "debugging"

fcd13a7

This reverts commit f26d909.

Merge branch 'main' into main

ff05345

benmccown commented Jul 18, 2024

View reviewed changes

This was referenced Sep 11, 2024

Support Flat Repository Format #55

Open

Add flat repository support for debian images #64

Open

jjmaestro mentioned this pull request Sep 13, 2024

feat: support flat repos [OLD, see #90] #86

Closed

jjmaestro mentioned this pull request Sep 19, 2024

feat: support flat repos #90

Closed

jjmaestro mentioned this pull request Sep 19, 2024

feat: support flat repos #97

Open

jjmaestro added a commit to jjmaestro/rules_distroless that referenced this pull request Sep 19, 2024

testing CUDA packages

f216178

See GoogleContainerTools#67 (comment)

jjmaestro added a commit to jjmaestro/rules_distroless that referenced this pull request Sep 20, 2024

testing CUDA packages

f9bd064

See GoogleContainerTools#67 (comment)

jjmaestro added a commit to jjmaestro/rules_distroless that referenced this pull request Sep 22, 2024

testing CUDA packages

7e713cd

See GoogleContainerTools#67 (comment)

jjmaestro added a commit to jjmaestro/rules_distroless that referenced this pull request Oct 28, 2024

testing CUDA packages

1ca19a3

See GoogleContainerTools#67 (comment)

jjmaestro added a commit to jjmaestro/rules_distroless that referenced this pull request Oct 29, 2024

testing CUDA packages

3fcf749

See GoogleContainerTools#67 (comment)

jjmaestro added a commit to jjmaestro/rules_distroless that referenced this pull request Oct 29, 2024

testing CUDA packages

8d16690

See GoogleContainerTools#67 (comment)

benmccown closed this Nov 13, 2024

jjmaestro added a commit to jjmaestro/rules_distroless that referenced this pull request Nov 14, 2024

testing CUDA packages

f3998c0

See GoogleContainerTools#67 (comment)

jjmaestro added a commit to jjmaestro/rules_distroless that referenced this pull request Nov 14, 2024

testing CUDA packages

f5e1646

See GoogleContainerTools#67 (comment)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix issues with deb_index and Nvidia's apt repository #67

Fix issues with deb_index and Nvidia's apt repository #67

benmccown commented Jul 18, 2024 •

edited

Loading

google-cla bot commented Jul 18, 2024

benmccown left a comment

benmccown Jul 18, 2024

jjmaestro Sep 12, 2024

benmccown Sep 17, 2024

jjmaestro Sep 19, 2024

jjmaestro Sep 19, 2024 •

edited

Loading

jjmaestro Sep 19, 2024

jjmaestro Sep 20, 2024

benmccown Jul 18, 2024

jjmaestro Sep 12, 2024 •

edited

Loading

benmccown Sep 17, 2024

benmccown Jul 18, 2024

benmccown Jul 18, 2024

benmccown commented Jul 19, 2024

benmccown commented Jul 19, 2024

benmccown commented Aug 21, 2024

thesayyn commented Aug 21, 2024

jjmaestro commented Sep 13, 2024

Fix issues with deb_index and Nvidia's apt repository #67

Fix issues with deb_index and Nvidia's apt repository #67

Conversation

benmccown commented Jul 18, 2024 • edited Loading

google-cla bot commented Jul 18, 2024

benmccown left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jjmaestro Sep 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jjmaestro Sep 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

benmccown commented Jul 19, 2024

benmccown commented Jul 19, 2024

benmccown commented Aug 21, 2024

thesayyn commented Aug 21, 2024

jjmaestro commented Sep 13, 2024

benmccown commented Jul 18, 2024 •

edited

Loading

jjmaestro Sep 19, 2024 •

edited

Loading

jjmaestro Sep 12, 2024 •

edited

Loading