✨ Use custom-built Bazel

[aaronmondal]

The variant with bazelisk didn't work well with nix-built multi-layer remote execution images. Switch to an overridden Bazel from nixpkgs and add logic to create such images.

The images about 2.6GB in size which is almost as small as we can get for an image containing Bazel and a Clang/LLVM C++ toolchain. We might be able to shave off ~250MB by building python with that same toolchain instead of building it with GCC.

Preparation for #47.

[aaronmondal]

Ignore pre-commit hooks for now. I'll split this up into smaller pieces before committing. The new hook failures are probably caused by updating our lockfiles.

The bigger issue at the moment is the size of the image. I think if we build python with the custom clang toolchain we no longer need GCC in the image. We'll still need libstdc++ but not the rest of the toolchain.

I'm also wondering whether we can kick out the jdk somehow. We need a jdk to build and run Bazel, but it might be possible to fetch a remotejdk instead of using a container-local one.

Current dependency log obtained with du -h -d1 | sort -h in the /nix/store of the running container:

Details

4.0K	./9kkavzbcljrvdr3f7sf24ar9glsmpdf8-nsswitch.conf
4.0K	./gbvqgg7pxgishv5iq6my6mj27ivgf4zz-set-java-classpath-hook
4.0K	./pbjzq3i1d5r4k5jjyzckhh5x48x9pffd-group
4.0K	./qqj6d0fvzwfqg92b8c6z6xh03g5nfby6-setup-debug-info-dirs-hook
4.0K	./v3v0ygf2h5k4bdynwph6wf2s01z8lckq-bazel
4.0K	./xim8w6x2ghx4vilcfbajfmk3l0q020zb-passwd
16K	./736qfwdicqrfva4xfi3rqam27ph0j91r-expand-response-params
44K	./l966ai4an2rcmx108fbx83ir8q6v525j-libXdmcp-1.1.3
48K	./6337pr6nbsy2wrm6d34b5376fmmrjdim-which-2.21
52K	./h0arq47yvi90s7wdfibgzyzx44hb6ngw-stdenv-linux
68K	./a7sx3qgvcyjnpp1xbsbxzalg8wymkadp-libXau-1.0.9
72K	./gy3i9akr5w2dqdzjijcn0nv1df93bnw8-libffi-3.4.4
84K	./kldzcblc7z1hgm80gj3g20kivd4zm36j-libpciaccess-0.16
96K	./fhy0p7rapikn15bx7rv797k7fl0shm7n-bzip2-1.0.8
100K	./jhf2q2pnnmyx19ksgx7l3f1swg79iqlh-attr-2.5.1
100K	./qggx5h4iywx3h6la1r9mr5zll9dzvfi5-bzip2-1.0.8-bin
108K	./rsnhn16mcc99scnfbygg453jk6iz5y9l-libXext-1.3.4
116K	./zjra345614dcy0gyhd2nir2rcz04xymf-mailcap-2.1.53
132K	./jbn6y1s3473zgg42hpv5nyyda337pqh1-acl-2.3.1
136K	./xfkzh15vxmxhiza2s2din1az0vzfaf56-zlib-1.2.13
144K	./h14srdmabzg0jvk6pajlb5df47yvhyjs-binutils-wrapper-2.40
148K	./0bxlqaklbcjy1cn5xw03jd5kxn39jn0l-libxcrypt-4.4.33
148K	./hajlnhg9d7wzlyx8836kjkw64jc41lnz-ed-1.19
184K	./k68f3bcgb9v7y6l7x1nfcydxyjhn66na-gzip-1.12
204K	./h0g80pkbx5y6yb2s2gl0i9dhm855y8yq-libunwind-15.0.7
204K	./kd98npria0775xxb7dpgd3nz0p9223ci-clang-wrapper-15.0.7
220K	./hrdczcf8pw9zmhxqyji8zn7ng1q27qlj-patchelf-0.15.0
224K	./p2vx0k58apphclz1p43vppx0skbfk9b2-patch-2.7.6
240K	./rmxg33s5z1qrv8ygc932phs2q5qfmy48-elfutils-0.189-dev
248K	./3p9fdg7aivd53r196pngqnm77kxwzmyr-xz-5.4.1-bin
268K	./dxrw8x2d270hnifmipif25mmp2alfhjs-libdrm-2.4.115-bin
268K	./hlvahl71n1jjdr7jlqpxwzcjpb6kwaf9-expat-2.5.0
288K	./7vlcc3i5z21j9shqh25yb8vhnrk21ywy-libX11-1.8.4-dev
308K	./dh0jscni558i609q1s299qlrdbdj9jxa-libidn2-2.3.2
332K	./kkphyn9m9z1wvvbcrdv0c3v9dahyv494-numactl-2.0.16
456K	./80fkjm4n0wlv4m9xiqc2kl3l8xn6zg4p-lcms2-2.15
480K	./7yrdb4cb3xqy8xlqmvdn5yalzcjz98r6-nss-cacert-3.86
480K	./vqwr89jhfgmiid7dxshp5d8ahp5k3a1s-readline-8.2p1
516K	./wv3qq3g1c4zmslan7d52snwwjds2ldrn-unzip-6.0
540K	./2yf4pf8wnv0mdb8s3nhsqw34mgcivhq7-pcre-8.45
588K	./iy2bwsb1ka42yi0643vrwpl19ybrb3fp-zip-3.0
604K	./wm7z7ny9p30471bgws2sh9zxrvxdmc9j-libdrm-2.4.115
752K	./fycm6z3hzs50vmla6inp8g3bywxjfj7h-compiler-rt-libc-15.0.7-dev
756K	./70nz77cfml6xqk813v7gzp2zgprpnxyz-gmp-with-cxx-stage4-6.2.1
760K	./hdq087rhq35fl5d5p6pw2m7hr2ib0f4h-gnused-4.9
820K	./skd2pbqccdddfxv3spqq0qlpcp4a7f8w-xz-5.4.1
856K	./xx3h5mam81l0v1hc31p0x2pr4zr55p8n-libdrm-2.4.115-dev
864K	./kp1chfvl8bai7rd2g80lk3xjz4v76j1v-gnugrep-3.7
880K	./drj64s6r8jsqpjlcbf13jbhdb7bnwma8-gdbm-1.23
1.1M	./052gqrqyg56ibmm0nf2njad0f4wkj5c4-libcxxabi-15.0.7
1.1M	./scl69wk3mswn05bhdxwvrlbkvq53a5ib-zstd-1.5.4
1.4M	./4i9pghwn2bg6xpiwyqgiajia0y38k0jj-findutils-4.9.0
1.4M	./hk8w4kray4jbc8bbpaxb0dfa95jm7syj-findutils-4.9.0
1.5M	./kvj4j4ssr2abq4583yc31nvswdf9vf38-sqlite-3.41.1
1.6M	./gwzg90l0189a7b2g70f76lv0aq1p9kki-gnumake-4.4.1
1.6M	./ja73jc04spx5ycdfidx4v7f93vhmq3g0-libxml2-2.10.3
1.6M	./wx5j7zk8cn7spmyq1f15i6yymakvgp8p-libjpeg-turbo-2.1.4
1.7M	./0hx32wk55ml88jrb1qxwg5c5yazfm6gf-bash-5.2-p15
1.7M	./bkhiljrfb5yninnky3k8wlamgs3q3n0b-diffutils-3.9
1.7M	./kbshlw7apays3b1isrk7b903qgr42w30-libxcb-1.14
1.8M	./cnjwrrhnqn114jf5x0dwhx3jiaw6600d-libunistring-1.1
1.9M	./lyicmql3ws929d7azr65h25b2hyakmb6-coreutils-9.1
1.9M	./v3kj3w95cyccpchfrxq31ja2zbg0xply-coreutils-9.1
2.1M	./awi1b18hc9fy439hqwr7wmkwn75kyrc1-xorgproto-2021.5
2.2M	./wckgwcmmm3jw67accq715qlfrggqlyvg-libxcb-1.14-dev
2.3M	./00kynq23n41hardvbigvsh73pla4bkwr-libglvnd-1.6.0-dev
2.6M	./ng2jici7wjrjx6c7pjsbpl9ki5pi4arj-elfutils-0.189-bin
2.6M	./vqp1l0z0psg0jy0lyh7ip5qb5y5zj6g1-libglvnd-1.6.0
2.8M	./3ncx1mqvfzxfd3731r2iy47yssjixlp3-gawk-5.2.1
2.8M	./3r56fyalpf2na0j4qiyfmxs1p5c0yvv7-binutils-2.40-lib
2.9M	./sdcicnwqb4ziv66nhkklsc06vkwb3vxs-gnutar-1.34
3.1M	./2fdhd8fchkqcc01mrq75cpcks6830f7h-glibc-2.35-224-bin
3.1M	./dhnj3ia6nwr0w9ddpyv7l1c3n308kcvy-libX11-1.8.4
3.3M	./2yqq5627yd9i5rba8r4div57jc2092y8-glibc-2.35-224-dev
3.4M	./3c8ib5l3cxnscwcy3nh29186aiy4my5a-libcxx-15.0.7
3.6M	./q40pj3swjqgsjql70l548pq1snbk7n6w-elfutils-0.189
5.2M	./w2swil1dw4fxmvkggcik4l0jiywwp5vp-tzdata-2022g
6.2M	./8dvm4zanl2bviazw216kvk5kkd4y4yy9-openssl-3.0.8
6.3M	./45x7bvx5sywsha7pmg19x1vkvg69zbbi-lld-15.0.7
7.9M	./gd5k5pvxqxmbnac01b8biga409nl4aw5-libcxx-15.0.7-dev
8.0M	./shasq3azl2298vqkvq5mc7vivdqp3yrj-gcc-12.2.0-lib
8.2M	./9gpn5dsgws1sg5mhmqrmkhwqs6bdjayy-file-5.44
8.2M	./b6c0wh2dkm8v5cklvf0svi81g1cpqkv1-file-5.44
8.3M	./m7yaryscghpp764ibwlaivl2k0c9v3cl-linux-headers-6.2
13M	./7y04gsisjii9wsxmf36a0d2kv1g72dn1-ncurses-6.4
13M	./axvqav9b9j2m393byn9wh6gbkj6li13s-lld-15.0.7-lib
29M	./8qm6sjqa09a03glzmafprpp69k74l4lm-binutils-2.40
29M	./wg5s523vrnphn3fy2yydypkn4bnbk1ws-compiler-rt-libc-15.0.7
31M	./8xk4yl1r3n6kbyn05qhan7nbag7npymx-glibc-2.35-224
43M	./am0lqpl9fbkp3qlg4qbxgz2smhaskv32-bazel-6.1.1
68M	./hhgbk0gvfml5z5y4y5g9f7qrk7a4cbx6-clang-15.0.7
91M	./syz2y6j53y5hpzbs7l0965zwxshi8iyl-python3-3.10.10
219M	./p975i9blgmkjfxpnlvdmm0xvjg573b6l-gcc-12.2.0
320M	./qf9hq56ki1ay8si97xip2ddvvbillw5x-bazel-deps
433M	./grazlyrwf7z6hr5wfabh0j17kqjmcmq2-llvm-15.0.7-lib
517M	./wj68kipky0353zkb9bk4sy53jvpsdsqq-openjdk-headless-11.0.18+10
666M	./wdx87ianmabqbgdbgdqmdnh4lrbzlyii-clang-15.0.7-lib

[aaronmondal]

Ironically the gcc dependency seems to be caused by llvmPackages_15.clang and llvmPackages_15.stdenv 😂

[SpamDoodler]

Could you shortly explain what the code in baze_ll.nix does? To my understanding, most of the magic is in

pkgs.bazel_6.overrideAttrs

Does this line automatically trigger a rebuild of bazel_6 with the altered attributes?

[aaronmondal]

The original build file for Bazel is here. The override works in two stages.

The overrideAttrs overrides the input arguments for the build file so that we can pass custom CFLAGS/LDFLAGS to the build.

The override then swaps the regular nix C++ toolchain with the one declared in the llvmOverride. This way we use Clang 14 as compiler, but can still statically link Clang 15 libcxx etc into the final executable.

[aaronmondal]

I'll probably need to completely rework this. This won't make it into the next release since I want it to work with remote execution properly first. That requires more effort both on the Nix side and on the cc_toolchain/Bazel side.

[aaronmondal]

Superceded by #83.