buildbarn configuration of goma

[qinjidong]

Hello developer,
Is there a wizard that can show me how to connect buildbarn with goma (remoteexec_proxy) please?

Best Regards

[EdSchouten]

Hey! I asked around on Slack whether anyone had any experience. Nobody seemingly responded.

If you did manage to figure this out, be sure to share your findings. This makes life easier for the next person.

[stagnation]

We have ongoing work to help other build systems in here: #95
I'd be happy to add Goma to that.

[stagnation]

Hi Again

I could not get goma to work with Buildbarn, it should work - but I get http errors from weird servers/endpoints
and this is not my forte. The configuration to proxy from goma server to Buildbarn is simple and can be reused. But I could not set up the goma client to work.

I wrote some notes here that I hope will speed up development if we make another attempt,
and maybe someone with more experience can spot my errors.
https://github.com/stagnation/bb-deployments/blob/feature/do-not-merge-failed-goma-attempt/goma-failed-attempt.rst

[qinjidong]

Hi Stagnation,

I read the steps you wrote, there are some trouble in Goma's authentication.
Try to avoid use goma auth may make things easier.
Follow the steps of bromite/bromite#1032 can avoid goma auth.

Best Regards

[stagnation]

Thanks for the help! I will try that.

Edit: I still trudge forward, and slowly get a better picture of everything. That more people have tinkered in this domain is reassuring, and gives more weight to the effort. I'm starting to think that there are two errors:

1. authenticating with OAuth, this can either be done with a proper service account like I did, or by patching it away.
   The latter is much better for others to reuse, so going through this in detail is good.

2. the compiled goma client pings against the backend, which should be the local server, and requests the endpoint /cxx-compiler-service, but for some reason this connection fails with the 502 error.

I still get the second error, but that could have been introduced in newer versions of the client, which were not present at the time of writing the two guides. So we may need to patch the client to do this right, or find good configuration to make it work.

---

I20230818 12:42:06.733821 2194498 http.cc:872] dest=127.0.0.1:19080 url_path_prefix=/cxx-compiler-service authorization:enabled socket_read_timeout=10s retry_backoff=500ms .. 5s
I20230818 12:42:06.733834 2194498 http_rpc.cc:179]  compression_level=3 start_compression accept_encoding=gzip content_type_for_protobuf=binary/x-protocol-buffer
I20230818 12:42:06.733837 2194498 http_rpc.cc:190] request encoding=gzip
I20230818 12:42:06.735414 2194498 compile_service.cc:237] include_processor_pool=3 num_thread=32
I20230818 12:42:06.735420 2194498 log_cleaner.cc:38] log basename:compiler_proxy
I20230818 12:42:06.735422 2194498 log_cleaner.cc:38] log basename:compiler_proxy-subproc
I20230818 12:42:06.735424 2194498 log_cleaner.cc:38] log basename:gomacc
I20230818 12:42:06.735425 2194498 log_cleaner.cc:38] log basename:cc
I20230818 12:42:06.735427 2194498 log_cleaner.cc:38] log basename:c++
I20230818 12:42:06.735428 2194498 log_cleaner.cc:38] log basename:gcc
I20230818 12:42:06.735430 2194498 log_cleaner.cc:38] log basename:g++
I20230818 12:42:06.735431 2194498 log_cleaner.cc:38] log basename:clang
I20230818 12:42:06.735432 2194498 log_cleaner.cc:38] log basename:clang++
I20230818 12:42:06.735433 2194498 log_cleaner.cc:38] log basename:goma_fetch
I20230818 12:42:06.735452 2194498 http_rpc.cc:210] ping /ping timeout=10s
I20230818 12:42:06.735457 2194504 log_cleaner.cc:44] clean old logs in /tmp/ ./
I20230818 12:42:06.735518 2194505 worker_thread.cc:620] io closure: 0x7fea800013d0
W20230818 12:42:06.735904 2194505 http.cc:2000] ping read  http=502 path=/cxx-compiler-service/ping Details:HTTP/1.1 502 Bad Gateway\r\nDate: Fri, 18 Aug 2023 10:42:06 GMT\r\nContent-Length: 0\r\n\r\n

---

If we add the port we do reach the remote_proxy.

$ curl -XPOST 'localhost:5050/cxx-compiler-service/ping'
ok

This complains for the user-agent, but is a clear sign of life.

---

I'm starting to think that something is fishy about the http_proxy,
note the destination IP in the log and in wireshark.

I cannot get it to accept the ping request through curl, by starting it on its own.
I would want it to just send the POST forward to the proxy_server, which does work as we saw above.

$out/Release/http_proxy -port 10000 -server-host 127.0.0
.1:5050
# curl to it in another terminal
2023/08/18 15:28:14 http: proxy error: tls: first record does not look like a TLS handshake

Leaving this for the weekend, will look into its certificates and such next week.
I'm a little bit slow as this is not my strong suit.

[qinjidong]

Hi,

I cannot connect remoteexec_proxy to bb_storage.
What remoteexec-addr should I use in remoteexec_proxy's command line?

Referring to buildgrid, I use the following command line to start remoteexec_proxy.
go run cmd/remoteexec_proxy/main.go -port 7979 -insecure-remoteexec -remoteexec-addr localhost:7980 -remote-instance-name default --exec-config-file config_file

Config file of buildgrid.
server:

• !channel
  port: 7980
  insecure-mode: true
  ...

[EdSchouten]

Hey @qinjidong,

I saw on Slack that you managed to get it working now. Just to help out others in the future, would you be willing to share the configs you used in this ticket? Be sure to close the issue afterwards. Thanks!

[qinjidong]

I mainly encountered the following two problems during the adaptation process.

1. connect remoteexec_proxy to bb_storage
   -> remoteexec_proxy needs a -platform-container-image as well.
   -> like this:
   goma/server $ go run
   cmd/remoteexec_proxy
   ...
   -platform-container-image 'docker://ghcr.io/catthehacker/ubuntu:act-22.04@sha256:5f9c35c25db1d51a8ddaae5c0ba8d3c163c5e9a4a6cc97acd409ac7eae239448'

2. Upload the toolchain to bb_runner, then run it inside docker. Because I need do cross compile, while the toolchain path is fixed.
   -> docker cp {toolchain} docker-compose_runner-fuse-ubuntu22-04_1:{toolchain_path}

[EdSchouten]

Thanks!