webhook issue.

[pmanimuthu-ai]

Hi,
I have setup bulldozer server on linux server VM and I am getting invalid webhook error and service unavailable exception on the github advanced tab. Github server is maintained internally in a separate domain.

https://:8080/api/github/hook
We couldn’t deliver this payload: Service Unavailable

`[root@smac-build01 linux-amd64]# 5:14PM INF Server listening on localhost:8080

[root@smac-build01 linux-amd64]# curl -XPOST http://127.0.0.1:8080/api/github/hook
5:14PM WRN Received invalid webhook headers or payload error="missing event type\ngithub.com/palantir/go-githubapp/githubapp.(*eventDispatcher).ServeHTTP\n\t/home/runner/work/bulldozer/bulldozer/vendor/github.com/palantir/go-githubapp/githubapp/dispatcher.go:164\ngoji%2eio.dispatch.ServeHTTP\n\t/home/runner/work/bulldozer/bulldozer/vendor/goji.io/dispatch.go:17\ngithub.com/bluekeyes/hatpear.Recover.func1.1\n\t/home/runner/work/bulldozer/bulldozer/vendor/github.com/bluekeyes/hatpear/hatpear.go:113\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/hostedtoolcache/go/1.16.3/x64/src/net/http/server.go:2069\ngithub.com/bluekeyes/hatpear.Catch.func1.1\n\t/home/runner/work/bulldozer/bulldozer/vendor/github.com/bluekeyes/hatpear/hatpear.go:60\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/hostedtoolcache/go/1.16.3/x64/src/net/http/server.go:2069\ngithub.com/palantir/go-baseapp/baseapp.AccessHandler.func1.1\n\t/home/runner/work/bulldozer/bulldozer/vendor/github.com/palantir/go-baseapp/baseapp/middleware.go:88\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/hostedtoolcache/go/1.16.3/x64/src/net/http/server.go:2069\ngithub.com/rs/zerolog/hlog.RequestIDHandler.func1.1\n\t/home/runner/work/bulldozer/bulldozer/vendor/github.com/rs/zerolog/hlog/hlog.go:169\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/hostedtoolcache/go/1.16.3/x64/src/net/http/server.go:2069\ngithub.com/palantir/go-baseapp/baseapp.NewMetricsHandler.func1.1\n\t/home/runner/work/bulldozer/bulldozer/vendor/github.com/palantir/go-baseapp/baseapp/middleware.go:55\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/hostedtoolcache/go/1.16.3/x64/src/net/http/server.go:2069\ngithub.com/rs/zerolog/hlog.NewHandler.func1.1\n\t/home/runner/work/bulldozer/bulldozer/vendor/github.com/rs/zerolog/hlog/hlog.go:30\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/hostedtoolcache/go/1.16.3/x64/src/net/http/server.go:2069\ngoji%2eio.(*Mux).ServeHTTP\n\t/home/runner/work/bulldozer/bulldozer/vendor/goji.io/mux.go:74\nnet/http.serverHandler.ServeHTTP\n\t/opt/hostedtoolcache/go/1.16.3/x64/src/net/http/server.go:2887\nnet/http.(*conn).serve\n\t/opt/hostedtoolcache/go/1.16.3/x64/src/net/http/server.go:1952\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.16.3/x64/src/runtime/asm_amd64.s:1371" rid=c2j9i45b63bej67e0pdg
5:14PM INF http_request client_ip=127.0.0.1:36240 elapsed=0.788108 method=POST path=/api/github/hook rid=c2j9i45b63bej67e0pdg size=35 status=400 user_agent=curl/7.29.0
Invalid webhook headers or payload`

[pmanimuthu-ai]

webhook url: https://ipaddress:8080/api/github/hook
can u please guide me on how to capture the logs. curl -XPOST http://127.0.0.1:8080/api/github/hook gives me the log.

[bluekeyes]

It's difficult to give advice here without knowing more about your environment and configuration, but based on the information you included, I think Bulldozer is only listening for connections on the local interface. This means that GitHub, which is running on a different host, cannot connect.

Try setting server.address to 0.0.0.0 in your configuration file, as in the example. This will allow Bulldozer to accept requests from any host (assuming your firewall and network rules also allow it.)

[pmanimuthu-ai]

Thanks for your reply. We worked on the network rules and port, now able to get the logs as well. Facing below issue now.

11:46PM DBG received pull_request opened event github_delivery_id=99735360-b9c5-11eb-9587-fb5279698ab1 github_event_type=pull_request github_pr_num=1278 github_repository_name=MAC github_repository_owner=MAC rid=c2jf9llb63bde54b97d0
11:46PM DBG received pull_request review_requested event github_delivery_id=99841c40-b9c5-11eb-8186-2ed3679076cc github_event_type=pull_request github_pr_num=1278 github_repository_name=MAC github_repository_owner=MAC rid=c2jf9llb63bde54b97cg
11:46PM DBG github_request cached=false elapsed=145.23796 github_delivery_id=99735360-b9c5-11eb-9587-fb5279698ab1 github_event_type=pull_request github_pr_num=1278 github_repository_name=MAC github_repository_owner=MAC method=GET path=https://api.git.**.com/repos/MAC/MAC/pulls/1278 rid=c2jf9llb63bde54b97d0 size=-1 status=-1
11:46PM ERR Unexpected error handling webhook error="failed to get pull request MAC/MAC#1278: Get "https://api.git.viasat.com/repos/MAC/MAC/pulls/1278\": could not refresh installation id 0's token: request &{Method:POST URL:https://api.git.**.com/app/installations/0/access_tokens Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Accept:[application/vnd.github.machine-man-preview+json application/vnd.github.machine-man-preview+json] Authorization:[Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2MjE1NTQ0NTAsImlhdCI6MTYyMTU1NDM5MCwiaXNzIjoiMjcifQ.i7TJMPhFIjcyXR6VislXvrP0WqVSrrfecOhkC54dnc-hhm60CWryxv5zguhnBTYbTeojOxm4zdphoQ0PYxqZCzlGivYakwA3tFUbYlLc5aG17UpcFrFYD24_Ft1m-cILYyOxCG2jc5xtPfC-jm2_i38pwqz8pnhkq60zXXwPJZ-ip0FkN0Y7a43Zk48J9piGJWoQ5rsJu8QoTEZ8PptVwJeQDatg1ZEB26agInG9hCunKJlpg6esebfTLsaWRn78MT-RNfpAtXLVP8uAjkCw5JozWQK3Z6cyVn3S4jKsfFSb7y5L1qaKKIs4m2b_iiP_8j-cDk50d161n1_Ize83hw] Content-Type:[application/json]] Body:{Reader:} GetBody:0x72fba0 ContentLength:5 TransferEncoding:[] Close:false Host:api.git.viasat.com Form:map[] PostForm:map[] MultipartForm: Trailer:map[] RemoteAddr: RequestURI: TLS: Cancel: Response: ctx:0xc00028eab0} received non 2xx response status &{[] %!q(*http.bodyEOFSignal=&{0xc0004f40c0 {0 0} false 0x731760 0x7316e0}) %!q(*gzip.Reader=) } with body &{Method:POST URL:https://api.git.**.com/app/installations/0/access_tokens Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Accept:[application/vnd.github.machine-man-preview+json application/vnd.github.machine-man-preview+json] Authorization:[Bearer ] Content-Type:[application/json]] Body:{Reader:} GetBody:0x72fba0 ContentLength:5 TransferEncoding:[] Close:false Host:api.git.viasat.com Form:map[] PostForm:map[] MultipartForm: Trailer:map[] RemoteAddr: RequestURI: TLS: Cancel: Response: ctx:0xc00028eab0} and TLS &{Version:771 HandshakeComplete:true DidResume:false CipherSuite:49199 NegotiatedProtocol:http/1.1 NegotiatedProtocolIsMutual:true ServerName:api.git.viasat.com PeerCertificates:[0xc000374580 0xc000375b80 0xc000376100] VerifiedChains:[[0xc000374580 0xc000375b80 0xc000376c00]] SignedCertificateTimestamps:[] OCSPResponse:[] TLSUnique:[84 42 39 162 90 183 47 41 170 160 208 26] ekm:0x68c8c0}\ngithub.com/palantir/bulldozer/server/handler.(*PullRequest).Handle\n\t/home/runner/work/bulldozer/bulldozer/server/handler/pull_request.go:63\ngithub.com/palantir/go-githubapp/githubapp.Dispatch.Execute\n\t/home/runner/work/bulldozer/bulldozer/vendor/github.com/palantir/go-githubapp/githubapp/scheduler.go:56\ngithub.com/palantir/go-githubapp/githubapp.(*scheduler).safeExecute\n\t/home/runner/work/bulldozer/bulldozer/vendor/github.com/palantir/go-githubapp/githubapp/scheduler.go:183\ngithub.com/palantir/go-githubapp/githubapp.QueueAsyncScheduler.func1\n\t/home/runner/work/bulldozer/bulldozer/vendor/github.com/palantir/go-githubapp/githubapp/scheduler.go:257\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.16.3/x64/src/runtime/asm_amd64.s:1371" github_delivery_id=99735360-b9c5-11eb-9587-fb5279698ab1 github_event_type=pull_request rid=c2jf9llb63bde54b97d0
11:46PM DBG github_request cached=false elapsed=349.629672 github_delivery_id=99841c40-b9c5-11eb-8186-2ed3679076cc github_event_type=pull_request github_pr_num=1278 github_repository_name=MAC github_repository_owner=MAC method=GET path=https://api.git.viasat.com/repos/MAC/MAC/pulls/1278 rid=c2jf9llb63bde54b97cg size=-1 status=-1
11:46PM ERR Unexpected error handling webhook error="failed to get pull request MAC/MAC#1278: Get "https://api.git.**.com/repos/MAC/MAC/pulls/1278\": could not refresh installation id 0's token: request &{Method:POST URL:https://api.git.**.com/app/installations/0/access_tokens Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Accept:[application/vnd.github.machine-man-preview+json application/vnd.github.machine-man-preview+json] Authorization:[Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2MjE1NTQ0NTAsImlhdCI6MTYyMTU1NDM5MCwiaXNzIjoiMjcifQ.i7TJMPhFIjcyXR6VislXvrP0WqVSrrfecOhkC54dnc-hhm60CWryxv5zguhnBTYbTeojOxm4zdphoQ0PYxqZCzlGivYakwA3tFUbYlLc5aG17UpcFrFYD24_Ft1m-cILYyOxCG2jc5xtPfC-jm2_i38pwqz8pnhkq60zXXwPJZ-ip0FkN0Y7a43Zk48J9piGJWoQ5rsJu8QoTEZ8PptVwJeQDatg1ZEB26agInG9hCunKJlpg6esebfTLsaWRn78MT-RNfpAtXLVP8uAjkCw5JozWQK3Z6cyVn3S4jKsfFSb7y5L1qaKKIs4m2b_iiP_8j-cDk50d161n1_Ize83hw] Content-Type:[application/json]] Body:{Reader:} GetBody:0x72fba0 ContentLength:5 TransferEncoding:[] Close:false Host:api.git..com Form:map[] PostForm:map[] MultipartForm: Trailer:map[] RemoteAddr: RequestURI: TLS: Cancel: Response: ctx:0xc00001e930} received non 2xx response status &{[] %!q(*http.bodyEOFSignal=&{0xc00006e100 {0 0} false 0x731760 0x7316e0}) %!q(*gzip.Reader=) } with body &{Method:POST URL:https://api.git.***.com/app/installations/0/access_tokens Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Accept:[application/vnd.github.machine-man-preview+json application/vnd.github.machine-man-preview+json] Authorization:[Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2MjE1NTQ0NTAsImlhdCI6MTYyMTU1NDM5MCwiaXNzIjoiMjcifQ.i7TJMPhFIjcyXR6VislXvrP0WqVSrrfecOhkC54dnc-hhm60CWryxv5zguhnBTYbTeojOxm4zdphoQ0PYxqZCzlGivYakwA3tFUbYlLc5aG17UpcFrFYD24_Ft1m-cILYyOxCG2jc5xtPfC-jm2_i38pwqz8pnhkq60zXXwPJZ-ip0FkN0Y7a43Zk48J9piGJWoQ5rsJu8QoTEZ8PptVwJeQDatg1ZEB26agInG9hCunKJlpg6esebfTLsaWRn78MT-RNfpAtXLVP8uAjkCw5JozWQK3Z6cyVn3S4jKsfFSb7y5L1qaKKIs4m2b_iiP_8j-cDk50d161n1_Ize83hw] Content-Type:[application/json]] Body:{Reader:} GetBody:0x72fba0 ContentLength:5 TransferEncoding:[] Close:false Host:api.git..com Form:map[] PostForm:map[] MultipartForm: Trailer:map[] RemoteAddr: RequestURI: TLS: Cancel: Response: ctx:0xc00001e930} and TLS &{Version:771 HandshakeComplete:true DidResume:false CipherSuite:49199 NegotiatedProtocol:http/1.1 NegotiatedProtocolIsMutual:true ServerName:api.git.**.com PeerCertificates:[0xc0001be580 0xc0001beb00 0xc0001bf080] VerifiedChains:[[0xc0001be580 0xc0001beb00 0xc000376c00]] SignedCertificateTimestamps:[] OCSPResponse:[] TLSUnique:[149 171 43 90 207 237 218 28 60 187 139 17] ekm:0x68c8c0}\ngithub.com/palantir/bulldozer/server/handler.(*PullRequest).Handle\n\t/home/runner/work/bulldozer/bulldozer/server/handler/pull_request.go:63\ngithub.com/palantir/go-githubapp/githubapp.Dispatch.Execute\n\t/home/runner/work/bulldozer/bulldozer/vendor/github.com/palantir/go-githubapp/githubapp/scheduler.go:56\ngithub.com/palantir/go-githubapp/githubapp.(*scheduler).safeExecute\n\t/home/runner/work/bulldozer/bulldozer/vendor/github.com/palantir/go-githubapp/githubapp/scheduler.go:183\ngithub.com/palantir/go-githubapp/githubapp.QueueAsyncScheduler.func1\n\t/home/runner/work/bulldozer/bulldozer/vendor/github.com/palantir/go-githubapp/githubapp/scheduler.go:257\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.16.3/x64/src/runtime/asm_amd64.s:1371" github_delivery_id=99841c40-b9c5-11eb-8186-2ed3679076cc github_event_type=pull_request rid=c2jf9llb63bde54b97cg

[bluekeyes]

That error usually indicates that the server is missing configuration or the webhook configuration is incorrect. To start, can you confirm that:

1. In the Bulldozer server configuration, the github.app.integration_id property is set to the correct integration ID. This value is sometimes called the "application ID" and can be found on the settings page for the application in GitHub
2. The GitHub app is installed on the organization and repository in question. In particular, the webhook should be configured once at the application level. You should not manually add a webhook to the repository.

[pmanimuthu-ai]

I have the right integration_is configured but why does it looking for installation id 0's token?

"https://api.git.**.com/repos/MAC/MAC/pulls/1278\": could not refresh installation id 0's token: request &

[pmanimuthu-ai]

Here is my config:

address: "0.0.0.0"
port: 80
text: true
level: debug

github:
web_url: "https://git.***.com"
v3_api_url: "https://api.git.***.com/"

app:
integration_id: 27
webhook_secret: "GWvqNrTLpN"
private_key: |
-----BEGIN RSA PRIVATE KEY-----
-----END RSA PRIVATE KEY-----

options:
configuration_path: .bulldozer.yml

datadog:

address: "127.0.0.1:8125"
interval: "10s"

tags:
- "bulldozer"`

GitHub

[bluekeyes]

The installation ID refers to the repository or organization where Bulldozer is installed. We expect that ID to be included in the event payload with webhook event send for the app. The fact that it is 0 suggests that the payloads are not correct.

How did you configure the webhook in GitHub? Was it in the GitHub App settings or did you add the webhook configuration to an individual repository using the repository settings page?

[pmanimuthu-ai]

webhook is configured as mentioned in the readme http://domain-name:port//api/github/hook. This has been manually added to the github app configurations. It was also added to repository hook settings.

[pmanimuthu-ai]

what should be the base api url? I am following as per the examples https://api.git.**.com. Is that a right one to use? or I have to update as per the url path from the payload request?

[bluekeyes]

This has been manually added to the github app configurations. It was also added to repository hook settings.

You only need to add the hook to GitHub app settings. If you have the same hook in the repository settings, you should remove it from there.

what should be the base api url?

Generally, the API URL for a GitHub Enterprise Server install looks like:

https://github.domain/api/v3

Your server might be configured differently, but the api subdomain is usually only used for github.com.

[pmanimuthu-ai]

Thanks, that helped to make the server run without errors. I have both the ==MERGE_WHEN_READY and ==UPDATE_ME== substrings in my PR. Update is not happening with PR body substring but it works for labels. Is there anything I have to update bulldozer.yml?

update: # "trigger" defines the set of pull requests that should be updated by # bulldozer. It accepts the same keys as the trigger in the "merge" block. trigger: labels: ["WIP", "Update Me"] # These are supposed to trigger updates on stale branches comment_substrings: ["==UPDATE_ME=="] pr_body_substrings: ["==UPDATE_ME=="] # "ignore" defines the set of pull requests that should not be updated by # bulldozer. It accepts the same keys as the ignore in the "merge" block. ignore: labels: ["Do Not Update"]

[bluekeyes]

I think you may be hitting #145. Currently, when updates are enabled by comment, the update will only happen when a change is pushed to the target branch after adding the comment. For labels, the behavior was fixed recently so that a PR will update when the label is added as well as when the target branch updates.

I think the substring in the PR body will also work if it is present when the PR is opened, but if added after the fact, the target branch must update before Bulldozer will update the PR.