Add retries to bypass BigQuery's 10 MB payload limit

[imor]

If the payload to the AppendRows call is greater than 10 MB, BigQuery will reject the call. Detect this and retry with the payload half the original size. Continue this logarithmic halving until the requests succeed. Also collect the retry data as metrics to tune the buffer sizes and retries.

[Ziinc]

Isn't it easier to cap the batches to like 8MB? That is the current logflare approach.

[imor]

@Ziinc can you share the logflare code where this is done? What exactly do you cap to 8MB? Is it the payload serialized in the protobuf format? Or something else?