KV API and misc

[thedodd]

• KV API for targeted updates of event data fields. Events identified by their partition & offset (which is absolute per partition).
• Hadron Client & CLI will expose interfaces for this.
  • Should support querying for an event by offset & partition.
  • Should support updating an event by offset and partition.
• Events should be updated when written to a stream to include the Hadron partition & offset as extension fields.

---

• (Move to new ticket) Delivery count tracking. Will just have Stream subscription controller & pipeline controller emit metrics for these when it happens. Different metric types, one for Streams the other for Pipelines.

• (Move to new ticket) We will be blocked on this by the upstream metrics project, as they do not yet have an API for removing registered metrics. Forking and running this should be quite easy.

• (Move to new ticket) Delivery count tracking should be able to cause alerts to be generated. Add a new panel with a table of Stream events which have high delivery counts, and one for Pipelines as well.

• (Move to different new ticket) Probably support nack with delay. This would only be useful for Streams when we support out of order processing. And only useful for Pipelines when we do not limit max concurrency for stale jobs due to not having active stage consumers.