Databus 2.0 Client Design
Databus Clients are interested in consuming the events from the Databus Relays. They have to integrate with a library called databus client library, which provides APIs for choosing the change stream they are interested in.
Here is a high-level overview of the client.
The connection used for obtaining real-time update stream from the relay using the relay HTTP interface.
The consumption of events follows the online stream consumption state machine.
The connection used for obtaining long look-back updates from the bootstrap server using the bootstrap HTTP interface
The consumption of events follows the bootstrap consumption state machine.
Persistence is either local (in a file) or shared (in Zookeeper).
The dispatcher reads incoming events from online streams or bootstrap and invokes the consumer code callbacks. Its main responsibilities are:
- Determine the right callback
- Monitor for errors and timeouts
- Ensures that the client library persist the consumer progress of event consumption
The dispatcher follows the state machine.
Consumer code that implements the Consumer callback API.
Callbacks are executed using the following execution model. See also the section on callbacks in the Databus migration wiki
The checkpoint is an internal representation of the point in the consumption of stream of updates by a consumer.
The default format is a JSON representation of the Checkpoint object.
{"windowOffset":-1,"snapshot_offset":-1,"prevScn":-1,"windowScn":5984508975840,"consumption_mode":"ONLINE_CONSUMPTION"}
{"snapshot_offset":0,"prevScn":5984488321377,"windowScn":5984488321377,"consumption_mode":"BOOTSTRAP_SNAPSHOT","windowOffset":-1,"bootstrap_since_scn":5984488321377,"bootstrap_start_scn":-1,"bootstrap_target_scn":-1,"bootstrap_snapshot_source_index":0,"bootstrap_catchup_source_index":0,"bootstrap_server_info":null,"snapshot_source":"com.linkedin.databus.example.Person"}