Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Transactional Pipeline to Stream outputs #50

Open
thedodd opened this issue May 24, 2021 · 0 comments
Open

Transactional Pipeline to Stream outputs #50

thedodd opened this issue May 24, 2021 · 0 comments
Labels
A-pipelines Hadron server pipelines A-streams Hadron server streams

Comments

@thedodd
Copy link
Collaborator

thedodd commented May 24, 2021

Transactional Pipeline -> Stream outputs.

Introduce the new streamOutput field for pipeline stages.

  • Stream outputs will be required to be supplied as part of the pipeline ack payload, just as a stage output is required.
  • The payloads will be written to disk along side the pipeline stage's normal output, but will be nested under a separate prefix as {prefix}{offset}{stage_name}.
  • A notification will prompt the Stream controller that Pipeline Stream outputs need to be added to the Stream. It will transactionally copy the data to the Stream in proper order, preserving Stream controller encapsulation.
  • This will provide a more structured model for transactional processing overall.
  • Only a normal stage output can be used as input to another stage, not stream outputs.
@thedodd thedodd added A-pipelines Hadron server pipelines A-streams Hadron server streams labels May 24, 2021
@thedodd thedodd changed the title Transactional cross-stream pipeline outputs Transactional Pipeline to Stream outputs Oct 11, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-pipelines Hadron server pipelines A-streams Hadron server streams
Projects
None yet
Development

No branches or pull requests

1 participant