Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Concurrent load is too vulnerable to slow extracts #74

Open
bhtucker opened this issue Dec 20, 2017 · 1 comment
Open

Concurrent load is too vulnerable to slow extracts #74

bhtucker opened this issue Dec 20, 2017 · 1 comment

Comments

@bhtucker
Copy link
Contributor

If extracts stop for an hour, a concurrent load will exit with a ETLRuntimeError, regardless of whether the relations it is waiting on extracts for are required.

@tvogels01
Copy link
Contributor

The first approach here would be to have "failure" events sent into the load queue, leaving it up to the load workers to discover whether the relation is required or not.

The concurrent load bears some refactoring with the monitor package. An idea here is that this pseudo-event of a failed extract (= failed due to timeout) is added to the queue where the monitor querier is the producer and the concurrent load is the consumer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants