Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Diff server should return a 503 status for upstream 503 responses #20

Open
Mr0grog opened this issue Nov 4, 2019 · 2 comments
Open
Labels
enhancement New feature or request server Specific to the diffing server, rather than diff algorithms

Comments

@Mr0grog
Copy link
Member

Mr0grog commented Nov 4, 2019

The diff server usually handles upstream errors (I.e. the result of fetching the URL for either the a or b parameters is an error) by returning a 502 (bad gateway) status. However, if the upstream response was a 503 (service unavailable) status, we should probably return the same, including the upstream Retry-After response header, if present.

503 is often used to tell the requesting server to slow down, so it would be helpful to send that info back to the downstream server, since are just acting dumb in the middle.

In fact, if an upstream response comes back as any error with a Retry-After header, we should probably send along that header to the downstream client.

(We sometimes get this status from S3 when running imports and auto analysis, so it would be nice to handle it better.)

@Mr0grog
Copy link
Member Author

Mr0grog commented Feb 13, 2020

We should also handle 429 (too many requests) statuses this way, too. They also frequently (but not always) have a Retry-After header.

@Mr0grog
Copy link
Member Author

Mr0grog commented Feb 13, 2020

The relevant place in the code for this is here, in fetch_diffable_content() in web_monitoring/diff_server/server.py: https://github.com/edgi-govdata-archiving/web-monitoring-processing/blob/5220721ff3b414f653cb66510d150c1bde2b9202/web_monitoring/diff_server/server.py#L290-L306

@edgi-govdata-archiving edgi-govdata-archiving deleted a comment from stale bot Aug 17, 2020
@Mr0grog Mr0grog transferred this issue from edgi-govdata-archiving/web-monitoring-processing Oct 26, 2020
@Mr0grog Mr0grog added the server Specific to the diffing server, rather than diff algorithms label Nov 5, 2020
@stale stale bot added the stale label Jun 2, 2021
@Mr0grog Mr0grog removed the stale label Jun 2, 2021
@edgi-govdata-archiving edgi-govdata-archiving deleted a comment from stale bot Jun 2, 2021
@stale stale bot added the stale label Jan 8, 2022
@stale stale bot closed this as completed Apr 16, 2022
@Mr0grog Mr0grog added enhancement New feature or request and removed stale labels Apr 17, 2022
@edgi-govdata-archiving edgi-govdata-archiving deleted a comment from stale bot Apr 17, 2022
@Mr0grog Mr0grog reopened this Apr 17, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request server Specific to the diffing server, rather than diff algorithms
Projects
None yet
Development

No branches or pull requests

1 participant