Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Delay requests differently #71

Open
rgaudin opened this issue May 28, 2020 · 2 comments
Open

Delay requests differently #71

rgaudin opened this issue May 28, 2020 · 2 comments
Labels
refactor

Comments

@rgaudin
Copy link
Member

rgaudin commented May 28, 2020

HTML content download and subtitles files are currently preceded by an arbitrary time.sleep within the download method.

Apparent goal was to prevent 429 errors.

This is somewhat unnatural.
@kelson42 kelson42 pinned this issue Jun 2, 2020
@kelson42
Copy link
Contributor

kelson42 commented Sep 4, 2020

This seems to slows down seriously Zim generation. Even with all videos in cache, TED playlist recipe takes many days to complete.

@rgaudin
Copy link
Member Author

rgaudin commented Sep 4, 2020

To be a bit more precise, the total 580 ZIMs are created in about 7days which averages each ZIM 17mn which is way too long to download even 20 videos (not sure how many are in average in each playlist) from S3.

I am not able to support this issue as the reason for this delay but we'll investigate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
refactor
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@rgaudin @kelson42