Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Avoid http.Client recreation on every relay #1484

Open
jorgecuesta opened this issue Oct 12, 2022 · 0 comments
Open

Avoid http.Client recreation on every relay #1484

jorgecuesta opened this issue Oct 12, 2022 · 0 comments
Assignees
@oten91
Labels
proposal Pocket Core client proposal

Comments

@jorgecuesta
Copy link
Contributor

Please explain your change in detail.
Create a single HTTP client that will be used to call chains, using the best tunning performance settings for it based on a bit of research (google it) about how to speed up and reduce open files in Golang http.Client

Please provide a justification for your change.
Every time a relay is received on the background of that process, a new http.Client creates and this is not the best way to do, it because it will not reuse connections, it will open more and more file descriptors and that could reach system limits or even grows up the needed RAM to process bigger relay bursts.
@jorgecuesta jorgecuesta added the proposal Pocket Core client proposal label Oct 12, 2022
jorgecuesta added a commit to pokt-scan/pocket-core that referenced this issue Oct 12, 2022
@jorgecuesta
Fix pokt-network#1457; Fix pokt-network#1484; Fix a high memory consu…
0aa4d5f 
…mption that also is part of the issue pokt-network#1457.

Under high load of requests (1000/rps or more) the RAM got crazy and scale up to 40GB or close to that.
Now after the fix of pokt-network#1457 with the worker pool, the node remains under 14gb of ram in my local tests.
@jorgecuesta jorgecuesta mentioned this issue Oct 12, 2022
Merged
jorgecuesta added a commit to pokt-scan/pocket-core that referenced this issue Oct 12, 2022
@jorgecuesta
Merge: Fix pokt-network#1457; Fix pokt-network#1484; Enhance http.Client
3dc05ac
@jessicadaugherty jessicadaugherty moved this to Up Next in V0 Dashboard Oct 19, 2022
@jessicadaugherty jessicadaugherty moved this from Up Next to In Progress in V0 Dashboard Nov 16, 2022
jorgecuesta added a commit to pokt-scan/pocket-core that referenced this issue Nov 18, 2022
@jorgecuesta
Fix pokt-network#1457; Fix pokt-network#1484;
94b75f2 
Fix a high memory consumption that also is part of the issue pokt-network#1457.
Under high load of requests (1000/rps or more) the RAM got crazy and scale up to 40GB or close to that.
Now after the fix of pokt-network#1457 with the worker pool, the node remains under 14gb of ram in my local tests.
jorgecuesta added a commit to pokt-scan/pocket-core that referenced this issue Nov 23, 2022
@jorgecuesta
Fix pokt-network#1457; Fix pokt-network#1484;
8a78559 
Fix a high memory consumption that also is part of the issue pokt-network#1457.
Under high load of requests (1000/rps or more) the RAM got crazy and scale up to 40GB or close to that.
Now after the fix of pokt-network#1457 with the worker pool, the node remains under 14gb of ram in my local tests.
@jessicadaugherty jessicadaugherty moved this from In Progress to In Review in V0 Dashboard Nov 28, 2022
oten91 pushed a commit that referenced this issue Nov 28, 2022
@jorgecuesta @oten91
Fix #1457; Fix #1484;
fa8fca5 
Fix a high memory consumption that also is part of the issue #1457.
Under high load of requests (1000/rps or more) the RAM got crazy and scale up to 40GB or close to that.
Now after the fix of #1457 with the worker pool, the node remains under 14gb of ram in my local tests.
oten91 pushed a commit that referenced this issue Dec 1, 2022
@jorgecuesta @oten91
Fix #1457; Fix #1484;
fd4649d 
Fix a high memory consumption that also is part of the issue #1457.
Under high load of requests (1000/rps or more) the RAM got crazy and scale up to 40GB or close to that.
Now after the fix of #1457 with the worker pool, the node remains under 14gb of ram in my local tests.
@Olshansk Olshansk moved this from In Review to Up Next in V0 Dashboard Mar 20, 2023
@Olshansk Olshansk moved this from Up Next to Backlog in V0 Dashboard Mar 20, 2023
@jessicadaugherty jessicadaugherty moved this from Backlog to Done in V0 Dashboard Mar 20, 2023
jorgecuesta added a commit to pokt-scan/pocket-core that referenced this issue Mar 22, 2023
@jorgecuesta
Fix pokt-network#1457; Fix pokt-network#1484;
0e8b845 
Fix a high memory consumption that also is part of the issue pokt-network#1457.
Under high load of requests (1000/rps or more) the RAM got crazy and scale up to 40GB or close to that.
Now after the fix of pokt-network#1457 with the worker pool, the node remains under 14gb of ram in my local tests.
jorgecuesta added a commit to pokt-scan/pocket-core that referenced this issue Jun 8, 2023
@jorgecuesta
Fix pokt-network#1457; Fix pokt-network#1484;
e6b88ee 
Fix a high memory consumption that also is part of the issue pokt-network#1457.
Under high load of requests (1000/rps or more) the RAM got crazy and scale up to 40GB or close to that.
Now after the fix of pokt-network#1457 with the worker pool, the node remains under 14gb of ram in my local tests.
jorgecuesta added a commit to pokt-scan/pocket-core that referenced this issue Jun 21, 2023
@jorgecuesta
Fix pokt-network#1457; Fix pokt-network#1484;
509a512 
Fix a high memory consumption that also is part of the issue pokt-network#1457.
Under high load of requests (1000/rps or more) the RAM got crazy and scale up to 40GB or close to that.
Now after the fix of pokt-network#1457 with the worker pool, the node remains under 14gb of ram in my local tests.
jorgecuesta added a commit to pokt-scan/pocket-core that referenced this issue Jul 10, 2023
@jorgecuesta
Fix pokt-network#1457; Fix pokt-network#1484;
8b6b83c 
Fix a high memory consumption that also is part of the issue pokt-network#1457.
Under high load of requests (1000/rps or more) the RAM got crazy and scale up to 40GB or close to that.
Now after the fix of pokt-network#1457 with the worker pool, the node remains under 14gb of ram in my local tests.
jorgecuesta added a commit to pokt-scan/pocket-core that referenced this issue Jul 10, 2023
@jorgecuesta
Fix pokt-network#1457; Fix pokt-network#1484;
85a4d15 
Fix a high memory consumption that also is part of the issue pokt-network#1457.
Under high load of requests (1000/rps or more) the RAM got crazy and scale up to 40GB or close to that.
Now after the fix of pokt-network#1457 with the worker pool, the node remains under 14gb of ram in my local tests.
jorgecuesta added a commit to pokt-scan/pocket-core that referenced this issue Jul 10, 2023
@jorgecuesta
Fix pokt-network#1457; Fix pokt-network#1484;
b86408c 
Fix a high memory consumption that also is part of the issue pokt-network#1457.
Under high load of requests (1000/rps or more) the RAM got crazy and scale up to 40GB or close to that.
Now after the fix of pokt-network#1457 with the worker pool, the node remains under 14gb of ram in my local tests.
jorgecuesta added a commit to pokt-scan/pocket-core that referenced this issue Jul 10, 2023
@jorgecuesta
Fix pokt-network#1457; Fix pokt-network#1484;
212e1ee 
Fix a high memory consumption that also is part of the issue pokt-network#1457.
Under high load of requests (1000/rps or more) the RAM got crazy and scale up to 40GB or close to that.
Now after the fix of pokt-network#1457 with the worker pool, the node remains under 14gb of ram in my local tests.
jorgecuesta added a commit to pokt-scan/pocket-core that referenced this issue Jul 10, 2023
@jorgecuesta
Fix pokt-network#1457; Fix pokt-network#1484;
c46904c 
Fix a high memory consumption that also is part of the issue pokt-network#1457.
Under high load of requests (1000/rps or more) the RAM got crazy and scale up to 40GB or close to that.
Now after the fix of pokt-network#1457 with the worker pool, the node remains under 14gb of ram in my local tests.
jorgecuesta added a commit to pokt-scan/pocket-core that referenced this issue Jul 10, 2023
@jorgecuesta
Fix pokt-network#1457; Fix pokt-network#1484;
6ecb307 
Fix a high memory consumption that also is part of the issue pokt-network#1457.
Under high load of requests (1000/rps or more) the RAM got crazy and scale up to 40GB or close to that.
Now after the fix of pokt-network#1457 with the worker pool, the node remains under 14gb of ram in my local tests.
nodiesBlade pushed a commit to pokt-scan/pocket-core that referenced this issue Jul 16, 2023
@jorgecuesta
Fix pokt-network#1457; Fix pokt-network#1484;
c560c34 
Fix a high memory consumption that also is part of the issue pokt-network#1457.
Under high load of requests (1000/rps or more) the RAM got crazy and scale up to 40GB or close to that.
Now after the fix of pokt-network#1457 with the worker pool, the node remains under 14gb of ram in my local tests.
jorgecuesta added a commit to pokt-scan/pocket-core that referenced this issue Jan 22, 2024
@jorgecuesta
Fix pokt-network#1457; Fix pokt-network#1484;
5cabe5c 
Fix a high memory consumption that also is part of the issue pokt-network#1457.
Under high load of requests (1000/rps or more) the RAM got crazy and scale up to 40GB or close to that.
Now after the fix of pokt-network#1457 with the worker pool, the node remains under 14gb of ram in my local tests.
jorgecuesta added a commit to pokt-scan/pocket-core that referenced this issue Jan 30, 2024
@jorgecuesta
Fix pokt-network#1457; Fix pokt-network#1484;
74b8dff 
Fix a high memory consumption that also is part of the issue pokt-network#1457.
Under high load of requests (1000/rps or more) the RAM got crazy and scale up to 40GB or close to that.
Now after the fix of pokt-network#1457 with the worker pool, the node remains under 14gb of ram in my local tests.
jorgecuesta added a commit to pokt-scan/pocket-core that referenced this issue Mar 6, 2024
@jorgecuesta
Fix pokt-network#1457; Fix pokt-network#1484;
63a6acf 
Fix a high memory consumption that also is part of the issue pokt-network#1457.
Under high load of requests (1000/rps or more) the RAM got crazy and scale up to 40GB or close to that.
Now after the fix of pokt-network#1457 with the worker pool, the node remains under 14gb of ram in my local tests.
jorgecuesta added a commit to pokt-scan/pocket-core that referenced this issue Mar 8, 2024
@jorgecuesta
Fix pokt-network#1457; Fix pokt-network#1484;
0c1cae6 
Fix a high memory consumption that also is part of the issue pokt-network#1457.
Under high load of requests (1000/rps or more) the RAM got crazy and scale up to 40GB or close to that.
Now after the fix of pokt-network#1457 with the worker pool, the node remains under 14gb of ram in my local tests.
jorgecuesta added a commit to pokt-scan/pocket-core that referenced this issue Mar 8, 2024
@jorgecuesta
Fix pokt-network#1457; Fix pokt-network#1484;
0970625 
Fix a high memory consumption that also is part of the issue pokt-network#1457.
Under high load of requests (1000/rps or more) the RAM got crazy and scale up to 40GB or close to that.
Now after the fix of pokt-network#1457 with the worker pool, the node remains under 14gb of ram in my local tests.
jorgecuesta added a commit to pokt-scan/pocket-core that referenced this issue May 2, 2024
@jorgecuesta
Fix pokt-network#1457; Fix pokt-network#1484;
49293f7 
Fix a high memory consumption that also is part of the issue pokt-network#1457.
Under high load of requests (1000/rps or more) the RAM got crazy and scale up to 40GB or close to that.
Now after the fix of pokt-network#1457 with the worker pool, the node remains under 14gb of ram in my local tests.
jorgecuesta added a commit to pokt-scan/pocket-core that referenced this issue Oct 9, 2024
@jorgecuesta
Fix pokt-network#1457; Fix pokt-network#1484;
63c9d45 
Fix a high memory consumption that also is part of the issue pokt-network#1457.
Under high load of requests (1000/rps or more) the RAM got crazy and scale up to 40GB or close to that.
Now after the fix of pokt-network#1457 with the worker pool, the node remains under 14gb of ram in my local tests.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@oten91 oten91

Labels
proposal Pocket Core client proposal
Projects
V0 Dashboard
Status: Done
Milestone
No milestone
Development

No branches or pull requests
2 participants
@oten91 @jorgecuesta