Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve report performance with high-cardinality import joins #4848

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open

Improve report performance with high-cardinality import joins #4848

wants to merge 4 commits into from
+158 −176

Commits on Nov 26, 2024

  1. Improve report performance in cases where site has a lot of unique pa… 

    …thnames

Ref: https://3.basecamp.com/5308029/buckets/39750953/card_tables/cards/8052057081

JOINs in ClickHouse are slow. In one degenerate case I found a user had
over 20 million unique paths in an import, which resulted in extremely slow
JOINs. This introduces a sort-of hacky solution to it by limiting the
amount of data analyzed.

Query timing without this change:
```
9 rows in set. Elapsed: 11.383 sec. Processed 49.16 million rows, 5.75 GB (4.32 million rows/s., 505.29 MB/s.)
Peak memory usage: 14.75 GiB.
```

After:
```
9 rows in set. Elapsed: 0.572 sec. Processed 49.18 million rows, 5.75 GB (86.03 million rows/s., 10.06 GB/s.)
Peak memory usage: 9.01 GiB.
```
    @macobo
    macobo committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    935a2a4 View commit details
    Browse the repository at this point in the history

  2. Splitting should no longer remove pagination. Handle special cases in… 

    … special_metrics.ex
    @macobo
    macobo committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    71ef817 View commit details
    Browse the repository at this point in the history

  3. select_merge_as in imports 

    This sets up selected_as aliases which will be used in a subsequent commit
    @macobo
    macobo committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    b6fb241 View commit details
    Browse the repository at this point in the history

  4. Add explicit ORDER BY to import

    @macobo
    macobo committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    2afbef2 View commit details
    Browse the repository at this point in the history