feat: support alter twcs compression options #4939

lyang24 · 2024-11-05T08:55:56Z

Refer to a related PR or issue link (optional)

What's changed and what's your intention?

support alter twcs compaction options on sql level

!!! DO NOT LEAVE THIS BLOCK EMPTY !!!

Please explain IN DETAIL what the changes are in this PR and why they are needed:
expand alter table set options to include twcs compaction options.

Summarize your change (mandatory)
The approach here is to expand ChangeOption enum with twcs compaction options.
The key and value type is validated at store-api end.
If value is missing (empty string) it will be set with default value.
Changes will be reflected on show create table statements.
How does this PR work? Need a brief introduction for the changed logic (optional)
Describe clearly one logical change and avoid lazy messages (optional)
Describe any limitations of the current code (optional)

Checklist

I have written the necessary rustdoc comments.
I have added the necessary unit tests and integration tests.
This PR requires documentation updates.

coderabbitai · 2024-11-05T08:56:03Z

Important

Review skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>, please review it.
- Generate unit testing code for this file.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai generate unit testing code for this file.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
- @coderabbitai read src/utils.ts and generate unit testing code.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
- @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Other keywords and placeholders

Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
Add @coderabbitai anywhere in the PR title to generate the title automatically.

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

src/table/src/metadata.rs

codecov · 2024-11-07T07:36:24Z

Codecov Report

Attention: Patch coverage is 13.10345% with 126 lines in your changes missing coverage. Please review.

Project coverage is 83.78%. Comparing base (22f31f5) to head (42db982).
Report is 4 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4939      +/-   ##
==========================================
- Coverage   84.05%   83.78%   -0.27%     
==========================================
  Files        1142     1143       +1     
  Lines      211545   212050     +505     
==========================================
- Hits       177804   177667     -137     
- Misses      33741    34383     +642

v0y4g3r · 2024-11-08T01:12:00Z

src/store-api/src/region_request.rs

+                value: humantime::format_duration(*duration).to_string(),
+            },
+            ChangeOption::TwscMaxActiveWindowRuns(runs) => ChangeTableOption {
+                key: "compaction.twcs.max_active_window_runs".to_string(),


Can we extract these string literals to consts? They appear multiple times in codebase.

yep its a good idea

v0y4g3r · 2024-11-08T01:13:34Z

src/store-api/src/region_request.rs

+                }),
+            },
+            ChangeOption::TwscRemoteCompaction(flag) => ChangeTableOption {
+                key: "compaction.twcs.remote_compaction".to_string(),


Not sure if this option can be modified dynamically? @zyy

I see i left this option out in this new update

src/store-api/src/region_request.rs

v0y4g3r · 2024-11-08T01:26:33Z

src/mito2/src/worker/handle_alter.rs

+                    let Twcs(options) = current_options.compaction;
+                    let runs = runs.unwrap_or(TwcsOptions::default().max_active_window_runs);
+                    info!(
+                        "Update region compaction.twcs.max_active_window_runs: {}, previous: {} new: {}",
+                        region.region_id, options.max_active_window_runs, runs
+                    );
+                    let mut new_option = options.clone();
+                    new_option.max_active_window_runs = runs;
+                    current_options.compaction = Twcs(new_option)


Can we avoid the clone here?

Suggested change

let Twcs(options) = current_options.compaction;

let runs = runs.unwrap_or(TwcsOptions::default().max_active_window_runs);

info!(

"Update region compaction.twcs.max_active_window_runs: {}, previous: {} new: {}",

region.region_id, options.max_active_window_runs, runs

);

let mut new_option = options.clone();

new_option.max_active_window_runs = runs;

current_options.compaction = Twcs(new_option)

let Twcs(options) = &mut current_options.compaction;

let runs = runs.unwrap_or(TwcsOptions::default().max_active_window_runs);

info!(

"Update region compaction.twcs.max_active_window_runs: {}, previous: {} new: {}",

region.region_id, options.max_active_window_runs, runs

);

options.max_active_window_runs = runs;

good call fixed it

killme2008 · 2024-11-08T08:26:38Z

src/store-api/src/region_request.rs

@@ -661,23 +667,126 @@ impl From<v1::ChangeColumnType> for ChangeColumnType {
 #[derive(Debug, Eq, PartialEq, Clone, Serialize, Deserialize)]
 pub enum ChangeOption {
    TTL(Duration),
+    TwscMaxActiveWindowRuns(Option<usize>),


What about changing these variants into one

Twcs(String)

The String value is the option name such as compaction.twcs.max_active_window_runs etc.

I think it looks much better and we can add new options in the future without adding new variant of ChangeOption.

I think the draw back would be that we cannot validate the type of value at storage api layer - i.e user can pass max_active_window_runs='abcde' and this will error out at mito engine code path (where we handle the alter options).

I think the draw back would be that we cannot validate the type of value at storage api layer - i.e user can pass max_active_window_runs='abcde' and this will error out at mito engine code path (where we handle the alter options).

We can match the option to verify its validity, as the next comment explains.

ah i see great call

I took a try at implement this approach, I had a few concerns:

we also need the value of the change, the enum would look like Twcs(String, String)

the second comments suggest that we match and validate values at Mito side when handling the alter request which will cause some drift in logics. (i.e. ttl change values are parsed and validated on building the request and it will return a metadata error - InvalidRegionOptionChangeRequest).

This is a draft of the working attempt https://github.com/GreptimeTeam/greptimedb/pull/4965/files if you like it i can push to this branch.

killme2008 · 2024-11-08T08:28:19Z

src/mito2/src/worker/handle_alter.rs

@@ -167,6 +169,67 @@ impl<S> RegionWorkerLoop<S> {
                        current_options.ttl = Some(new_ttl);
                    }
                }
+                ChangeOption::TwscMaxActiveWindowRuns(runs) => {


If we change all these variants into Twcs(String), then we can refactor these match clauses into:

ChangeOption::Twcs(options) => { match options.as_str() => { "compaction.twcs.max_active_window_runs" => { .. } ...... } }

lyang24 · 2024-11-11T04:44:29Z

the new approach look cleaner going to close this one in favor of
#4965

github-actions bot added the docs-not-required This change does not impact docs. label Nov 5, 2024

lyang24 commented Nov 5, 2024

View reviewed changes

src/table/src/metadata.rs Outdated Show resolved Hide resolved

lyang24 changed the title ~~support alter twcs compression options~~ feat: support alter twcs compression options Nov 5, 2024

lyang24 force-pushed the alter_compression branch 2 times, most recently from 0ff73fa to 1e58c0c Compare November 7, 2024 07:06

github-actions bot added docs-required This change requires docs update. and removed docs-not-required This change does not impact docs. labels Nov 7, 2024

lyang24 force-pushed the alter_compression branch from 1e58c0c to e146d81 Compare November 7, 2024 07:19

github-actions bot added docs-not-required This change does not impact docs. and removed docs-required This change requires docs update. labels Nov 7, 2024

lyang24 marked this pull request as ready for review November 7, 2024 07:20

lyang24 requested review from v0y4g3r, evenyag, waynexia and a team as code owners November 7, 2024 07:20

v0y4g3r reviewed Nov 8, 2024

View reviewed changes

lyang24 force-pushed the alter_compression branch from e146d81 to 39f64b6 Compare November 8, 2024 07:57

support alter twcs compression options

42db982

lyang24 force-pushed the alter_compression branch from 39f64b6 to 42db982 Compare November 8, 2024 08:22

lyang24 requested a review from v0y4g3r November 8, 2024 08:23

killme2008 reviewed Nov 8, 2024

View reviewed changes

lyang24 closed this Nov 11, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: support alter twcs compression options #4939

feat: support alter twcs compression options #4939

lyang24 commented Nov 5, 2024 •

edited

Loading

coderabbitai bot commented Nov 5, 2024

Review skipped

Chat

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

Documentation and Community

codecov bot commented Nov 7, 2024 •

edited

Loading

v0y4g3r Nov 8, 2024

lyang24 Nov 8, 2024

v0y4g3r Nov 8, 2024

lyang24 Nov 8, 2024

v0y4g3r Nov 8, 2024

lyang24 Nov 8, 2024

killme2008 Nov 8, 2024

lyang24 Nov 8, 2024 •

edited

Loading

killme2008 Nov 8, 2024

lyang24 Nov 8, 2024

lyang24 Nov 9, 2024 •

edited

Loading

killme2008 Nov 8, 2024

lyang24 commented Nov 11, 2024

feat: support alter twcs compression options #4939

feat: support alter twcs compression options #4939

Conversation

lyang24 commented Nov 5, 2024 • edited Loading

Refer to a related PR or issue link (optional)

What's changed and what's your intention?

Checklist

coderabbitai bot commented Nov 5, 2024

Review skipped

Chat

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

Documentation and Community

codecov bot commented Nov 7, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lyang24 Nov 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lyang24 Nov 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lyang24 commented Nov 11, 2024

lyang24 commented Nov 5, 2024 •

edited

Loading

codecov bot commented Nov 7, 2024 •

edited

Loading

lyang24 Nov 8, 2024 •

edited

Loading

lyang24 Nov 9, 2024 •

edited

Loading