Expand procedure architecture for distributed execution, and support iceberg procedure rewrite_data_files

[hantangwangd]

Description

This PR expand the current procedure architecture in presto, support defining, registering and calling procedures which need to be executed in a distributed way. Then support distributed procedure in Iceberg connector and implement a specific procedure rewrite_data_files for it.

Referring to: prestodb/rfcs#12

The whole PR is separated into 6 parts:

1. Re-factor ProcedureRegistry/Procedure data structure to support the creation and register of DistributedProcedure. And make sure ProcedureRegistry be available in presto-analyzer module, so that we can recognize distributed procedures in call statement during prepare and analyze stages.

2. Handle call statement on distributed procedures in preparer stage. In this stage, we figure out the procedure's type in call statement, and define a new query type CALL_DISTRIBUTED_PROCEDURE for call distributed procedure in BuiltInPreparedQuery. In this way, call distributed procedure statement would be handled by SqlQueryExecutionFactory, then be created and handled as a SqlQueryExecution.

3. Analyze and plan the call distributed procedure statement, and finally generate a logical plan for it as follows:

OutputNode <- TableFinishNode <- CallDistributedProcedureNode <- FilterNode <- TableScanNode

4. Optimize, segmentation, grouped tag and local plan for the logical plan generated above. The handle logical for CallDistributedProcedureNode is similar as TableWriterNode. Besides, a new optimizer RewriteWriterTarget is added, which is placed after all optimization rules. It is used to update the TableHandle held in TableFinishNode and CallDistributedProcedureNode based on the underlying TableScanNode after the entire optimization is completed, considering the possible filter pushing down.

5. Re-factor Iceberg connector to support call distributed procedure. Introduce Iceberg's transaction context and expand IcebergSplitManager to support split source planned by IcebergAbstractMetadata.beginCallDistributedProcedure(...). This split source will be set to transaction context, and use transaction context to hold all the files to be rewritten as well.

6. Support Iceberg rewrite_data_files procedure. It build a customized split source, set the split source to transaction context in order to be used in IcebergSplitManager. And register a file scan task consumer to collector and hold all the scanned files into transaction context. Then finally in the commit stage, get all the data files and delete files that has been rewritten, and all the files that has been newly generated, change and commit their metadata through Iceberg table's RewriteFiles transaction.

Motivation and Context

N/A

Impact

N/A

Test Plan

N/A

Contributor checklist

• Please make sure your submission complies with our development, formatting, commit message, and attribution guidelines.
• PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced.
• Documented new properties (with its default value), SQL syntax, functions, or other functionality.
• If release notes are required, they follow the release notes guidelines.
• Adequate tests were added if applicable.
• CI passed.

Release Notes

== NO RELEASE NOTE ==

[steveburnett]

Thanks for the draft doc! Some nits about punctuation, formatting, and some suggested rephrasing for readability and conciseness, but the content looks good.

[hantangwangd]

@steveburnett Thanks a lot for your suggestion, all be fixed. Please take a look when convenient!

[steveburnett]

LGTM! (docs)

Pull updated branch, new local doc build, looks good. Thanks!