Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CORE] Register native shuffles through backend API #8060

Open
zhztheplayer opened this issue Nov 27, 2024 · 0 comments
Open

[CORE] Register native shuffles through backend API #8060

zhztheplayer opened this issue Nov 27, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@zhztheplayer
Copy link
Member

zhztheplayer commented Nov 27, 2024

Part of #6920.

So far Gluten's native shuffle implementations, including VL, CH, VL + RSS are activated by specifying corresponding shuffle manager types in Spark configuration.

However if we want to mix different native shuffles in the same Spark session even in the same query plan, as Spark only allows one shuffle manager to be registered (see also a related SPIP) yet, we should enhance out current approach by providing a registry of native shuffles internally under an umbrella shuffle manager. Then query planner could choose an optimal one among all the candidates.

@zhztheplayer zhztheplayer added the enhancement New feature or request label Nov 27, 2024
@zhztheplayer zhztheplayer changed the title [CORE] Register native shuffle through backend API [CORE] Register native shuffles through backend API Nov 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant