Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Supports apache-celeborn #287

Closed
richox opened this issue Nov 3, 2023 · 2 comments · Fixed by #596 or #615
Closed

Supports apache-celeborn #287

richox opened this issue Nov 3, 2023 · 2 comments · Fixed by #596 or #615
Labels
feature required Functionalities must have help wanted Extra attention is needed

Comments

@richox
Copy link
Collaborator

richox commented Nov 3, 2023

Is your feature request related to a problem? Please describe.
apache-celeborn is widely used as an RSS (remote shuffle service) for spark. currently blaze has supported an internal RSS implementation from Kuaishou.Inc, however celeborn is not yet supported.

Describe the solution you'd like
integrate blaze with celeborn, this should be done by extending rss-xxx-repartitioner.rs. furthermore, we can support an individual rss shim module for supporting different RSS implementations.

Describe alternatives you've considered

Additional context

@richox richox added the feature required Functionalities must have label Nov 3, 2023
@zuston
Copy link
Contributor

zuston commented Nov 3, 2023

Uniffle is another candidate. Do you mind integrating uniffle with blaze?

@richox
Copy link
Collaborator Author

richox commented Nov 6, 2023

Uniffle is another candidate. Do you mind integrating uniffle with blaze?

i suggest we create a abstract layer for integrating different rss implementations. of course we can have one working celeborn/uniffle module, then we can have more idea on designing the abstract layer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature required Functionalities must have help wanted Extra attention is needed
Projects
None yet
2 participants