Skip to content

[FLINK-40331][cdc-connect][fluss] Support forward shuffle in fluss sink. - #4498

Open
loserwang1024 wants to merge 1 commit into
apache:masterfrom
loserwang1024:forward-hash
Open

[FLINK-40331][cdc-connect][fluss] Support forward shuffle in fluss sink.#4498
loserwang1024 wants to merge 1 commit into
apache:masterfrom
loserwang1024:forward-hash

Conversation

@loserwang1024

Copy link
Copy Markdown
Contributor

What is the purpose of this pull request?

https://issues.apache.org/jira/browse/FLINK-40331

Brief change log

Add sink.partitioning.strategy in fluss sink to support FORWARD.


Verifying this change

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change added tests and can be verified as follows:

  • Added/Updated unit tests in ...*
  • Added/Updated integration tests in ...*
  • Manually tested by ...

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

@github-actions github-actions Bot added docs Improvements or additions to documentation common runtime labels Aug 5, 2026
* <p>Implementations that do not depend on the source subtask index can continue implementing
* {@link #hashcode(Object)} only.
*/
default int hashcode(int sourceIndex, T event) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @loserwang1024, thanks for the PR! One suggestion on the HashFunction change: instead of a raw int sourceIndex, consider passing a small immutable HashContext object:

@Internal
public class HashContext {

    private final int sourceSubtaskIndex;
    private final int downstreamParallelism;

    public HashContext(int sourceSubtaskIndex, int downstreamParallelism) { ... }

    public int getSourceSubtaskIndex() { ... }

    public int getDownstreamParallelism() { ... }
}
default int hashcode(HashContext context, T event) {
    return hashcode(event);
}

Reasons:

  1. Self-documenting call siteshashcode(subTaskId, event) gives no hint what the bare int means; hashcode(hashContext, event) does.
  2. Extensible without signature churn — future strategies (bucket-aware mapping, fallback when sourceIndex >= downstreamParallelism) can read from the context instead of stacking more int overloads.
  3. Fails fast instead of silently skewingForwardHashFunction.hashcode(event) can throw UnsupportedOperationException rather than returning 0 and routing everything to subtask 0 if someone calls the single-arg method by mistake.

The context can be created once in open() and reused, so there's no per-record allocation cost. Since HashFunction is @Internal, this is the cheapest time to make the change. WDYT?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

common docs Improvements or additions to documentation runtime

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants