ddl_puller.go(ticdc): fix DDLs are ignored when schema versions are out of order

[wlwilliamx]

What problem does this PR solve?

Issue Number: close #11714

What is changed and how it works?

Description:
This pull request addresses a bug in TiCDC where a DDL job could be inadvertently dropped if a subsequent DDL job with a higher SchemaVersion but lower CommitTs is processed first. This behavior occurs because the ddlJobPuller processes DDL jobs based on their CommitTs in ascending order and uses a check against SchemaVersion that can prevent older DDL jobs from being processed, leading to data inconsistencies.

Background:
The issue manifests when two DDL jobs are processed:

• Job 60: ALTER TABLE a ADD COLUMN x, y, z with CommitTs 400
• Job 62: ALTER TABLE b ADD COLUMN y with CommitTs 300

In this scenario, Job 62 is processed first due to its lower CommitTs. The ddlJobPuller then updates its SchemaVersion to that of Job 62. When Job 60 is subsequently processed, the current logic discards it because its SchemaVersion is deemed older, even though its CommitTs is higher.

This issue occurs because the SchemaVersion increment and job metadata write to TiKV are separate transactions. During a TiDB owner change, different instances might write these transactions without synchronization, leading to potential out-of-order CommitTs relative to SchemaVersion.

Solution:
To resolve this, the check on SchemaVersion is removed, ensuring that only the CommitTs is verified:

Updated Code:

if job.BinlogInfo.FinishedTS <= p.getResolvedTs() {
    log.Info("ddl job finishedTs less than puller resolvedTs," +
        "discard the ddl job",
        zap.String("namespace", p.changefeedID.Namespace),
        zap.String("changefeed", p.changefeedID.ID),
        zap.String("schema", job.SchemaName),
        zap.String("table", job.TableName),
        zap.Uint64("startTs", job.StartTS),
        zap.Uint64("finishedTs", job.BinlogInfo.FinishedTS),
        zap.String("query", job.Query),
        zap.Uint64("pullerResolvedTs", p.getResolvedTs()))
    return true, nil
}

Reasoning:
The ResolvedTs check ensures that only DDL jobs with FinishedTS greater than the current ResolvedTs are processed. Since the ddlJobPuller receives DDLs sorted by CommitTs, any new DDL received with a FinishedTS greater than ResolvedTs must be handled, making the SchemaVersion check redundant.

Check List

Tests

• Unit test
• Integration test

Questions

Will it cause performance regression or break compatibility?

No.

Do you need to update user documentation, design documentation or monitoring documentation?

No.

Release note

Fix the issue that DDL jobs could be incorrectly discarded in TiCDC when their schema versions were not in a strictly linear order with commit timestamps due to TiDB owner changes.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 55.1547%. Comparing base (1da37a2) to head (a45ec46).
Report is 5 commits behind head on master.

Additional details and impacted files

Components	Coverage Δ
cdc	59.5685% <100.0000%> (-0.0644%)	⬇️
dm	50.0577% <ø> (+0.0134%)	⬆️
engine	53.2336% <ø> (+0.0056%)	⬆️

Flag	Coverage Δ
unit	55.1547% <100.0000%> (+0.0025%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

@@               Coverage Diff                @@
##             master     #11733        +/-   ##
================================================
+ Coverage   55.1522%   55.1547%   +0.0025%     
================================================
  Files          1001       1002         +1     
  Lines        136524     137397       +873     
================================================
+ Hits          75296      75781       +485     
- Misses        55718      56065       +347     
- Partials       5510       5551        +41     

[wlwilliamx]

/retest

[3AceShowHand]

can you add some unit test to cover this ?

[wlwilliamx]

can you add some unit test to cover this ?

This issue is difficult to reproduce and occurs in rare cases, making it hard to mock.

[wlwilliamx]

/hold

[wlwilliamx]

/unhold

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: asddongmen, lidezhu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [asddongmen,lidezhu]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2024-11-15 03:45:33.061569071 +0000 UTC m=+587095.252438069: ☑️ agreed by lidezhu.
• 2024-11-15 08:34:12.029160759 +0000 UTC m=+604414.220029747: ☑️ agreed by asddongmen.

[wlwilliamx]

/retest

[ti-chi-bot]

In response to a cherrypick label: new pull request created to branch release-8.5: #11755.

[ti-chi-bot]

In response to a cherrypick label: new pull request created to branch release-6.5: #11756.

[ti-chi-bot]

In response to a cherrypick label: new pull request created to branch release-7.1: #11757.

[ti-chi-bot]

In response to a cherrypick label: new pull request created to branch release-7.5: #11758.

[ti-chi-bot]

In response to a cherrypick label: new pull request created to branch release-8.1: #11759.