[DSIP-95][API] Complete the functionality of using dependencies in the complement data by det101 · Pull Request #18003 · apache/dolphinscheduler

det101 · 2026-02-27T08:17:24Z

Was this PR generated or assisted by AI?

close #17748

Purpose of the pull request

Brief change log

Verify this pull request

This pull request is code cleanup without any test coverage.

(or)

This pull request is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(or)

Pull Request Notice

If your pull request contains incompatible change, you should also add it to docs/docs/en/guide/upgrade/incompatible.md

Copilot

Pull request overview

Implements downstream workflow triggering for complement/backfill runs in the API layer, adding support for “trigger dependent workflows” behavior and accompanying unit tests.

Changes:

Implemented doBackfillDependentWorkflow to fetch downstream workflow definitions and trigger backfill runs for them.
Added visited-code tracking intended to prevent self/cyclic triggering and duplicate downstream triggers.
Added BackfillWorkflowExecutorDelegateTest with basic scenarios for downstream triggering and filtering.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/executor/workflow/BackfillWorkflowExecutorDelegate.java	Adds dependent workflow backfill triggering logic and wiring for lineage + workflow definition lookup.
dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/executor/workflow/BackfillWorkflowExecutorDelegateTest.java	Adds unit tests for the new dependent backfill triggering logic (single-hop scenarios).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

...java/org/apache/dolphinscheduler/api/executor/workflow/BackfillWorkflowExecutorDelegate.java

.../org/apache/dolphinscheduler/api/executor/workflow/BackfillWorkflowExecutorDelegateTest.java

...java/org/apache/dolphinscheduler/api/executor/workflow/BackfillWorkflowExecutorDelegate.java

SbloodyS

You didn't fill in the content according to the PR template, please fix it.

...java/org/apache/dolphinscheduler/api/executor/workflow/BackfillWorkflowExecutorDelegate.java

sonarqubecloud · 2026-03-23T04:15:39Z

Quality Gate failed

Failed conditions
0.0% Coverage on New Code (required ≥ 60%)

See analysis details on SonarQube Cloud

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-24T03:25:49Z

.../org/apache/dolphinscheduler/api/executor/workflow/BackfillWorkflowExecutorDelegateTest.java

+        BackfillWorkflowDTO.BackfillParamsDTO params = BackfillWorkflowDTO.BackfillParamsDTO.builder()
+                .runMode(RunMode.RUN_MODE_SERIAL)
+                .backfillDateList(Collections.<ZonedDateTime>emptyList())
+                .backfillDependentMode(ComplementDependentMode.ALL_DEPENDENT)
+                .allLevelDependent(true)
+                .executionOrder(ExecutionOrder.ASC_ORDER)
+                .build();
+


The new dependent-triggering behavior isn’t covered for RunMode.RUN_MODE_PARALLEL. All tests build params with RUN_MODE_SERIAL, so regressions around splitting dates / triggering dependencies per chunk (and interaction with visitedCodes) won’t be caught. Adding a test that uses RUN_MODE_PARALLEL with multiple backfill dates and asserts downstream workflows are triggered for each split would improve coverage.

Copilot · 2026-03-24T03:25:50Z

...java/org/apache/dolphinscheduler/api/executor/workflow/BackfillWorkflowExecutorDelegate.java

        for (List<ZonedDateTime> stringDate : splitDateTime(listDate, expectedParallelismNumber)) {
            final Integer workflowInstanceId = doBackfillWorkflow(
                    backfillWorkflowDTO,
-                    stringDate.stream().map(DateUtils::dateToString).collect(Collectors.toList()));
+                    stringDate.stream().map(DateUtils::dateToString).collect(Collectors.toList()),
+                    visitedCodes);
            workflowInstanceIdList.add(workflowInstanceId);


In parallel backfill mode, the same mutable visitedCodes instance is passed into every doBackfillWorkflow(...) call. Since visitedCodes is also used to prevent duplicate/cyclic dependent triggers, the first chunk will add downstream codes and later chunks will skip triggering downstream workflows, causing dependent backfill to only run for the first subset of dates. Consider scoping visitedCodes to a single dependency traversal per backfill chunk (e.g., pass null or a fresh copy like new HashSet<>(visitedCodes) for each split) so downstream workflows are triggered for each chunk’s backfillTimeList.

Copilot · 2026-03-24T03:25:50Z

...java/org/apache/dolphinscheduler/api/executor/workflow/BackfillWorkflowExecutorDelegate.java

+        List<DependentWorkflowDefinition> downstreamDefinitions =
+                workflowLineageService.queryDownstreamDependentWorkflowDefinitions(upstreamWorkflowCode);
+
+        if (downstreamDefinitions == null || downstreamDefinitions.isEmpty()) {
+            log.info("No downstream dependent workflows found for workflow code {}", upstreamWorkflowCode);
+            return;
+        }
+
+        // 2) Convert upstream backfill time from string to ZonedDateTime as the base business dates for downstream
+        // backfill
+        final List<ZonedDateTime> upstreamBackfillDates = backfillTimeList.stream()
+                .map(DateUtils::stringToZoneDateTime)
+                .collect(Collectors.toList());
+
+        // 3) Iterate downstream workflows and build/trigger corresponding BackfillWorkflowDTO
+        for (DependentWorkflowDefinition dependentWorkflowDefinition : downstreamDefinitions) {
+            long downstreamCode = dependentWorkflowDefinition.getWorkflowDefinitionCode();
+
+            // Prevent self-dependency and circular dependency chains
+            if (visitedCodes.contains(downstreamCode)) {
+                log.warn("Skip circular dependent workflow {}", downstreamCode);
+                continue;
+            }
+
+            WorkflowDefinition downstreamWorkflow =
+                    workflowDefinitionDao.queryByCode(downstreamCode).orElse(null);
+            if (downstreamWorkflow == null) {
+                log.warn("Skip dependent workflow {}, workflow definition not found", downstreamCode);
+                continue;


doBackfillDependentWorkflow queries each downstream workflow definition one-by-one via workflowDefinitionDao.queryByCode(downstreamCode), which will result in an N+1 query pattern for workflows with many downstream dependencies. Since WorkflowDefinitionDao#queryByCodes(Collection<Long>) exists, consider collecting downstream codes first and fetching definitions in a single batch, then iterating in-memory (still applying the ONLINE filter and visited/cycle checks).

…endent workflow - Problem 1: Replace single-level self-dep check with ThreadLocal visited set to detect and skip indirect circular dependencies (A→B→A), preventing StackOverflowError when allLevelDependent=true - Problem 2: Set startNodes=null for downstream workflows; upstream task node codes are not valid in a different workflow definition - Add tests for OFFLINE skip, not-found skip, and startNodes null assertion Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…tion ThreadLocal is unnecessary here since the call chain is synchronous and private. Passing visitedCodes as a parameter is simpler, clearer, and avoids ThreadLocal lifecycle management. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Align Java source and test formatting with project spotless rules to keep style checks consistent. Made-with: Cursor

det101 requested review from SbloodyS and caishunfeng as code owners February 27, 2026 08:17

det101 force-pushed the DSIP-95 branch from 91a1353 to a3fe99a Compare February 27, 2026 08:18

SbloodyS requested a review from Copilot March 20, 2026 03:50

Copilot started reviewing on behalf of SbloodyS March 20, 2026 03:51 View session

SbloodyS assigned det101 Mar 20, 2026

SbloodyS added backend DSIP labels Mar 20, 2026

SbloodyS added this to the 3.4.2 milestone Mar 20, 2026

github-actions bot added the test label Mar 20, 2026

Copilot AI reviewed Mar 20, 2026

View reviewed changes

SbloodyS reviewed Mar 20, 2026

View reviewed changes

...java/org/apache/dolphinscheduler/api/executor/workflow/BackfillWorkflowExecutorDelegate.java Outdated Show resolved Hide resolved

SbloodyS closed this Mar 23, 2026

SbloodyS reopened this Mar 23, 2026

det101 requested review from SbloodyS and Copilot March 24, 2026 03:20

Copilot started reviewing on behalf of det101 March 24, 2026 03:21 View session

Copilot AI reviewed Mar 24, 2026

View reviewed changes

luxl and others added 5 commits March 24, 2026 14:57

close apache#17748

4c6e003

[Chore] Apply spotless formatting for backfill workflow executor files

7caba0a

Align Java source and test formatting with project spotless rules to keep style checks consistent. Made-with: Cursor

fix review

b9ef592

det101 force-pushed the DSIP-95 branch from 11c9f03 to b9ef592 Compare March 24, 2026 06:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DSIP-95][API] Complete the functionality of using dependencies in the complement data#18003

[DSIP-95][API] Complete the functionality of using dependencies in the complement data#18003
det101 wants to merge 5 commits intoapache:devfrom
det101:DSIP-95

det101 commented Feb 27, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SbloodyS left a comment

Uh oh!

Uh oh!

sonarqubecloud bot commented Mar 23, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Mar 24, 2026

Uh oh!

Copilot AI Mar 24, 2026

Uh oh!

Copilot AI Mar 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

det101 commented Feb 27, 2026

Was this PR generated or assisted by AI?

Purpose of the pull request

Brief change log

Verify this pull request

Pull Request Notice

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SbloodyS left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sonarqubecloud bot commented Mar 23, 2026

Quality Gate failed

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants