Skip to content

refactor: keep a thin PR eval pipeline that extends an external template - #144

Draft
Saurabh Ravindra Badenkal (saurabhrb) wants to merge 4 commits into
mainfrom
users/saurabhrb/remove-migrated-eval-pipelines
Draft

Saurabh Ravindra Badenkal (saurabhrb) wants to merge 4 commits into
mainfrom
users/saurabhrb/remove-migrated-eval-pipelines

Conversation

@saurabhrb

@saurabhrb Saurabh Ravindra Badenkal (saurabhrb) commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Moves the PR evaluation configuration out of this repository while keeping PR evaluations as a native pull request check.

Azure DevOps only runs native PR builds, and only posts a check that branch protection can require, for the repository that holds the pipeline file. So this repository keeps one small pipeline file. It declares when the evaluation runs and then extends a template maintained in the internal evaluation repository. The test corpus, agent model, runner version and secrets all live in that template, so changing the evaluation no longer needs a change here.

Changes

  • .azdo/DataversePluginEvals_PR.yml is reduced from 115 lines to a 29-line file:

    • trigger and pr on main;
    • one repository resource;
    • an extends pointing at the evaluation template.

    The pipeline keeps its name and the check it posts on pull requests.

  • The evaluation corpus under evals/tests/ is removed. It now lives with the evaluation template.

  • The warning workflow for changes to the pipeline file stays, because the file still exists and Azure DevOps may require re-approval after it changes.

Validation

  • python .github/evals/static_checks.py passes.
  • The new pipeline file was compiled by Azure DevOps in preview mode against the template. It compiled with no errors, and the expanded pipeline contains the same setup, prepare, execute and publish steps as before. This was compile-only; no evaluation ran.
  • The evaluation check on this PR is the end-to-end proof. It fails until the template is available on the evaluation repository's default branch, and should pass when re-run after that.

Notes for reviewers

  • After merge, the pipeline may need to be re-approved in Azure DevOps (the warning workflow posts that reminder).
  • Making this check a required status check on main is a follow-up repository-settings change.

Comment thread .azdo/DataversePluginEvals_PR.yml Resolved
@saurabhrb Saurabh Ravindra Badenkal (saurabhrb) changed the title refactor: remove migrated eval pipeline assets refactor: keep a thin PR eval pipeline that extends an external template Oct 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant