Skip to content

fix: skill guidance and routing behind recurring live-eval failures - #155

Open
Saurabh Ravindra Badenkal (saurabhrb) wants to merge 13 commits into
mainfrom
users/saurabhrb/dv-data-lookup-association
Open

Saurabh Ravindra Badenkal (saurabhrb) wants to merge 13 commits into
mainfrom
users/saurabhrb/dv-data-lookup-association

Conversation

@saurabhrb

@saurabhrb Saurabh Ravindra Badenkal (saurabhrb) commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Skill guidance and routing fixes for failures that recur in the live evaluations. Each change traces a repeated failure to skill guidance, not to the platform. This PR now also includes the changes from #158 (closed in favor of this one) so they are reviewed and merged together.

Why

Recurring failures in the live evaluations, with the cause in skill guidance:

  1. Currency columns created as Decimal (metadata_001_table_and_simple_columns). dv-metadata listed "money" as a supported type and used it in an example, but the SDK maps "money" to DecimalAttributeMetadata. Org verification then fails with "expected column exma_estimatedcost with type Money, got ... Decimal". This was still the most common run failure in the first week of November.
  2. Record association through raw HTTP (data_008_associate_contact). dv-data listed association as an SDK gap with a raw POST .../$ref alternative. The agent guessed a client.records.associate method, got AttributeError, then fell back to the SDK's private _odata helper or a hand-built requests POST, which the grader marks non-compliant.
  3. Skill not loaded. connect_001 / connect_002 (verify an existing connection or that DATAVERSE_URL matches the live org) answered correctly but never loaded dv-connect: dv-overview sends "connect" and "set up" requests to dv-connect but tells the agent to skip setup when auth already works, and "verify the connection" matched only the skip path. overview_001 / overview_003 ask which skills to use without saying "Dataverse", so dv-overview did not load; on overview_003 the agent also skipped the connect and solution steps.
  4. Activity state and partial re-runs (data_005_support_system). The agent created a phone call already closed with a status reason from another state, then re-ran the whole script after the failure and duplicated a company.
  5. dv-admin OrgDB reads. A request for named OrgDB settings could leave out a setting that was never written.
  6. ERP routing (PR-gate dry-run suites). erp_data_001_update_contract, erp_admin_001_list_running and erp_metadata_001_runtime_schema answered from templates in dv-overview's references/erp-target.md without loading the owning skill.

Changes

dv-data

  • No raw $ref; states there is no client.records.associate; new "Link existing records (lookup)" example with client.records.update + parentcustomerid_account@odata.bind (CLI: dataverse data associate).
  • New references/activities.md: bind the regarding row with the per-table navigation property, create the activity open, then close it with one update that sets a matching statecode + statuscode pair.
  • references/multi-table-fk-import.md: if a create script fails partway, look up what already exists and create only what is missing instead of re-running.

dv-metadata

  • "money" documented as a "decimal" alias; the constraint example uses "decimal".
  • New references/currency-columns.md: create a real MoneyAttributeMetadata column with dataverse api request, including the solution header and a verify step.

dv-connect / dv-overview

  • dv-connect description: verifying which environment or identity is connected (WhoAmI, DATAVERSE_URL match).
  • dv-overview description: requests for which skill, tool, or sequence of steps to use; "verify connection" requests load dv-connect.
  • dv-overview change lifecycle: states the skill sequence for "new table to source control" (dv-connect -> dv-solution publisher and solution -> dv-metadata -> dv-solution export and unpack). The duplicated publisher/solution snippet is shortened to a pointer (dv-solution owns the full pattern) to stay under the 5,000-token body cap.
  • dv-overview/references/erp-target.md is a router again: reads and runtime schema go to dv-query, writes including previews to dv-data, batch jobs to dv-admin. Every removed template still exists in those references. Also corrects "DMF (tier 5 above)" to tier 6.

dv-admin / dv-query

  • dv-admin/references/orgdb-settings.md: "read specific OrgDB settings" example that checks requested keys against the documented allowlist, reads only those keys, reports an absent key as not set (platform default), and never writes back.
  • dv-admin description names Finance and Operations / F&O for batch jobs; dv-query skill boundaries send ERP writes to dv-data and batch jobs to dv-admin.

Version 1.17.0 -> 1.18.0 across all manifests (minor: new references and trigger keywords).

Testing

  • python .github/evals/static_checks.py: passed (9 skill files, 51 Python blocks); all bodies stay under the 5,000-token cap.
  • python .github/evals/version_bump_check.py: passed.

Live validation

The combined changes were run on a proof branch (users/saurabhrb/evalarm-skills-155-158) against a live test environment on 10/17:

  • 45 of 46 live tests passed and none failed; 604/604 P1 assertions and all 50 org-state checks passed. connect_002 produced no output in the evaluation harness (infrastructure), so it was not graded in that run.
  • metadata_001 created a real Money column; data_008, connect_001 and overview_003 passed; data_005 scored effort 5/5 (3/5 before) with exactly 3 companies.
  • dv-admin 5/5 tests passed; the OrgDB settings read scored 5/5 on every grader.
  • PR-gate dry run: 8/8 suites green, 15/15 tests; each ERP test now loads its owning skill.

The dv-overview skill-sequence line was added after that run; the live dry-run gate on this PR covers it.

Saurabh Badenkal added 3 commits October 14, 2026 16:14
The skill listed record association as an SDK gap with a raw POST to $ref as the alternative, which led the live eval agent to guess a non-existent client.records.associate and then hand-build the HTTP request. Point lookups at client.records.update with @odata.bind or dataverse data associate. Bump to 1.16.2.
The SDK maps 'money' to DecimalAttributeMetadata, but dv-metadata listed it as a supported type and used it in an example, so agents created Decimal columns when asked for currency. Clarify the alias and add a reference for creating a real MoneyAttributeMetadata column with dataverse api request.
@saurabhrb Saurabh Ravindra Badenkal (saurabhrb) changed the title fix: dv-data lookup association uses managed surfaces, not raw $ref fix: dv-data lookup association and dv-metadata currency column guidance Oct 15, 2026
Live evals skipped loading dv-connect for 'verify DATAVERSE_URL / WhoAmI' requests and dv-overview for 'which skill sequence' planning prompts that never name Dataverse. Name both request shapes in the descriptions. Bump to 1.17.0 (new trigger keywords).
@saurabhrb Saurabh Ravindra Badenkal (saurabhrb) changed the title fix: dv-data lookup association and dv-metadata currency column guidance fix: dv-data/dv-metadata guidance and dv-connect/dv-overview triggers from live-eval failures Oct 15, 2026
@saurabhrb
Saurabh Ravindra Badenkal (saurabhrb) marked this pull request as ready for review October 15, 2026 02:59
…ead of re-running it

Live evals showed the agent trying to create a phone call that was already
closed, pairing a status reason with the wrong state, and then re-running
its whole create script after the failure. The re-run left a duplicate
company in the environment.

- New references/activities.md: bind the regarding row with the per-table
  navigation property (regardingobjectid_<regarding>_<activity>), create the
  activity open, then close it with one update that sets a matching
  statecode/statuscode pair. Codes for phonecall, task, email and
  appointment come from the Microsoft Learn table references.
- multi-table-fk-import.md: a create script that fails partway is resumed
  by looking up what already exists, not re-run.
- SKILL.md: an activity row in the @odata.bind table and two pointers.
  Body stays under the 5,000-token budget (4,960).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Saurabh Badenkal and others added 8 commits November 5, 2026 09:39
PR-gate dry-run evals showed two ERP routing misses:

- A preview of an ERP record update loaded dv-overview, found complete
  create/update/delete templates in references/erp-target.md, and answered
  without loading dv-data. The answer was right but bypassed the skill that
  owns ERP writes. erp-target.md now keeps the read templates and sends
  writes, including previews, to dv-data, whose erp-writes.md has the
  templates. Batch jobs likewise point to dv-admin.
- 'Show me the latest Finance and Operations batch jobs' matched dv-query's
  'ERP (Finance and Operations / F&O) ... reads' and never reached dv-admin,
  whose description said only 'ERP batch jobs'. dv-admin now names Finance
  and Operations, and dv-query's boundaries send ERP writes to dv-data and
  batch jobs to dv-admin instead of linking another skill's reference file.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…cialists' templates

A PR-gate dry run showed the same skip for ERP reads: asked to inspect an
ERP entity's schema, the agent loaded dv-overview, copied the
'data describe --target erp' template from references/erp-target.md and
never loaded dv-query. The reference also still repeated dv-data's write
templates under 'Writes for ERP', contradicting the previous commit's note
that they are not repeated here.

erp-target.md now routes reads, counts and runtime schema discovery to
dv-query (references/erp-reads.md) and writes to dv-data
(references/erp-writes.md), and keeps only routing, the
ERP-vs-Dataverse differences, Custom Services, X++ and scope notes. Every
removed template is still in the owning skill's reference. Also corrects
'DMF (tier 5 above)' to tier 6.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…in dv-overview

overview_003 asks for the shortest skill sequence for a new table exported to source control; agents skipped connect and solution setup. The duplicated publisher/solution snippet is shortened to a pointer (dv-solution owns it) to stay under the body token cap.
@saurabhrb Saurabh Ravindra Badenkal (saurabhrb) changed the title fix: dv-data/dv-metadata guidance and dv-connect/dv-overview triggers from live-eval failures fix: skill guidance and routing behind recurring live-eval failures Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant