Repository navigation
fix: skill guidance and routing behind recurring live-eval failures - #155
Open
Saurabh Ravindra Badenkal (saurabhrb) wants to merge 13 commits into
Open
Saurabh Ravindra Badenkal (saurabhrb) wants to merge 13 commits into
Saurabh Ravindra Badenkal (saurabhrb) wants to merge 13 commits into
Conversation
added 3 commits
October 14, 2026 16:14
The skill listed record association as an SDK gap with a raw POST to $ref as the alternative, which led the live eval agent to guess a non-existent client.records.associate and then hand-build the HTTP request. Point lookups at client.records.update with @odata.bind or dataverse data associate. Bump to 1.16.2.
The SDK maps 'money' to DecimalAttributeMetadata, but dv-metadata listed it as a supported type and used it in an example, so agents created Decimal columns when asked for currency. Clarify the alias and add a reference for creating a real MoneyAttributeMetadata column with dataverse api request.
Live evals skipped loading dv-connect for 'verify DATAVERSE_URL / WhoAmI' requests and dv-overview for 'which skill sequence' planning prompts that never name Dataverse. Name both request shapes in the descriptions. Bump to 1.17.0 (new trigger keywords).
Saurabh Ravindra Badenkal (saurabhrb)
marked this pull request as ready for review
October 15, 2026 02:59
…ead of re-running it Live evals showed the agent trying to create a phone call that was already closed, pairing a status reason with the wrong state, and then re-running its whole create script after the failure. The re-run left a duplicate company in the environment. - New references/activities.md: bind the regarding row with the per-table navigation property (regardingobjectid_<regarding>_<activity>), create the activity open, then close it with one update that sets a matching statecode/statuscode pair. Codes for phonecall, task, email and appointment come from the Microsoft Learn table references. - multi-table-fk-import.md: a create script that fails partway is resumed by looking up what already exists, not re-run. - SKILL.md: an activity row in the @odata.bind table and two pointers. Body stays under the 5,000-token budget (4,960). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
PR-gate dry-run evals showed two ERP routing misses: - A preview of an ERP record update loaded dv-overview, found complete create/update/delete templates in references/erp-target.md, and answered without loading dv-data. The answer was right but bypassed the skill that owns ERP writes. erp-target.md now keeps the read templates and sends writes, including previews, to dv-data, whose erp-writes.md has the templates. Batch jobs likewise point to dv-admin. - 'Show me the latest Finance and Operations batch jobs' matched dv-query's 'ERP (Finance and Operations / F&O) ... reads' and never reached dv-admin, whose description said only 'ERP batch jobs'. dv-admin now names Finance and Operations, and dv-query's boundaries send ERP writes to dv-data and batch jobs to dv-admin instead of linking another skill's reference file. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…cialists' templates A PR-gate dry run showed the same skip for ERP reads: asked to inspect an ERP entity's schema, the agent loaded dv-overview, copied the 'data describe --target erp' template from references/erp-target.md and never loaded dv-query. The reference also still repeated dv-data's write templates under 'Writes for ERP', contradicting the previous commit's note that they are not repeated here. erp-target.md now routes reads, counts and runtime schema discovery to dv-query (references/erp-reads.md) and writes to dv-data (references/erp-writes.md), and keeps only routing, the ERP-vs-Dataverse differences, Custom Services, X++ and scope notes. Every removed template is still in the owning skill's reference. Also corrects 'DMF (tier 5 above)' to tier 6. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…in dv-overview overview_003 asks for the shortest skill sequence for a new table exported to source control; agents skipped connect and solution setup. The duplicated publisher/solution snippet is shortened to a pointer (dv-solution owns it) to stay under the body token cap.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Skill guidance and routing fixes for failures that recur in the live evaluations. Each change traces a repeated failure to skill guidance, not to the platform. This PR now also includes the changes from #158 (closed in favor of this one) so they are reviewed and merged together.
Why
Recurring failures in the live evaluations, with the cause in skill guidance:
metadata_001_table_and_simple_columns). dv-metadata listed"money"as a supported type and used it in an example, but the SDK maps"money"toDecimalAttributeMetadata. Org verification then fails with "expected columnexma_estimatedcostwith typeMoney, got ...Decimal". This was still the most common run failure in the first week of November.data_008_associate_contact). dv-data listed association as an SDK gap with a rawPOST .../$refalternative. The agent guessed aclient.records.associatemethod, gotAttributeError, then fell back to the SDK's private_odatahelper or a hand-builtrequestsPOST, which the grader marks non-compliant.connect_001/connect_002(verify an existing connection or thatDATAVERSE_URLmatches the live org) answered correctly but never loadeddv-connect: dv-overview sends "connect" and "set up" requests to dv-connect but tells the agent to skip setup when auth already works, and "verify the connection" matched only the skip path.overview_001/overview_003ask which skills to use without saying "Dataverse", so dv-overview did not load; onoverview_003the agent also skipped the connect and solution steps.data_005_support_system). The agent created a phone call already closed with a status reason from another state, then re-ran the whole script after the failure and duplicated a company.erp_data_001_update_contract,erp_admin_001_list_runninganderp_metadata_001_runtime_schemaanswered from templates in dv-overview'sreferences/erp-target.mdwithout loading the owning skill.Changes
dv-data
$ref; states there is noclient.records.associate; new "Link existing records (lookup)" example withclient.records.update+parentcustomerid_account@odata.bind(CLI:dataverse data associate).references/activities.md: bind the regarding row with the per-table navigation property, create the activity open, then close it with one update that sets a matchingstatecode+statuscodepair.references/multi-table-fk-import.md: if a create script fails partway, look up what already exists and create only what is missing instead of re-running.dv-metadata
"money"documented as a"decimal"alias; the constraint example uses"decimal".references/currency-columns.md: create a realMoneyAttributeMetadatacolumn withdataverse api request, including the solution header and a verify step.dv-connect / dv-overview
DATAVERSE_URLmatch).dv-connect.dv-connect->dv-solutionpublisher and solution ->dv-metadata->dv-solutionexport and unpack). The duplicated publisher/solution snippet is shortened to a pointer (dv-solution owns the full pattern) to stay under the 5,000-token body cap.dv-overview/references/erp-target.mdis a router again: reads and runtime schema go todv-query, writes including previews todv-data, batch jobs todv-admin. Every removed template still exists in those references. Also corrects "DMF (tier 5 above)" to tier 6.dv-admin / dv-query
dv-admin/references/orgdb-settings.md: "read specific OrgDB settings" example that checks requested keys against the documented allowlist, reads only those keys, reports an absent key asnot set (platform default), and never writes back.dv-dataand batch jobs todv-admin.Version 1.17.0 -> 1.18.0 across all manifests (minor: new references and trigger keywords).
Testing
python .github/evals/static_checks.py: passed (9 skill files, 51 Python blocks); all bodies stay under the 5,000-token cap.python .github/evals/version_bump_check.py: passed.Live validation
The combined changes were run on a proof branch (
users/saurabhrb/evalarm-skills-155-158) against a live test environment on 10/17:connect_002produced no output in the evaluation harness (infrastructure), so it was not graded in that run.metadata_001created a real Money column;data_008,connect_001andoverview_003passed;data_005scored effort 5/5 (3/5 before) with exactly 3 companies.The dv-overview skill-sequence line was added after that run; the live dry-run gate on this PR covers it.