Add compileable task evaluation and auto-verification support

[alexgshaw]

Summary

• Add compile and exec configuration models plus executor/compiler plumbing for compileable tasks.
• Wire new CLI and trial flow support for evaluating compiled tasks and auto-verifying artifacts.
• Update task templates and validation tests to cover the new compile/check/annotate/analyze paths.

Testing

• Added and updated unit tests for compile config, exec config, compiler, executor, trial cleanup, and task validation.
• Validation coverage includes template artifact schema checks and CLI exec behavior.
• Not run (not requested)

---

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
harbor-docs	Ready	Preview, Comment	Jul 2, 2026 6:58am

[github-actions]

Enjoy a better diff viewing experience by clicking one of these URLs:

• devinreview
• diffshub
• linear

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

[kobe0938]

I wish we can discuss in person before merging. But I'll just keep some notes for later reference. List keeps updating as I review more deeply.

1. add harbor compile that only does the first half of harbor exec which only do task compilation NOT job related stuff like -a, -m, -e, -k, -n, --ak, --ae, --job-name, --jobs-dir, -q, probably the interface of harbor compile and harbor exec should be designed together.

also harbor compile ... --run != harbor exec ... because compile does not carry job related flags so that it's not runnable (just call back to Ryan's question last Friday)

2. the current interface, i hope it could be more consistent with harbor run, especially some shorthands like -i, -p, -o; in harbor run, -p is --path (local task/dataset directory), in harbor exec, it makes sense to use it the same way (not a -i )

under --scan, harbor exec -p cases/ produces one task per immediate subdirectory, which is almost exactly what harbor run -p dataset-dir/ does (one trial per task subdirectory). So the two commands would share a coherent rule.

This might be critical especially for users who are familiar with the harbor run commands, or even for AI agents, so that we are not confusing AI agents as well.

In terms of -i, it's a bit tricky, it means --include-task-name in harbor run, I hope -i could be the --instruction in the harbor exec. for the -i in harbor run, we can either 1. leave it be or 2. remove the shorthand; but i think either one is fine because the -i in harbor run is not widely used, and it will not be widely used as harbor exec in the future.

3. --collect rename to --artifact, --no-verify renamed to --disable-verification? it seems the functionalities are the same for those two. no need to introduce new cli flags. i really want to keep things conssitent
4. i feel there's definitely a better/expressive name for it--scan
5. add .yaml examples for harbor exec
6. "reduce" related, Is it possible that we reduce more than one time?
7. retries, timeouts, resource overrides, mounts, skills, MCP, logs, upload/share, plugins are absent in harbor exec, they maybe less frequently being used, but do we need to add them?
8. less important but seems liek --n-concurrent-agents is also absent, I really want to do a full parity comparison between harbor run and exec before merging.

[alexgshaw]

1. I agree we should add harbor compile. But lets save it for a future PR
2. Agreed. I aligned them with harbor run except I made --instruction shorthand -i (i'm not worried about the -i name clash w harbor run since it's more bespoke as you mentioned) and changed --input to --path. I also kept --output-dir as -o bc by default, jobs get nested in here too. By default, if only one -p is passed in, --scan functionality is applied. Lmk if you can think of a better name.
3. Renamed both
4. Wdyt is better? --scan matches some of the harbor add commands I think.
5. Added
6. Added --rk / --n-reduce-attempts
7. Let me think on this one. I'm tempted to wait until someone asks us for these, but I could be convinced for --skill, --plugin, --upload, --share-*
8. Not worried about this one

I think we don't want perfect configurable parity with harbor run quite yet bc:

1. Anyone can easily create a config instead which gives them full configurability
2. This is still an experimental feature and the CLI is already quite large

[kobe0938]

for record keeping purpose:

cool, if in a rush, let’s merge it then.
[10:19 PM]i can almost be sure that

”

FILE_MENTION_PATTERN = re.compile(
    r”(?<![A-Za-z0-9_./-])”
    r”((?:/|.{1,2}/)?[A-Za-z0-9_.-]+(?:/[A-Za-z0-9_.-]+)”
    r”.[A-Za-z0-9][A-Za-z0-9_.-])”
    r”(?![A-Za-z0-9_./-])”
)

”

this is too strict so that it will capture things we dont want
[10:20 PM]like “e.g.”
[10:20 PM]but it can be fixed later.
[10:21 PM]i also recommend adding some warning in the cli when using “harbor --exec” saying that this is experimental, and the flags and implementation are subject to change. (edited) 
Kobe  [10:49 PM]
i’m e2e running it. curious why in

examples/exec/map-reduce.yaml

n_attempts: 3 in reduce.job part? seems not necessary; also job_name: reduce-summary will get a conflicts if running it again.
Kobe  [11:09 PM]
tested, working fine with example; what do you say? let’s merge it and patch afterwards?
2 files Kobe  [11:17 PM]
overall i think this auto-detect artifacts and verify their existence is a bit redundant & unnecessary

it has 2 problems:

regex will mis-classify for terms like “e.g.“, “python3.12” as files needed
even if it works, this thin verification does not bring too much extra value to user. - as in, what if user wants a json? the verifier only reports the json exsits but not verifying the format.
for that, i actually think a preflight llm call or harbor run to generate the verifier with richer features is even better(more intelligent?) than this basic one, which user can opt in, of course (edited) 
Kobe  [11:24 PM]
anyway, i’ll approve & merge at 11:59pm; let’s fix afterwards!