CodeEvo

This is the code repository for "CodeEvo: Interaction-Driven Synthesis of Code-centric Data through Hybrid and Iterative Feedback".

To better illustrate the pipeline of our work, we provide some sample data under /data folder.

code_complete_seed_samples.jsonl is the sample seeds for code completion task, and code_instruct_seed_samples.jsonl is the sample seeds for code instruction task.

In our experiment, we make our backend models into inference servers. Our servers follow the OpenAI API protocol. One can easily build this kind of service with the vllm or sglang framework.

The pipeline lives in the codeevo/ package and is driven by generate_trajectory.py. A --task flag selects the code-generation (instruct) or code-completion (complete) variant. A --mode flag selects the synthesis method:

• full (default): the paper's method (Sections 3.1-3.4). The Reviewer first plans a structured Schema and then writes each new instruction from it (Eq. 1-2, in both harder and simpler directions); the Coder solves an instruction and refines its solution from feedback; a hybrid feedback loop fuses the deterministic compiler signal f_comp with the Reviewer's semantic judgement f_NL into f_hybrid (Eq. 3-4), driving the bidirectional difficulty adaptation of Algorithm 1; keywords are drawn by stratified sampling (Algorithm 2). f_comp is a hard gate: code that does not run cleanly is rejected regardless of the semantic judgement.
• fast: a lightweight variant that evolves each instruction in a single shot, without the Schema planning step.

After deploying the inference servers, generate trajectories with:

python generate_trajectory.py --task [instruct/complete] --mode [full/fast] \
		--solver-agent-url url/of/solver/agent \
		--solver-agent-name model-name-of-solver-agent \
		--solver-agent-key key-of-solver-agent \
		--judge-agent-url url/of/judge/agent \
		--judge-agent-name model-name-of-judge-agent \
		--judge-agent-key key-of-judge-agent \
		--seed-data-path path/to/seed/data \
		--output-base directory/to/generated/files \
		--max-solve-trial 2 --max-explore-trial 3

Here --max-solve-trial is the number of Coder refinement rounds per instruction, and --max-explore-trial is N in Algorithm 1: the maximum number of validated escalation steps per seed. --solver-agent-key and --judge-agent-key default to the OPENAI_API_KEY environment variable, or EMPTY for auth-less local servers. Each seed produces one structured trajectory JSON under <output-base>/trajectory/, holding every step, its Schema, the per-round hybrid feedback, and the validated instruction-code pairs.

build_sft_data.py turns the validated instruction-code pairs into conversational data under the OpenAI template:

python build_sft_data.py --task [instruct/complete] \
		--trajectory-root path/to/trajectory/folder \
		--seed-data-path path/to/seed/data \
		--output-path path/to/generated/sft/data/file

Seed gold solutions are appended by default when --seed-data-path is given; pass --no-seed-solution to exclude them.

The legacy/ folder keeps the earlier standalone scripts (generate_trajectory_[instruct/complete].py and build_sft_data_[instruct/complete].py). They are superseded by the unified --mode fast path above and are retained for reference.

This project is released under the MIT License. See LICENSE.

🫶 If you are interested in our work or find this repository / our data helpful, please consider using the following citation format when referencing our paper:

@inproceedings{sun2026codeevo,
  title={Codeevo: Interaction-driven synthesis of code-centric data through hybrid and iterative feedback},
  author={Sun, Qiushi and Gong, Jingyang and Li, Lei and Guo, Qipeng and Yuan, Fei},
  booktitle={Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  pages={9668--9687},
  year={2026}
}