Alfworld Concatenated Multi-turn RFT SFT format AND settings.

[kokolerk]

Description

alfworld concatenated multi-turn rft sft format

We find that

(1) update NOTE: For the Concatenated Multi-Turn RFT setup in the Qwen-2.5 series, you need to perform SFT first, then GRPO, or else the reward will be -0.1 during GRPO training due to not following the <think></think><action></action> format.
(2) enable_concatenated_multi_turn: true is important for concatenated multi-turn SFT, the default is false.
(3) max_prompt_tokens: 10240 is important for concatenated multi-turn SFT, the default is 4096
(4) update the SFT data format for Concatenated Multi-Turn RFT setup.

Checklist

Please check the following items before code is ready to be reviewed.

• Code has passed all tests
• Docstrings have been added/updated in Google Style
• Documentation has been updated
• Code is ready for review

[gemini-code-assist]

Summary of Changes

Hello @kokolerk, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request refines the Alfworld GRPO example to better support concatenated multi-turn RFT with the Qwen-2.5 series. It provides essential guidance on the SFT data format and adjusts key model configuration parameters, such as the model path and token limits, to ensure successful training and prevent common issues like negative rewards during GRPO.

Highlights

• Documentation Update: The README.md for the Alfworld GRPO example has been updated with crucial notes regarding the Concatenated Multi-Turn RFT setup for the Qwen-2.5 series, emphasizing the need for SFT before GRPO and detailing the required SFT data format.
• Model Configuration Changes: The alfworld.yaml configuration now specifies Qwen/Qwen2.5-3B-Instruct as the model_path, shifting from the 7B version.
• Token Limit Adjustments: The max_prompt_tokens has been increased to 10240 and max_response_tokens set to 4096 in alfworld.yaml to accommodate concatenated multi-turn SFT requirements.
• SFT Data Preprocessing Flag: A commented-out line for enable_concatenated_multi_turn: true has been added to the SFT trainer configuration in alfworld.yaml to highlight its importance for proper data preprocessing.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request updates the documentation and configuration for running a concatenated multi-turn RFT SFT setup for ALFWorld. The changes include updating the SFT data format in the README and adjusting model parameters in the alfworld.yaml configuration.

My review focuses on improving the clarity and correctness of the new documentation and configuration. I've pointed out a few issues in the README.md file, including typos and formatting problems in the system prompt and JSON example, which could be confusing for users. I also found a minor typo in a commented-out line in alfworld.yaml that could cause issues if uncommented.

Overall, the changes are in line with the PR's goal, and with the suggested fixes, the documentation and example configuration will be much clearer.

[garyzhang99]

LGTM