Add batch level std calculation #311

garyzhang99 · 2025-10-09T09:37:24Z

Description

Calculating the mean at the local (group) level and the standard deviation at the global (batch) level enables more robust reward shaping(https://arxiv.org/pdf/2508.08221v1).

Ready to merge.

Checklist

Please check the following items before code is ready to be reviewed.

Code has passed all tests
Docstrings have been added/updated in Google Style
Documentation has been updated
Code is ready for review

gemini-code-assist · 2025-10-09T09:37:48Z

Summary of Changes

Hello @garyzhang99, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the GRPO and Step-wise GRPO advantage functions by introducing an option to calculate the standard deviation for reward normalization at the batch level. Previously, standard deviation was computed per group. This new "batch" level calculation, while retaining group-level mean calculation, is based on recent research indicating improved robustness in reward shaping. The changes involve modifying the constructor and processing logic of the relevant advantage functions, along with updating the base abstract class and adding comprehensive unit tests to verify the new functionality.

Highlights

Batch-level Standard Deviation: Introduced a new "std_cal_level" parameter for GRPO and Step-wise GRPO advantage functions, allowing the standard deviation for reward normalization to be calculated across the entire batch of experiences, rather than per group.
Robust Reward Shaping: This change is motivated by recent research suggesting that calculating the mean at the local (group) level and the standard deviation at the global (batch) level leads to more robust reward shaping in reinforcement learning.
Core Logic Adaptation: Modified the GRPOAdvantage and MultiStepGRPOAdvantage classes to implement this new calculation logic, including passing precomputed batch standard deviation values to individual group advantage calculations.
Expanded Test Coverage: Added new unit tests (test_batch_level_std_grpo and test_batch_level_step_wise_grpo_advantage) to advantage_fn_test.py to specifically validate the correctness of the batch-level standard deviation calculation for both GRPO variants.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

garyzhang99 · 2025-10-09T09:38:31Z

/unittest-all

gemini-code-assist

Code Review

This pull request introduces a feature to calculate standard deviation at the batch level for reward normalization, which can lead to more robust reward shaping. The changes are implemented for GRPOAdvantage and StepWiseGRPOAdvantageFn, and new tests are added. My review found a few issues: a bug in one of the new tests, a minor inconsistency in grpo_advantage.py, and an opportunity for performance improvement in multi_step_grpo_advantage.py by avoiding redundant computations. I've provided suggestions to address these points.

tests/algorithm/advantage_fn_test.py

trinity/algorithm/advantage_fn/grpo_advantage.py

trinity/algorithm/advantage_fn/multi_step_grpo_advantage.py

garyzhang99 · 2025-10-09T09:59:16Z

/unittest-module-algorithm

github-actions · 2025-10-09T10:37:56Z

Summary

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Other ❓	Flaky 🍂	Duration ⏱️
13	13	0	0	0	0	4ms

Tests

Test Name	Status	Duration
tests/algorithm/advantage_fn_test.py::TestGroupedAdvantageFn::test_batch_level_std_grpo	✅	1ms
tests/algorithm/advantage_fn_test.py::TestGroupedAdvantageFn::test_batch_level_step_wise_grpo_advantage	✅	1ms
tests/algorithm/advantage_fn_test.py::TestGroupedAdvantageFn::test_duplicate_grpo	✅	1ms
tests/algorithm/advantage_fn_test.py::TestGroupedAdvantageFn::test_grpo_advantage	✅	1ms
tests/algorithm/advantage_fn_test.py::TestGroupedAdvantageFn::test_grpo_correct_bias	✅	1ms
tests/algorithm/advantage_fn_test.py::TestGroupedAdvantageFn::test_grpo_reward_std	✅	1ms
tests/algorithm/advantage_fn_test.py::TestGroupedAdvantageFn::test_step_wise_grpo_advantage	✅	1ms
tests/algorithm/policy_loss_test.py::VerlPolicyLossTest::test_dpo_policy_loss	✅	1ms
tests/algorithm/policy_loss_test.py::VerlPolicyLossTest::test_gspo_policy_loss	✅	1ms
tests/algorithm/policy_loss_test.py::VerlPolicyLossTest::test_mix_policy_loss	✅	1ms
tests/algorithm/policy_loss_test.py::VerlPolicyLossTest::test_opmd_policy_loss	✅	1ms
tests/algorithm/policy_loss_test.py::VerlPolicyLossTest::test_ppo_policy_loss	✅	1ms
tests/algorithm/policy_loss_test.py::VerlPolicyLossTest::test_sft_policy_loss	✅	1ms

Github Test Reporter by CTRF 💚

garyzhang99 · 2025-10-10T06:45:56Z

/unittest-module-algorithm

github-actions · 2025-10-10T06:47:17Z

Summary

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Other ❓	Flaky 🍂	Duration ⏱️
13	12	1	0	0	0	4ms

Failed Tests

Failed Tests ❌	Fail Message
❌ tests/algorithm/advantage_fn_test.py::TestGroupedAdvantageFn::test_step_wise_grpo_advantage	The test failed in the call phase

Tests

Test Name	Status	Duration
tests/algorithm/advantage_fn_test.py::TestGroupedAdvantageFn::test_batch_level_std_grpo	✅	1ms
tests/algorithm/advantage_fn_test.py::TestGroupedAdvantageFn::test_batch_level_step_wise_grpo_advantage	✅	1ms
tests/algorithm/advantage_fn_test.py::TestGroupedAdvantageFn::test_duplicate_grpo	✅	1ms
tests/algorithm/advantage_fn_test.py::TestGroupedAdvantageFn::test_grpo_advantage	✅	1ms
tests/algorithm/advantage_fn_test.py::TestGroupedAdvantageFn::test_grpo_correct_bias	✅	1ms
tests/algorithm/advantage_fn_test.py::TestGroupedAdvantageFn::test_grpo_reward_std	✅	1ms
tests/algorithm/advantage_fn_test.py::TestGroupedAdvantageFn::test_step_wise_grpo_advantage	❌	1ms
tests/algorithm/policy_loss_test.py::VerlPolicyLossTest::test_dpo_policy_loss	✅	1ms
tests/algorithm/policy_loss_test.py::VerlPolicyLossTest::test_gspo_policy_loss	✅	1ms
tests/algorithm/policy_loss_test.py::VerlPolicyLossTest::test_mix_policy_loss	✅	1ms
tests/algorithm/policy_loss_test.py::VerlPolicyLossTest::test_opmd_policy_loss	✅	1ms
tests/algorithm/policy_loss_test.py::VerlPolicyLossTest::test_ppo_policy_loss	✅	1ms
tests/algorithm/policy_loss_test.py::VerlPolicyLossTest::test_sft_policy_loss	✅	1ms

Github Test Reporter by CTRF 💚

garyzhang99 · 2025-10-10T11:08:08Z

/unittest-module-algorithm

github-actions · 2025-10-10T11:09:28Z

Summary

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Other ❓	Flaky 🍂	Duration ⏱️
13	13	0	0	0	0	4ms

Tests

Test Name	Status	Duration
tests/algorithm/advantage_fn_test.py::TestGroupedAdvantageFn::test_batch_level_std_grpo	✅	1ms
tests/algorithm/advantage_fn_test.py::TestGroupedAdvantageFn::test_batch_level_step_wise_grpo_advantage	✅	1ms
tests/algorithm/advantage_fn_test.py::TestGroupedAdvantageFn::test_duplicate_grpo	✅	1ms
tests/algorithm/advantage_fn_test.py::TestGroupedAdvantageFn::test_grpo_advantage	✅	1ms
tests/algorithm/advantage_fn_test.py::TestGroupedAdvantageFn::test_grpo_correct_bias	✅	1ms
tests/algorithm/advantage_fn_test.py::TestGroupedAdvantageFn::test_grpo_reward_std	✅	1ms
tests/algorithm/advantage_fn_test.py::TestGroupedAdvantageFn::test_step_wise_grpo_advantage	✅	1ms
tests/algorithm/policy_loss_test.py::VerlPolicyLossTest::test_dpo_policy_loss	✅	1ms
tests/algorithm/policy_loss_test.py::VerlPolicyLossTest::test_gspo_policy_loss	✅	1ms
tests/algorithm/policy_loss_test.py::VerlPolicyLossTest::test_mix_policy_loss	✅	1ms
tests/algorithm/policy_loss_test.py::VerlPolicyLossTest::test_opmd_policy_loss	✅	1ms
tests/algorithm/policy_loss_test.py::VerlPolicyLossTest::test_ppo_policy_loss	✅	1ms
tests/algorithm/policy_loss_test.py::VerlPolicyLossTest::test_sft_policy_loss	✅	1ms

Github Test Reporter by CTRF 💚

trinity/algorithm/advantage_fn/grpo_advantage.py

trinity/algorithm/advantage_fn/asymre_advantage.py

trinity/algorithm/advantage_fn/grpo_advantage.py

add batch level std calculation

d4baca6

gemini-code-assist bot reviewed Oct 9, 2025

View reviewed changes

tests/algorithm/advantage_fn_test.py Show resolved Hide resolved

trinity/algorithm/advantage_fn/grpo_advantage.py Outdated Show resolved Hide resolved

trinity/algorithm/advantage_fn/multi_step_grpo_advantage.py Show resolved Hide resolved

update test case

0932499

remove **kwargs for code format

40d7320

fix std none

4f2f5db

garyzhang99 changed the title ~~[WIP]Add batch level std calculation~~ Add batch level std calculation Oct 13, 2025

pan-x-c reviewed Oct 14, 2025

View reviewed changes

trinity/algorithm/advantage_fn/grpo_advantage.py Outdated Show resolved Hide resolved

trinity/algorithm/advantage_fn/asymre_advantage.py Outdated Show resolved Hide resolved

yanxi-chen reviewed Oct 14, 2025

View reviewed changes

trinity/algorithm/advantage_fn/grpo_advantage.py Outdated Show resolved Hide resolved

问昊 added 2 commits October 14, 2025 19:08

fix comments

3d157ee

add default args

d4ddd5c

pan-x-c approved these changes Oct 15, 2025

View reviewed changes

pan-x-c merged commit e29bd29 into modelscope:main Oct 15, 2025
1 check passed

Add batch level std calculation #311

Add batch level std calculation #311

Uh oh!

Conversation

garyzhang99 commented Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Uh oh!

gemini-code-assist bot commented Oct 9, 2025

Summary of Changes

Highlights

Footnotes

Uh oh!

garyzhang99 commented Oct 9, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

garyzhang99 commented Oct 9, 2025

Uh oh!

github-actions bot commented Oct 9, 2025

Summary

Tests

Uh oh!

garyzhang99 commented Oct 10, 2025

Uh oh!

github-actions bot commented Oct 10, 2025

Summary

Failed Tests

Tests

Uh oh!

garyzhang99 commented Oct 10, 2025

Uh oh!

github-actions bot commented Oct 10, 2025

Summary

Tests

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

garyzhang99 commented Oct 9, 2025 •

edited

Loading