Refactor advantage computation, and delete RayPPOTrainer.fit #61

yanxi-chen · 2025-05-28T12:19:26Z

Description

As the title says.

TODO: update config manage, yaml config files, docs (espcially gamma and lam, which are now placed under AlgorithmConfig.advantage_fn_args)

Checklist

Please check the following items before code is ready to be reviewed.

Code has passed all tests
Docstrings have been added/updated in Google Style
Documentation has been updated
Code is ready for review

… configs and config manager)

yanxi-chen · 2025-05-29T04:03:46Z

/run-unittest

github-actions · 2025-05-29T04:09:23Z

Summary

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
24	19	5	0	0	0	0	303ms

Failed Tests

Failed Tests ❌	Fail Message
❌ tests/common/config_test.py::TestConfig::test_all_examples_are_valid	The test failed in the call phase due to an exception
❌ tests/common/config_test.py::TestConfig::test_load_default_config	The test failed in the call phase
❌ tests/explorer/explorer_test.py::TestExplorerCountdownEval::test_explorer	The test failed in the call phase
❌ tests/explorer/explorer_test.py::TestExplorerCountdownNoEval::test_explorer	The test failed in the call phase
❌ tests/trainer/trainer_test.py::TestTrainerCountdown::test_trainer	The test failed in the call phase

Flaky Tests

No flaky tests ✨

Skipped

No skipped tests ✨

Tests

Test Name	Status	Duration
tests/buffer/queue_test.py::TestQueueBuffer::test_queue_buffer	✅	4ms
tests/buffer/sql_test.py::TestSQLBuffer::test_create_sql_buffer	✅	1ms
tests/common/config_test.py::TestConfig::test_all_examples_are_valid	❌	1ms
tests/common/config_test.py::TestConfig::test_load_default_config	❌	5ms
tests/common/experience_test.py::TestExperienceConversion::test_batch_conversion	✅	1ms
tests/common/experience_test.py::TestExperienceConversion::test_experience_model_experience_conversion	✅	1ms
tests/common/vllm_test.py::TestModelWrapperSyncV0::test_generate	✅	46ms
tests/common/vllm_test.py::TestModelWrapperAsyncV0::test_generate	✅	45ms
tests/common/vllm_test.py::TestModelWrapperAsyncTPV0::test_generate	✅	52ms
tests/common/vllm_test.py::TestModelWrapperAsyncTPV1::test_generate	✅	53ms
tests/common/vllm_test.py::TestModelWrapperAsyncV1::test_generate	✅	38ms
tests/common/vllm_test.py::TestAPIServer::test_api	✅	21ms
tests/common/vllm_test.py::TestTokenizer::test_assistant_token_mask	✅	1ms
tests/explorer/explorer_test.py::BaseExplorerCase::test_explorer	✅	1ms
tests/explorer/explorer_test.py::TestExplorerCountdownEval::test_explorer	❌	1ms
tests/explorer/explorer_test.py::TestExplorerCountdownNoEval::test_explorer	❌	1ms
tests/explorer/runner_pool_test.py::RunnerPoolTest::test_runner_pool	✅	19ms
tests/explorer/runner_pool_test.py::RunnerPoolTest::test_runner_pool_with_auxiliary_models	✅	3ms
tests/explorer/workflow_test.py::WorkflowTest::test_gsm8k_workflow	✅	1ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_complex_workflow	✅	1ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_fraction_workflow	✅	1ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_workflow	✅	1ms
tests/trainer/trainer_test.py::BaseTrainerCase::test_trainer	✅	1ms
tests/trainer/trainer_test.py::TestTrainerCountdown::test_trainer	❌	1ms

Github Test Reporter by CTRF 💚

yanxi-chen · 2025-05-29T06:09:05Z

/run-unittest

github-actions · 2025-05-29T06:14:41Z

Summary

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
24	19	5	0	0	0	0	302ms

Failed Tests

Failed Tests ❌	Fail Message
❌ tests/common/config_test.py::TestConfig::test_all_examples_are_valid	The test failed in the call phase due to an exception
❌ tests/common/config_test.py::TestConfig::test_load_default_config	The test failed in the call phase due to an exception
❌ tests/explorer/explorer_test.py::TestExplorerCountdownEval::test_explorer	The test failed in the call phase due to an exception
❌ tests/explorer/explorer_test.py::TestExplorerCountdownNoEval::test_explorer	The test failed in the call phase due to an exception
❌ tests/trainer/trainer_test.py::TestTrainerCountdown::test_trainer	The test failed in the call phase due to an exception

Flaky Tests

No flaky tests ✨

Skipped

No skipped tests ✨

Tests

Test Name	Status	Duration
tests/buffer/queue_test.py::TestQueueBuffer::test_queue_buffer	✅	4ms
tests/buffer/sql_test.py::TestSQLBuffer::test_create_sql_buffer	✅	1ms
tests/common/config_test.py::TestConfig::test_all_examples_are_valid	❌	1ms
tests/common/config_test.py::TestConfig::test_load_default_config	❌	5ms
tests/common/experience_test.py::TestExperienceConversion::test_batch_conversion	✅	1ms
tests/common/experience_test.py::TestExperienceConversion::test_experience_model_experience_conversion	✅	1ms
tests/common/vllm_test.py::TestModelWrapperSyncV0::test_generate	✅	45ms
tests/common/vllm_test.py::TestModelWrapperAsyncV0::test_generate	✅	43ms
tests/common/vllm_test.py::TestModelWrapperAsyncTPV0::test_generate	✅	52ms
tests/common/vllm_test.py::TestModelWrapperAsyncTPV1::test_generate	✅	52ms
tests/common/vllm_test.py::TestModelWrapperAsyncV1::test_generate	✅	40ms
tests/common/vllm_test.py::TestAPIServer::test_api	✅	23ms
tests/common/vllm_test.py::TestTokenizer::test_assistant_token_mask	✅	1ms
tests/explorer/explorer_test.py::BaseExplorerCase::test_explorer	✅	1ms
tests/explorer/explorer_test.py::TestExplorerCountdownEval::test_explorer	❌	1ms
tests/explorer/explorer_test.py::TestExplorerCountdownNoEval::test_explorer	❌	1ms
tests/explorer/runner_pool_test.py::RunnerPoolTest::test_runner_pool	✅	19ms
tests/explorer/runner_pool_test.py::RunnerPoolTest::test_runner_pool_with_auxiliary_models	✅	3ms
tests/explorer/workflow_test.py::WorkflowTest::test_gsm8k_workflow	✅	1ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_complex_workflow	✅	1ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_fraction_workflow	✅	1ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_workflow	✅	1ms
tests/trainer/trainer_test.py::BaseTrainerCase::test_trainer	✅	1ms
tests/trainer/trainer_test.py::TestTrainerCountdown::test_trainer	❌	1ms

Github Test Reporter by CTRF 💚

yanxi-chen · 2025-05-29T07:28:16Z

/run-unittest

github-actions · 2025-05-29T07:37:50Z

Summary

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
24	23	1	0	0	0	0	540ms

Failed Tests

Failed Tests ❌	Fail Message
❌ tests/trainer/trainer_test.py::TestTrainerCountdown::test_trainer	The test failed in the call phase due to an exception

Flaky Tests

No flaky tests ✨

Skipped

No skipped tests ✨

Tests

Test Name	Status	Duration
tests/buffer/queue_test.py::TestQueueBuffer::test_queue_buffer	✅	4ms
tests/buffer/sql_test.py::TestSQLBuffer::test_create_sql_buffer	✅	1ms
tests/common/config_test.py::TestConfig::test_all_examples_are_valid	✅	1ms
tests/common/config_test.py::TestConfig::test_load_default_config	✅	4ms
tests/common/experience_test.py::TestExperienceConversion::test_batch_conversion	✅	1ms
tests/common/experience_test.py::TestExperienceConversion::test_experience_model_experience_conversion	✅	1ms
tests/common/vllm_test.py::TestModelWrapperSyncV0::test_generate	✅	44ms
tests/common/vllm_test.py::TestModelWrapperAsyncV0::test_generate	✅	44ms
tests/common/vllm_test.py::TestModelWrapperAsyncTPV0::test_generate	✅	52ms
tests/common/vllm_test.py::TestModelWrapperAsyncTPV1::test_generate	✅	53ms
tests/common/vllm_test.py::TestModelWrapperAsyncV1::test_generate	✅	39ms
tests/common/vllm_test.py::TestAPIServer::test_api	✅	21ms
tests/common/vllm_test.py::TestTokenizer::test_assistant_token_mask	✅	1ms
tests/explorer/explorer_test.py::BaseExplorerCase::test_explorer	✅	1ms
tests/explorer/explorer_test.py::TestExplorerCountdownEval::test_explorer	✅	116ms
tests/explorer/explorer_test.py::TestExplorerCountdownNoEval::test_explorer	✅	121ms
tests/explorer/runner_pool_test.py::RunnerPoolTest::test_runner_pool	✅	19ms
tests/explorer/runner_pool_test.py::RunnerPoolTest::test_runner_pool_with_auxiliary_models	✅	3ms
tests/explorer/workflow_test.py::WorkflowTest::test_gsm8k_workflow	✅	1ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_complex_workflow	✅	1ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_fraction_workflow	✅	1ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_workflow	✅	1ms
tests/trainer/trainer_test.py::BaseTrainerCase::test_trainer	✅	1ms
tests/trainer/trainer_test.py::TestTrainerCountdown::test_trainer	❌	4ms

Github Test Reporter by CTRF 💚

yanxi-chen added 4 commits May 28, 2025 18:52

Refactor advantage calculation. TODO: adjust config accordingly.

0eab23b

Adjust API and config. TODO: update yaml config files and test.

453d0d4

Merge branch 'algorithm_dev' into dev/refactor_advantage

3742f06

Split adv fn into separate files, and other update (TODO: update yaml…

0d5bad2

… configs and config manager)

Fix gamma/lam issue

b9346fd

Add gamma and lam back to verl_config to ensure compatibility

6d0caa1

Fix import error

db96689

yanxi-chen changed the title ~~[WIP] Refactor advantage computation, and delete RayPPOTrainer.fit~~ Refactor advantage computation, and delete RayPPOTrainer.fit Jun 3, 2025

yanxi-chen merged commit fe217aa into modelscope:algorithm_dev Jun 3, 2025
1 check passed

yanxi-chen deleted the dev/refactor_advantage branch June 3, 2025 05:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Refactor advantage computation, and delete RayPPOTrainer.fit #61

Refactor advantage computation, and delete RayPPOTrainer.fit #61

Uh oh!

yanxi-chen commented May 28, 2025 •

edited

Loading

Uh oh!

yanxi-chen commented May 29, 2025

Uh oh!

github-actions bot commented May 29, 2025

Uh oh!

yanxi-chen commented May 29, 2025

Uh oh!

github-actions bot commented May 29, 2025

Uh oh!

yanxi-chen commented May 29, 2025

Uh oh!

github-actions bot commented May 29, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Refactor advantage computation, and delete RayPPOTrainer.fit #61

Refactor advantage computation, and delete RayPPOTrainer.fit #61

Uh oh!

Conversation

yanxi-chen commented May 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Uh oh!

yanxi-chen commented May 29, 2025

Uh oh!

github-actions bot commented May 29, 2025

Summary

Failed Tests

Flaky Tests

Skipped

Tests

Uh oh!

yanxi-chen commented May 29, 2025

Uh oh!

github-actions bot commented May 29, 2025

Summary

Failed Tests

Flaky Tests

Skipped

Tests

Uh oh!

yanxi-chen commented May 29, 2025

Uh oh!

github-actions bot commented May 29, 2025

Summary

Failed Tests

Flaky Tests

Skipped

Tests

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

yanxi-chen commented May 28, 2025 •

edited

Loading