Async RL support multiple explorers #100

pan-x-c · 2025-06-25T08:10:09Z

Description

Supports running multiple explore mode experiments simultaneously in a ray cluster.

Checklist

Please check the following items before code is ready to be reviewed.

Code has passed all tests
Docstrings have been added/updated in Google Style
Documentation has been updated
Code is ready for review

pan-x-c · 2025-06-25T09:44:21Z

/run-unittest

pan-x-c · 2025-06-25T09:55:03Z

/run-unittest

github-actions · 2025-06-25T10:13:43Z

Summary

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
40	38	2	0	0	0	0	1.1s

Failed Tests

Failed Tests ❌	Fail Message
❌ tests/explorer/runner_pool_test.py::RunnerPoolTest::test_runner_pool_with_auxiliary_models	The test failed in the call phase due to an assertion error
❌ tests/trainer/trainer_test.py::TestFullyAsyncMode::test_fully_async_mode	The test failed in the call phase

Flaky Tests

No flaky tests ✨

Skipped

No skipped tests ✨

Tests

Test Name	Status	Duration
tests/algorithm/policy_loss_test.py::VerlPolicyLossTest::test_dpo_policy_loss	✅	1ms
tests/algorithm/policy_loss_test.py::VerlPolicyLossTest::test_mix_policy_loss	✅	1ms
tests/algorithm/policy_loss_test.py::VerlPolicyLossTest::test_opmd_policy_loss	✅	1ms
tests/algorithm/policy_loss_test.py::VerlPolicyLossTest::test_ppo_policy_loss	✅	1ms
tests/algorithm/policy_loss_test.py::VerlPolicyLossTest::test_sft_policy_loss	✅	1ms
tests/buffer/file_test.py::TestFileBuffer::test_file_buffer	✅	3ms
tests/buffer/file_test.py::TestFileBuffer::test_file_reader	✅	1ms
tests/buffer/file_test.py::TestFileBuffer::test_file_writer	✅	1ms
tests/buffer/queue_test.py::TestQueueBuffer::test_queue_buffer	✅	2ms
tests/buffer/sql_test.py::TestSQLBuffer::test_create_sql_buffer	✅	3ms
tests/common/config_test.py::TestConfig::test_all_examples_are_valid	✅	1ms
tests/common/config_test.py::TestConfig::test_load_default_config	✅	5ms
tests/common/experience_test.py::TestExperienceConversion::test_batch_conversion	✅	1ms
tests/common/experience_test.py::TestExperienceConversion::test_experience_model_experience_conversion	✅	1ms
tests/common/vllm_test.py::TestModelWrapperSyncV0::test_generate	✅	42ms
tests/common/vllm_test.py::TestModelWrapperAsyncV0::test_generate	✅	40ms
tests/common/vllm_test.py::TestModelWrapperAsyncTPV0::test_generate	✅	50ms
tests/common/vllm_test.py::TestModelWrapperAsyncTPV1::test_generate	✅	50ms
tests/common/vllm_test.py::TestModelWrapperAsyncV1::test_generate	✅	38ms
tests/common/vllm_test.py::TestAPIServer::test_api	✅	24ms
tests/common/vllm_test.py::TestTokenizer::test_assistant_token_mask	✅	1ms
tests/explorer/explorer_test.py::BaseExplorerCase::test_explorer	✅	1ms
tests/explorer/explorer_test.py::TestExplorerCountdownEval::test_explorer	✅	100ms
tests/explorer/explorer_test.py::TestExplorerCountdownNoEval::test_explorer	✅	107ms
tests/explorer/runner_pool_test.py::RunnerPoolTest::test_runner_pool	✅	22ms
tests/explorer/runner_pool_test.py::RunnerPoolTest::test_runner_pool_with_auxiliary_models	❌	5ms
tests/explorer/workflow_test.py::WorkflowTest::test_gsm8k_workflow	✅	1ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_boxed_workflow	✅	1ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_complex_workflow	✅	1ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_fraction_workflow	✅	1ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_workflow	✅	1ms
tests/explorer/workflow_test.py::WorkflowTest::test_workflow_resettable	✅	1ms
tests/trainer/trainer_test.py::BaseTrainerCase::test_trainer	✅	1ms
tests/trainer/trainer_test.py::TestTrainerCountdown::test_trainer	✅	279ms
tests/trainer/trainer_test.py::TestStepAheadAsyncRL::test_trainer	✅	102ms
tests/trainer/trainer_test.py::TestTrainerGSM8K::test_trainer	✅	68ms
tests/trainer/trainer_test.py::TestTrainerSFTWarmupGSM8K::test_trainer	✅	70ms
tests/trainer/trainer_test.py::TestTrainerDPO::test_trainer	✅	53ms
tests/trainer/trainer_test.py::TestFullyAsyncMode::test_fully_async_mode	❌	1ms
tests/utils/plugin_test.py::TestPluginLoader::test_load_plugins	✅	4ms

Github Test Reporter by CTRF 💚

pan-x-c · 2025-06-25T11:22:51Z

/run-unittest

docs/sphinx_doc/source/tutorial/trinity_configs.md

github-actions · 2025-06-25T11:43:24Z

Summary

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
40	39	1	0	0	0	0	1.2s

Failed Tests

Failed Tests ❌	Fail Message
❌ tests/explorer/runner_pool_test.py::RunnerPoolTest::test_runner_pool_with_auxiliary_models	The test failed in the call phase due to an assertion error

Flaky Tests

No flaky tests ✨

Skipped

No skipped tests ✨

Tests

Test Name	Status	Duration
tests/algorithm/policy_loss_test.py::VerlPolicyLossTest::test_dpo_policy_loss	✅	1ms
tests/algorithm/policy_loss_test.py::VerlPolicyLossTest::test_mix_policy_loss	✅	1ms
tests/algorithm/policy_loss_test.py::VerlPolicyLossTest::test_opmd_policy_loss	✅	1ms
tests/algorithm/policy_loss_test.py::VerlPolicyLossTest::test_ppo_policy_loss	✅	1ms
tests/algorithm/policy_loss_test.py::VerlPolicyLossTest::test_sft_policy_loss	✅	1ms
tests/buffer/file_test.py::TestFileBuffer::test_file_buffer	✅	3ms
tests/buffer/file_test.py::TestFileBuffer::test_file_reader	✅	1ms
tests/buffer/file_test.py::TestFileBuffer::test_file_writer	✅	1ms
tests/buffer/queue_test.py::TestQueueBuffer::test_queue_buffer	✅	2ms
tests/buffer/sql_test.py::TestSQLBuffer::test_create_sql_buffer	✅	3ms
tests/common/config_test.py::TestConfig::test_all_examples_are_valid	✅	1ms
tests/common/config_test.py::TestConfig::test_load_default_config	✅	5ms
tests/common/experience_test.py::TestExperienceConversion::test_batch_conversion	✅	1ms
tests/common/experience_test.py::TestExperienceConversion::test_experience_model_experience_conversion	✅	1ms
tests/common/vllm_test.py::TestModelWrapperSyncV0::test_generate	✅	41ms
tests/common/vllm_test.py::TestModelWrapperAsyncV0::test_generate	✅	40ms
tests/common/vllm_test.py::TestModelWrapperAsyncTPV0::test_generate	✅	50ms
tests/common/vllm_test.py::TestModelWrapperAsyncTPV1::test_generate	✅	51ms
tests/common/vllm_test.py::TestModelWrapperAsyncV1::test_generate	✅	37ms
tests/common/vllm_test.py::TestAPIServer::test_api	✅	24ms
tests/common/vllm_test.py::TestTokenizer::test_assistant_token_mask	✅	1ms
tests/explorer/explorer_test.py::BaseExplorerCase::test_explorer	✅	1ms
tests/explorer/explorer_test.py::TestExplorerCountdownEval::test_explorer	✅	99ms
tests/explorer/explorer_test.py::TestExplorerCountdownNoEval::test_explorer	✅	132ms
tests/explorer/runner_pool_test.py::RunnerPoolTest::test_runner_pool	✅	22ms
tests/explorer/runner_pool_test.py::RunnerPoolTest::test_runner_pool_with_auxiliary_models	❌	5ms
tests/explorer/workflow_test.py::WorkflowTest::test_gsm8k_workflow	✅	1ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_boxed_workflow	✅	1ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_complex_workflow	✅	1ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_fraction_workflow	✅	1ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_workflow	✅	1ms
tests/explorer/workflow_test.py::WorkflowTest::test_workflow_resettable	✅	1ms
tests/trainer/trainer_test.py::BaseTrainerCase::test_trainer	✅	1ms
tests/trainer/trainer_test.py::TestTrainerCountdown::test_trainer	✅	266ms
tests/trainer/trainer_test.py::TestStepAheadAsyncRL::test_trainer	✅	94ms
tests/trainer/trainer_test.py::TestTrainerGSM8K::test_trainer	✅	69ms
tests/trainer/trainer_test.py::TestTrainerSFTWarmupGSM8K::test_trainer	✅	69ms
tests/trainer/trainer_test.py::TestTrainerDPO::test_trainer	✅	53ms
tests/trainer/trainer_test.py::TestFullyAsyncMode::test_fully_async_mode	✅	108ms
tests/utils/plugin_test.py::TestPluginLoader::test_load_plugins	✅	4ms

Github Test Reporter by CTRF 💚

pan-x-c · 2025-06-25T12:14:56Z

/run-unittest

github-actions · 2025-06-25T12:35:39Z

Summary

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
40	40	0	0	0	0	0	1.2s

Failed Tests

No failed tests ✨

Flaky Tests

No flaky tests ✨

Skipped

No skipped tests ✨

Tests

Test Name	Status	Duration
tests/algorithm/policy_loss_test.py::VerlPolicyLossTest::test_dpo_policy_loss	✅	1ms
tests/algorithm/policy_loss_test.py::VerlPolicyLossTest::test_mix_policy_loss	✅	1ms
tests/algorithm/policy_loss_test.py::VerlPolicyLossTest::test_opmd_policy_loss	✅	1ms
tests/algorithm/policy_loss_test.py::VerlPolicyLossTest::test_ppo_policy_loss	✅	1ms
tests/algorithm/policy_loss_test.py::VerlPolicyLossTest::test_sft_policy_loss	✅	1ms
tests/buffer/file_test.py::TestFileBuffer::test_file_buffer	✅	3ms
tests/buffer/file_test.py::TestFileBuffer::test_file_reader	✅	1ms
tests/buffer/file_test.py::TestFileBuffer::test_file_writer	✅	1ms
tests/buffer/queue_test.py::TestQueueBuffer::test_queue_buffer	✅	2ms
tests/buffer/sql_test.py::TestSQLBuffer::test_create_sql_buffer	✅	3ms
tests/common/config_test.py::TestConfig::test_all_examples_are_valid	✅	1ms
tests/common/config_test.py::TestConfig::test_load_default_config	✅	5ms
tests/common/experience_test.py::TestExperienceConversion::test_batch_conversion	✅	1ms
tests/common/experience_test.py::TestExperienceConversion::test_experience_model_experience_conversion	✅	1ms
tests/common/vllm_test.py::TestModelWrapperSyncV0::test_generate	✅	42ms
tests/common/vllm_test.py::TestModelWrapperAsyncV0::test_generate	✅	40ms
tests/common/vllm_test.py::TestModelWrapperAsyncTPV0::test_generate	✅	50ms
tests/common/vllm_test.py::TestModelWrapperAsyncTPV1::test_generate	✅	51ms
tests/common/vllm_test.py::TestModelWrapperAsyncV1::test_generate	✅	38ms
tests/common/vllm_test.py::TestAPIServer::test_api	✅	24ms
tests/common/vllm_test.py::TestTokenizer::test_assistant_token_mask	✅	1ms
tests/explorer/explorer_test.py::BaseExplorerCase::test_explorer	✅	1ms
tests/explorer/explorer_test.py::TestExplorerCountdownEval::test_explorer	✅	94ms
tests/explorer/explorer_test.py::TestExplorerCountdownNoEval::test_explorer	✅	122ms
tests/explorer/runner_pool_test.py::RunnerPoolTest::test_runner_pool	✅	22ms
tests/explorer/runner_pool_test.py::RunnerPoolTest::test_runner_pool_with_auxiliary_models	✅	5ms
tests/explorer/workflow_test.py::WorkflowTest::test_gsm8k_workflow	✅	1ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_boxed_workflow	✅	1ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_complex_workflow	✅	1ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_fraction_workflow	✅	1ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_workflow	✅	1ms
tests/explorer/workflow_test.py::WorkflowTest::test_workflow_resettable	✅	1ms
tests/trainer/trainer_test.py::BaseTrainerCase::test_trainer	✅	1ms
tests/trainer/trainer_test.py::TestTrainerCountdown::test_trainer	✅	273ms
tests/trainer/trainer_test.py::TestStepAheadAsyncRL::test_trainer	✅	100ms
tests/trainer/trainer_test.py::TestTrainerGSM8K::test_trainer	✅	67ms
tests/trainer/trainer_test.py::TestTrainerSFTWarmupGSM8K::test_trainer	✅	69ms
tests/trainer/trainer_test.py::TestTrainerDPO::test_trainer	✅	54ms
tests/trainer/trainer_test.py::TestFullyAsyncMode::test_fully_async_mode	✅	118ms
tests/utils/plugin_test.py::TestPluginLoader::test_load_plugins	✅	4ms

Github Test Reporter by CTRF 💚

pan-x-c added 3 commits June 25, 2025 16:03

support multiple explorer

8ecb77d

fix checkpoint sync

3a62dac

clean config

1b97d44

fix checkpoint sync

a889c10

pan-x-c added 2 commits June 25, 2025 18:22

fix tests

c173cd8

fix multiprocessing init method

e02c5a3

yanxi-chen reviewed Jun 25, 2025

View reviewed changes

docs/sphinx_doc/source/tutorial/trinity_configs.md Outdated Show resolved Hide resolved

yanxi-chen reviewed Jun 25, 2025

View reviewed changes

docs/sphinx_doc/source/tutorial/trinity_configs.md Show resolved Hide resolved

pan-x-c added 2 commits June 25, 2025 20:08

add docs for async mode

d924e9b

fix comments

815e6e0

fix typo

da34830

yanxi-chen approved these changes Jun 25, 2025

View reviewed changes

pan-x-c merged commit 5be7436 into modelscope:main Jun 25, 2025
2 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Async RL support multiple explorers #100

Async RL support multiple explorers #100

Uh oh!

pan-x-c commented Jun 25, 2025 •

edited

Loading

Uh oh!

pan-x-c commented Jun 25, 2025

Uh oh!

pan-x-c commented Jun 25, 2025

Uh oh!

github-actions bot commented Jun 25, 2025

Uh oh!

pan-x-c commented Jun 25, 2025

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jun 25, 2025

Uh oh!

pan-x-c commented Jun 25, 2025

Uh oh!

github-actions bot commented Jun 25, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Async RL support multiple explorers #100

Async RL support multiple explorers #100

Uh oh!

Conversation

pan-x-c commented Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Uh oh!

pan-x-c commented Jun 25, 2025

Uh oh!

pan-x-c commented Jun 25, 2025

Uh oh!

github-actions bot commented Jun 25, 2025

Summary

Failed Tests

Flaky Tests

Skipped

Tests

Uh oh!

pan-x-c commented Jun 25, 2025

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jun 25, 2025

Summary

Failed Tests

Flaky Tests

Skipped

Tests

Uh oh!

pan-x-c commented Jun 25, 2025

Uh oh!

github-actions bot commented Jun 25, 2025

Summary

Failed Tests

Flaky Tests

Skipped

Tests

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

pan-x-c commented Jun 25, 2025 •

edited

Loading