Fix in Config

[chenyushuo]

Description

1. Add ppo_epochs.
2. Add more help messages.
3. Fix constraint on DPO mode.
4. Rename sync_iteration_interval to sync_interval.
5. Rename step and iteration. Detailed information of the renaming table:

sft_warmup_iteration -> sft_warmup_steps    (In config.py and other files)
get_checkpoint_dir_with_iteration -> get_checkpoint_dir_with_step_num   (In utils.py)
iteration_num -> step_num                                               (In utils.py)

self.iteration -> self.step_num         (In explorer.py)
explore_iter_num -> explore_step_num    (In explorer.py)
explore_step -> explore_one_period      (In explorer.py)

current_iteration -> current_step       (In manager.py)

train_step -> train_one_period          (In trainer.py)
train_iteration -> train_step           (In trainer.py)
train_sft_iteration -> train_sft_step   (In trainer.py)
train_rft_iteration -> train_rft_step   (In trainer.py)
train_dpo_iteration -> train_dpo_step   (In trainer.py)
train_iter_num -> train_step_num        (In trainer.py)

sft_iter_num -> sft_warmup_step_num     (In verl_trainer.py)

iterations -> steps     (Some comments)
step -> period          (Some comments)

6. Change the type of reward_fn in Task from RewardFn to Type[RewardFn].

Checklist

Please check the following items before code is ready to be reviewed.

• Code has passed all tests
• Docstrings have been added/updated in Google Style
• Documentation has been updated
• Code is ready for review

[chenyushuo]

/run-unittest

[github-actions]

Summary

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
19	16	3	0	0	0	0	736ms

Failed Tests

Failed Tests ❌	Fail Message
❌ tests/buffer/queue_test.py::TestQueueBuffer::test_queue_buffer	The test failed in the call phase
❌ tests/common/config_test.py::TestConfig::test_all_examples_are_valid	The test failed in the call phase due to an exception
❌ tests/explorer/runner_pool_test.py::RunnerPoolTest::test_runner_pool	The test failed in the call phase

Flaky Tests

No flaky tests ✨

Skipped

No skipped tests ✨

Tests

Test Name	Status	Flaky	Duration
tests/buffer/queue_test.py::TestQueueBuffer::test_queue_buffer	❌		1ms
tests/buffer/sql_test.py::TestSQLBuffer::test_create_sql_buffer	✅		1ms
tests/common/config_test.py::TestConfig::test_all_examples_are_valid	❌		1ms
tests/common/config_test.py::TestConfig::test_load_default_config	✅		1ms
tests/common/experience_test.py::TestExperienceConversion::test_batch_conversion	✅		1ms
tests/common/experience_test.py::TestExperienceConversion::test_experience_model_experience_conversion	✅		1ms
tests/common/vllm_test.py::TestModelWrapperSync::test_generate	✅		43ms
tests/common/vllm_test.py::TestModelWrapperAsync::test_generate	✅		40ms
tests/common/vllm_test.py::TestTokenizer::test_assistant_token_mask	✅		1ms
tests/explorer/explorer_test.py::BaseExplorerCase::test_explorer	✅		1ms
tests/explorer/explorer_test.py::TestExplorerCountdownEval::test_explorer	✅		209ms
tests/explorer/explorer_test.py::TestExplorerCountdownNoEval::test_explorer	✅		196ms
tests/explorer/runner_pool_test.py::RunnerPoolTest::test_runner_pool	❌		1ms
tests/explorer/workflow_test.py::WorkflowTest::test_gsm8k_workflow	✅		1ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_complex_workflow	✅		1ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_fraction_workflow	✅		1ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_workflow	✅		1ms
tests/trainer/trainer_test.py::BaseTrainerCase::test_trainer	✅		1ms
tests/trainer/trainer_test.py::TestTrainerCountdown::test_trainer	✅		240ms

Github Test Reporter by CTRF 💚

[chenyushuo]

/run-unittest

[github-actions]

Summary

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
22	21	1	0	0	0	0	823ms

Failed Tests

Failed Tests ❌	Fail Message
❌ tests/common/config_test.py::TestConfig::test_all_examples_are_valid	The test failed in the call phase due to an exception

Flaky Tests

No flaky tests ✨

Skipped

No skipped tests ✨

Tests

Test Name	Status	Flaky	Duration
tests/buffer/queue_test.py::TestQueueBuffer::test_queue_buffer	✅		3ms
tests/buffer/sql_test.py::TestSQLBuffer::test_create_sql_buffer	✅		1ms
tests/common/config_test.py::TestConfig::test_all_examples_are_valid	❌		1ms
tests/common/config_test.py::TestConfig::test_load_default_config	✅		1ms
tests/common/experience_test.py::TestExperienceConversion::test_batch_conversion	✅		1ms
tests/common/experience_test.py::TestExperienceConversion::test_experience_model_experience_conversion	✅		1ms
tests/common/vllm_test.py::TestModelWrapperSyncV0::test_generate	✅		36ms
tests/common/vllm_test.py::TestModelWrapperAsyncV0::test_generate	✅		41ms
tests/common/vllm_test.py::TestModelWrapperAsyncTPV0::test_generate	✅		51ms
tests/common/vllm_test.py::TestModelWrapperAsyncTPV1::test_generate	✅		53ms
tests/common/vllm_test.py::TestModelWrapperAsyncV1::test_generate	✅		38ms
tests/common/vllm_test.py::TestTokenizer::test_assistant_token_mask	✅		1ms
tests/explorer/explorer_test.py::BaseExplorerCase::test_explorer	✅		1ms
tests/explorer/explorer_test.py::TestExplorerCountdownEval::test_explorer	✅		175ms
tests/explorer/explorer_test.py::TestExplorerCountdownNoEval::test_explorer	✅		186ms
tests/explorer/runner_pool_test.py::RunnerPoolTest::test_runner_pool	✅		18ms
tests/explorer/workflow_test.py::WorkflowTest::test_gsm8k_workflow	✅		1ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_complex_workflow	✅		1ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_fraction_workflow	✅		1ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_workflow	✅		1ms
tests/trainer/trainer_test.py::BaseTrainerCase::test_trainer	✅		1ms
tests/trainer/trainer_test.py::TestTrainerCountdown::test_trainer	✅		210ms

Github Test Reporter by CTRF 💚

[pan-x-c]

LGTM