Skip to content

Conversation

@pan-x-c
Copy link
Collaborator

@pan-x-c pan-x-c commented Dec 2, 2025

Description

  1. Save debug output experiences into sqlite database
  2. Make profiling optional
  3. Support dynamically module import

Checklist

Please check the following items before code is ready to be reviewed.

  • Code has passed all tests
  • Docstrings have been added/updated in Google Style
  • Documentation has been updated
  • Code is ready for review

@gemini-code-assist
Copy link
Contributor

Summary of Changes

Hello @pan-x-c, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly upgrades the debugging capabilities by centralizing debug outputs into an SQLite database, making performance profiling an opt-in feature, and introducing dynamic module loading. These changes streamline the debugging workflow, offer better control over resource usage, and enhance the system's adaptability for custom modules.

Highlights

  • Enhanced Debug Output Management: Debug mode now saves output experiences into an SQLite database within a specified output directory, providing a structured and persistent record.
  • Optional Performance Profiling: Performance profiling using VizTracer is now optional and can be enabled via a new flag, giving users more control over debug overhead.
  • Dynamic Module Import Support: The system now supports dynamically importing modules, particularly for workflows, improving flexibility and extensibility.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request enhances the debug mode by allowing debug outputs to be saved to a SQLite database, making performance profiling optional, and adding support for dynamic module imports. The changes are well-implemented across the documentation, application logic, and tests. I have a few suggestions to improve test robustness, ensure interface consistency, and improve code clarity.

@pan-x-c
Copy link
Collaborator Author

pan-x-c commented Dec 2, 2025

/unittest-module-utils

@pan-x-c
Copy link
Collaborator Author

pan-x-c commented Dec 2, 2025

/unittest-module-cli

@github-actions
Copy link

github-actions bot commented Dec 2, 2025

Summary

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Other ❓ Flaky 🍂 Duration ⏱️
22 22 0 0 0 0 1m 41s

Tests

Test Name Status Flaky Duration
tests/utils/eval_utils_test.py::TestComputeScore::test_both_boxed_and_equivalent 55ms
tests/utils/eval_utils_test.py::TestComputeScore::test_both_boxed_and_not_equivalent 1ms
tests/utils/eval_utils_test.py::TestComputeScore::test_empty_ground_truth 2ms
tests/utils/eval_utils_test.py::TestComputeScore::test_empty_solution_string 1ms
tests/utils/eval_utils_test.py::TestComputeScore::test_multiple_boxed_answers_in_solution 2ms
tests/utils/eval_utils_test.py::TestComputeScore::test_solution_boxed_truth_raw_and_equivalent 1ms
tests/utils/eval_utils_test.py::TestComputeScore::test_solution_boxed_truth_raw_and_not_equivalent 1ms
tests/utils/eval_utils_test.py::TestComputeScore::test_solution_not_boxed 1ms
tests/utils/eval_utils_test.py::TestComputeScore::test_solution_raw_and_ground_truth_boxed_equivalent 1ms
tests/utils/eval_utils_test.py::TestMathEvalUtils::test_extract_answer 4ms
tests/utils/eval_utils_test.py::TestMathEvalUtils::test_verify_math_answer 149ms
tests/utils/eval_utils_test.py::TestEvalUtils::test_is_equiv 5ms
tests/utils/log_test.py::LogTest::test_actor_log 10.9s
tests/utils/log_test.py::LogTest::test_group_by_node 4.9s
tests/utils/log_test.py::LogTest::test_no_actor_log 908ms
tests/utils/plugin_test.py::TestPluginLoader::test_load_plugins_local_0__workspace_tests_utils_plugins 98ms
tests/utils/plugin_test.py::TestPluginLoader::test_load_plugins_local_1_tests_utils_plugins 94ms
tests/utils/plugin_test.py::TestPluginLoader::test_load_plugins_remote_0__workspace_tests_utils_plugins 23.1s
tests/utils/plugin_test.py::TestPluginLoader::test_load_plugins_remote_1_tests_utils_plugins 21.9s
tests/utils/plugin_test.py::TestPluginLoader::test_passing_custom_class_0__workspace_tests_utils_plugins 11.8s
tests/utils/plugin_test.py::TestPluginLoader::test_passing_custom_class_1_tests_utils_plugins 11.4s
tests/utils/registry_test.py::TestRegistry::test_dynamic_import 1ms

Github Test Reporter by CTRF 💚

@github-actions
Copy link

github-actions bot commented Dec 2, 2025

Summary

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Other ❓ Flaky 🍂 Duration ⏱️
5 5 0 0 0 0 1m 22s

Tests

Test Name Status Flaky Duration
tests/cli/launcher_test.py::TestLauncherMain::test_debug_mode 53.7s
tests/cli/launcher_test.py::TestLauncherMain::test_main_run_command 6.8s
tests/cli/launcher_test.py::TestLauncherMain::test_main_run_in_dlc 1.5s
tests/cli/launcher_test.py::TestLauncherMain::test_main_studio_command 321ms
tests/cli/launcher_test.py::TestLauncherMain::test_multi_stage_run 1.7s

Github Test Reporter by CTRF 💚

@pan-x-c
Copy link
Collaborator Author

pan-x-c commented Dec 2, 2025

/unittest-module-buffer

@github-actions
Copy link

github-actions bot commented Dec 2, 2025

Summary

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Other ❓ Flaky 🍂 Duration ⏱️
37 37 0 0 0 0 2m 26s

Tests

Test Name Status Flaky Duration
tests/buffer/experience_pipeline_test.py::TestExperiencePipeline::test_experience_pipeline 35.1s
tests/buffer/experience_pipeline_test.py::TestExperiencePipeline::test_pass_rate_calculation 15.8s
tests/buffer/experience_storage_test.py::ExperienceStorageTest::test_sql_experience_buffer 3.9s
tests/buffer/experience_storage_test.py::ExperienceStorageTest::test_sql_storage_0_sft 5.9s
tests/buffer/experience_storage_test.py::ExperienceStorageTest::test_sql_storage_1_dpo 6.6s
tests/buffer/file_test.py::TestFileBuffer::test_file_reader 157ms
tests/buffer/file_test.py::TestFileBuffer::test_file_writer 4.3s
tests/buffer/formatter_test.py::TestFormatter::test_dpo_messages_formatter 529ms
tests/buffer/formatter_test.py::TestFormatter::test_dpo_plaintext_formatter 453ms
tests/buffer/formatter_test.py::TestFormatter::test_multi_modal_sft_formatter 879ms
tests/buffer/formatter_test.py::TestFormatter::test_sft_messages_formatter 1.0s
tests/buffer/formatter_test.py::TestFormatter::test_sft_plaintext_formatter 731ms
tests/buffer/formatter_test.py::TestFormatter::test_task_formatter 221ms
tests/buffer/queue_test.py::TestQueueBuffer::test_priority_queue_buffer_reuse 9.1s
tests/buffer/queue_test.py::TestQueueBuffer::test_priority_queue_capacity 4.9s
tests/buffer/queue_test.py::TestQueueBuffer::test_priority_queue_reuse_count_control 6.7s
tests/buffer/queue_test.py::TestQueueBuffer::test_queue_buffer_0_queue 6.1s
tests/buffer/queue_test.py::TestQueueBuffer::test_queue_buffer_1_priority_queue 6.0s
tests/buffer/queue_test.py::TestQueueBuffer::test_queue_buffer_capacity 6.5s
tests/buffer/reader_test.py::TestBufferReader::test_buffer_reader_registration 814ms
tests/buffer/reward_shaping_mapper_test.py::TestRewardShapingMapper::test_basic_usage 6ms
tests/buffer/sql_test.py::TestSQLBuffer::test_sql_exp_buffer_read_write 4.1s
tests/buffer/sql_test.py::TestSQLBuffer::test_sql_task_buffer_read_write 4.6s
tests/buffer/task_scheduler_test.py::TestTaskScheduler::test_task_scheduler_0 91ms
tests/buffer/task_scheduler_test.py::TestTaskScheduler::test_task_scheduler_1 71ms
tests/buffer/task_scheduler_test.py::TestTaskScheduler::test_task_scheduler_2 110ms
tests/buffer/task_scheduler_test.py::TestTaskScheduler::test_task_scheduler_3 111ms
tests/buffer/task_scheduler_test.py::TestTaskScheduler::test_task_scheduler_4 111ms
tests/buffer/task_scheduler_test.py::TestTaskScheduler::test_task_scheduler_5 116ms
tests/buffer/task_scheduler_test.py::TestTaskScheduler::test_task_scheduler_6 133ms
tests/buffer/task_scheduler_test.py::TestTaskScheduler::test_task_scheduler_simple 58ms
tests/buffer/task_storage_test.py::TaskStorageTest::test_read_task_0_file 72ms
tests/buffer/task_storage_test.py::TaskStorageTest::test_read_task_1_sql 4.2s
tests/buffer/task_storage_test.py::TaskStorageTest::test_read_task_2_file 52ms
tests/buffer/task_storage_test.py::TaskStorageTest::test_read_task_3_sql 3.9s
tests/buffer/task_storage_test.py::TaskStorageTest::test_read_task_4_file 51ms
tests/buffer/task_storage_test.py::TaskStorageTest::test_read_task_5_sql 4.4s

Github Test Reporter by CTRF 💚

@pan-x-c
Copy link
Collaborator Author

pan-x-c commented Dec 2, 2025

/unittest-diff

@github-actions
Copy link

github-actions bot commented Dec 2, 2025

Summary

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Other ❓ Flaky 🍂 Duration ⏱️
109 107 1 1 0 0 18m 54s

Failed Tests

Failed Tests ❌ Fail Message
❌ tests/cli/launcher_test.py::TestLauncherMain::test_debug_mode The test failed in the call phase due to an exception

Skipped

Tests Status
tests/explorer/workflow_test.py::TestAgentScopeWorkflowAdapter::test_adapter skipped ⏭️

Tests

Test Name Status Flaky Duration
tests/buffer/experience_pipeline_test.py::TestExperiencePipeline::test_experience_pipeline 25.7s
tests/buffer/experience_pipeline_test.py::TestExperiencePipeline::test_pass_rate_calculation 16.5s
tests/buffer/experience_storage_test.py::ExperienceStorageTest::test_sql_experience_buffer 3.9s
tests/buffer/experience_storage_test.py::ExperienceStorageTest::test_sql_storage_0_sft 5.9s
tests/buffer/experience_storage_test.py::ExperienceStorageTest::test_sql_storage_1_dpo 6.4s
tests/buffer/file_test.py::TestFileBuffer::test_file_reader 157ms
tests/buffer/file_test.py::TestFileBuffer::test_file_writer 4.4s
tests/buffer/formatter_test.py::TestFormatter::test_dpo_messages_formatter 535ms
tests/buffer/formatter_test.py::TestFormatter::test_dpo_plaintext_formatter 476ms
tests/buffer/formatter_test.py::TestFormatter::test_multi_modal_sft_formatter 906ms
tests/buffer/formatter_test.py::TestFormatter::test_sft_messages_formatter 987ms
tests/buffer/formatter_test.py::TestFormatter::test_sft_plaintext_formatter 733ms
tests/buffer/formatter_test.py::TestFormatter::test_task_formatter 222ms
tests/buffer/queue_test.py::TestQueueBuffer::test_priority_queue_buffer_reuse 9.1s
tests/buffer/queue_test.py::TestQueueBuffer::test_priority_queue_capacity 4.9s
tests/buffer/queue_test.py::TestQueueBuffer::test_priority_queue_reuse_count_control 6.8s
tests/buffer/queue_test.py::TestQueueBuffer::test_queue_buffer_0_queue 5.8s
tests/buffer/queue_test.py::TestQueueBuffer::test_queue_buffer_1_priority_queue 6.0s
tests/buffer/queue_test.py::TestQueueBuffer::test_queue_buffer_capacity 6.4s
tests/buffer/reader_test.py::TestBufferReader::test_buffer_reader_registration 814ms
tests/buffer/reward_shaping_mapper_test.py::TestRewardShapingMapper::test_basic_usage 6ms
tests/buffer/sql_test.py::TestSQLBuffer::test_sql_exp_buffer_read_write 3.9s
tests/buffer/sql_test.py::TestSQLBuffer::test_sql_task_buffer_read_write 4.4s
tests/buffer/task_scheduler_test.py::TestTaskScheduler::test_task_scheduler_0 91ms
tests/buffer/task_scheduler_test.py::TestTaskScheduler::test_task_scheduler_1 71ms
tests/buffer/task_scheduler_test.py::TestTaskScheduler::test_task_scheduler_2 114ms
tests/buffer/task_scheduler_test.py::TestTaskScheduler::test_task_scheduler_3 111ms
tests/buffer/task_scheduler_test.py::TestTaskScheduler::test_task_scheduler_4 111ms
tests/buffer/task_scheduler_test.py::TestTaskScheduler::test_task_scheduler_5 116ms
tests/buffer/task_scheduler_test.py::TestTaskScheduler::test_task_scheduler_6 132ms
tests/buffer/task_scheduler_test.py::TestTaskScheduler::test_task_scheduler_simple 59ms
tests/buffer/task_storage_test.py::TaskStorageTest::test_read_task_0_file 73ms
tests/buffer/task_storage_test.py::TaskStorageTest::test_read_task_1_sql 3.8s
tests/buffer/task_storage_test.py::TaskStorageTest::test_read_task_2_file 52ms
tests/buffer/task_storage_test.py::TaskStorageTest::test_read_task_3_sql 3.9s
tests/buffer/task_storage_test.py::TaskStorageTest::test_read_task_4_file 52ms
tests/buffer/task_storage_test.py::TaskStorageTest::test_read_task_5_sql 4.6s
tests/cli/launcher_test.py::TestLauncherMain::test_debug_mode 55.5s
tests/cli/launcher_test.py::TestLauncherMain::test_main_run_command 6.9s
tests/cli/launcher_test.py::TestLauncherMain::test_main_run_in_dlc 1.5s
tests/cli/launcher_test.py::TestLauncherMain::test_main_studio_command 313ms
tests/cli/launcher_test.py::TestLauncherMain::test_multi_stage_run 1.7s
tests/explorer/explorer_test.py::TestExplorerCountdownEval::test_explorer 1m 19s
tests/explorer/explorer_test.py::TestExplorerGSM8KRULERNoEval::test_explorer 1m 43s
tests/explorer/explorer_test.py::TestExplorerGSM8k::test_explorer 3m 39s
tests/explorer/explorer_test.py::ServeTest::test_serve 1m 20s
tests/explorer/scheduler_test.py::SchedulerTest::test_async_workflow 12.3s
tests/explorer/scheduler_test.py::SchedulerTest::test_concurrent_operations 12.2s
tests/explorer/scheduler_test.py::SchedulerTest::test_dynamic_timeout 20.5s
tests/explorer/scheduler_test.py::SchedulerTest::test_get_results 27.6s
tests/explorer/scheduler_test.py::SchedulerTest::test_metric_calculation_with_non_repeatable_workflow_0 12.3s
tests/explorer/scheduler_test.py::SchedulerTest::test_metric_calculation_with_non_repeatable_workflow_1 12.5s
tests/explorer/scheduler_test.py::SchedulerTest::test_metric_calculation_with_repeatable_workflow_0 12.2s
tests/explorer/scheduler_test.py::SchedulerTest::test_metric_calculation_with_repeatable_workflow_1 12.2s
tests/explorer/scheduler_test.py::SchedulerTest::test_multi_step_execution 12.7s
tests/explorer/scheduler_test.py::SchedulerTest::test_non_repeatable_workflow 12.5s
tests/explorer/scheduler_test.py::SchedulerTest::test_over_rollout_min_wait 16.4s
tests/explorer/scheduler_test.py::SchedulerTest::test_scheduler_all_methods 22.4s
tests/explorer/scheduler_test.py::SchedulerTest::test_scheduler_restart_after_stop 23.6s
tests/explorer/scheduler_test.py::SchedulerTest::test_split_tasks 15.2s
tests/explorer/scheduler_test.py::SchedulerTest::test_stepwise_experience_eid 32.6s
tests/explorer/scheduler_test.py::SchedulerTest::test_wait_all 15.3s
tests/explorer/scheduler_test.py::SchedulerTest::test_wait_all_timeout_with_multi_batch 20.7s
tests/explorer/scheduler_test.py::TestRunnerStateCollection::test_runner_state_collection 17.4s
tests/explorer/step_wise_workflow_test.py::WorkflowTest::test_reward_propagation_workflow_0 1ms
tests/explorer/step_wise_workflow_test.py::WorkflowTest::test_reward_propagation_workflow_1 603ms
tests/explorer/step_wise_workflow_test.py::WorkflowTest::test_step_wise_reward_workflow_0 355ms
tests/explorer/step_wise_workflow_test.py::WorkflowTest::test_step_wise_reward_workflow_1 1.0s
tests/explorer/step_wise_workflow_test.py::WorkflowTest::test_workflows_raise_error 1ms
tests/explorer/step_wise_workflow_test.py::WorkflowTest::test_workflows_stop_at_max_env_steps 1.0s
tests/explorer/workflow_test.py::WorkflowTest::test_gsm8k_workflow 32ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_boxed_workflow 23ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_complex_workflow 176ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_eval_workflow 4ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_fraction_workflow 13ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_workflow 8ms
tests/explorer/workflow_test.py::WorkflowTest::test_rm_gallery_workflow 134ms
tests/explorer/workflow_test.py::WorkflowTest::test_workflow_repeatable_0 1ms
tests/explorer/workflow_test.py::WorkflowTest::test_workflow_repeatable_1 100ms
tests/explorer/workflow_test.py::WorkflowTest::test_workflow_resettable_0 1ms
tests/explorer/workflow_test.py::WorkflowTest::test_workflow_resettable_1 201ms
tests/explorer/workflow_test.py::MultiTurnWorkflowTest_0::test_multi_turn_workflow 15.3s
tests/explorer/workflow_test.py::MultiTurnWorkflowTest_1::test_multi_turn_workflow 14.7s
tests/explorer/workflow_test.py::TestWorkflowStateRecording::test_workflow_state_recording 4.0s
tests/explorer/workflow_test.py::TestAgentScopeWorkflowAdapter::test_adapter ⏭️ 1ms
tests/explorer/workflow_test.py::TestWorkflowRunner::test_workflow_runner 297ms
tests/explorer/workflow_test.py::TestWorkflowRunner::test_workflow_runner_get_state 8.1s
tests/utils/eval_utils_test.py::TestComputeScore::test_both_boxed_and_equivalent 2ms
tests/utils/eval_utils_test.py::TestComputeScore::test_both_boxed_and_not_equivalent 1ms
tests/utils/eval_utils_test.py::TestComputeScore::test_empty_ground_truth 2ms
tests/utils/eval_utils_test.py::TestComputeScore::test_empty_solution_string 1ms
tests/utils/eval_utils_test.py::TestComputeScore::test_multiple_boxed_answers_in_solution 2ms
tests/utils/eval_utils_test.py::TestComputeScore::test_solution_boxed_truth_raw_and_equivalent 1ms
tests/utils/eval_utils_test.py::TestComputeScore::test_solution_boxed_truth_raw_and_not_equivalent 1ms
tests/utils/eval_utils_test.py::TestComputeScore::test_solution_not_boxed 1ms
tests/utils/eval_utils_test.py::TestComputeScore::test_solution_raw_and_ground_truth_boxed_equivalent 1ms
tests/utils/eval_utils_test.py::TestMathEvalUtils::test_extract_answer 3ms
tests/utils/eval_utils_test.py::TestMathEvalUtils::test_verify_math_answer 73ms
tests/utils/eval_utils_test.py::TestEvalUtils::test_is_equiv 5ms
tests/utils/log_test.py::LogTest::test_actor_log 4.8s
tests/utils/log_test.py::LogTest::test_group_by_node 4.8s
tests/utils/log_test.py::LogTest::test_no_actor_log 906ms
tests/utils/plugin_test.py::TestPluginLoader::test_load_plugins_local_0__workspace_tests_utils_plugins 97ms
tests/utils/plugin_test.py::TestPluginLoader::test_load_plugins_local_1_tests_utils_plugins 96ms
tests/utils/plugin_test.py::TestPluginLoader::test_load_plugins_remote_0__workspace_tests_utils_plugins 22.0s
tests/utils/plugin_test.py::TestPluginLoader::test_load_plugins_remote_1_tests_utils_plugins 21.9s
tests/utils/plugin_test.py::TestPluginLoader::test_passing_custom_class_0__workspace_tests_utils_plugins 11.9s
tests/utils/plugin_test.py::TestPluginLoader::test_passing_custom_class_1_tests_utils_plugins 11.5s
tests/utils/registry_test.py::TestRegistry::test_dynamic_import 4.0s

Github Test Reporter by CTRF 💚

@pan-x-c
Copy link
Collaborator Author

pan-x-c commented Dec 3, 2025

/unittest-module-cli

@github-actions
Copy link

github-actions bot commented Dec 3, 2025

Summary

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Other ❓ Flaky 🍂 Duration ⏱️
5 5 0 0 0 0 1m 27s

Tests

Test Name Status Flaky Duration
tests/cli/launcher_test.py::TestLauncherMain::test_debug_mode 58.5s
tests/cli/launcher_test.py::TestLauncherMain::test_main_run_command 6.5s
tests/cli/launcher_test.py::TestLauncherMain::test_main_run_in_dlc 1.4s
tests/cli/launcher_test.py::TestLauncherMain::test_main_studio_command 317ms
tests/cli/launcher_test.py::TestLauncherMain::test_multi_stage_run 1.7s

Github Test Reporter by CTRF 💚

@chenyushuo chenyushuo merged commit 54c2d72 into modelscope:main Dec 3, 2025
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants