Add benchmark scripts for Guru-Math

[chenyushuo]

Description

As the title says.

Checklist

Please check the following items before code is ready to be reviewed.

• Code has passed all tests
• Docstrings have been added/updated in Google Style
• Documentation has been updated
• Code is ready for review

[gemini-code-assist]

Summary of Changes

Hello @chenyushuo, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the benchmarking framework by integrating the Guru-Math task, enabling automated dataset generation, and providing more flexible configuration options. The changes aim to simplify the process of running and evaluating mathematical benchmarks, ensuring consistency and accuracy in performance metrics. It introduces new scripts for data preparation and a sophisticated reward function for grading complex mathematical solutions.

Highlights

• Guru-Math Benchmark Integration: Added comprehensive benchmarking support for the Guru-Math task, including a new configuration file, data generation script, and a custom reward function for evaluating mathematical solutions.
• Automated Dataset Generation: Implemented a new utility function, check_taskset_path, in benchmark/bench.py that automatically generates required datasets (like Countdown and Guru) if they are not found at the specified path, streamlining the benchmarking process.
• Flexible Configuration and Synchronization: Updated benchmark configuration files (countdown-template.yaml, gsm8k-template.yaml) to use environment variables for group and name, and introduced new command-line arguments (--sync_offset, --sync_style) in benchmark/bench.py for more granular control over synchronization settings.
• Refactored Countdown Data Generation: The gen-countdown-data.py script was removed and replaced with a refactored gen_countdown_data.py, which includes a DEFAULT_DATA_PATH and simplified data processing.
• Advanced Math Answer Grading: Introduced naive_dapo.py and reward.py plugins, providing robust normalization and grading utilities for mathematical answers, supporting LaTeX parsing, SymPy evaluation, and handling various numerical formats for accurate reward calculation.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request adds benchmark capabilities for the Guru-Math dataset, which is a great addition. The changes include new configuration files, data generation scripts, and custom reward functions. The automatic data generation for benchmarks is a nice usability improvement. My review includes a few points for improvement:

• The docstring for a new core function in bench.py could be clarified.
• There's a potential portability issue in naive_dapo.py due to a Unix-specific module import, which would prevent the benchmark from running on Windows.
• I also found a minor discrepancy between a comment and the code regarding reward values in the new reward logic.
  Overall, the changes are well-structured and significantly enhance the benchmarking capabilities of the repository.