Non-verifiable Medicine QA Task #317
Conversation
Summary of ChangesHello @hiyuchang, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request introduces a new example for evaluating Large Language Models (LLMs) in a non-verifiable medicine question-answering context. It establishes a framework for using an LLM-as-a-judge approach, guided by rubrics, to assign rewards to generated responses. This enhancement is crucial for assessing model performance in domains where traditional ground-truth answers are unavailable, providing a robust method for qualitative evaluation and reinforcement learning. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
There was a problem hiding this comment.
Code Review
This pull request introduces a new example for non-verifiable medicine QA using an LLM judge with rubrics, which is a valuable addition. The implementation includes a new RubricJudgeWorkflow, along with its configuration and documentation. The overall structure is sound. My review provides feedback to enhance the robustness and correctness of the new workflow and suggests improvements to the documentation for better clarity.
|
/unittest-module-common |
Summary
Tests
Github Test Reporter by CTRF 💚 |
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces a new example for a non-verifiable medicine QA task using an LLM judge with rubrics. The implementation adds a new RubricJudgeWorkflow and associated configuration and documentation. The changes are well-structured.
My review focuses on improving the robustness and maintainability of the new workflow. I've identified a few areas for improvement:
- The JSON parsing from the LLM judge's response is brittle and could fail with slightly different outputs. I've suggested a more robust regex-based approach.
- The reward normalization logic doesn't map the rating to the full [0, 1] range as indicated by a comment.
- The prompts used for the LLM judge are hardcoded, which limits flexibility. I recommend making them configurable.
- The rubric numbering in the prompt starts from 0, which is less human-friendly than starting from 1.
- There's a minor JSON formatting error in the new README file.
Overall, this is a great addition. Addressing these points will make the new workflow more robust and easier to use and adapt.
Description
This example shows how to use LLM judge and rubrics to compute reward for a non-verifiable medicine QA task. This is inspired by the RaR-Implicit method.
Checklist
Please check the following items before code is ready to be reviewed.