Repository navigation
fix(grader): ignore internal whitespace in fallback string comparison (fixes failing test on main) - #88
Open
Virtuoso461 wants to merge 1 commit into
Conversation
When an answer cannot be converted to a sympy expression (e.g. "28 \%"),
verify() falls back to comparing the extracted strings. The same answer
reaches the fallback with or without internal spaces depending on where
it was extracted from: "28 \%" from '$28 \%$' vs "28\%" from
'\boxed{28\%}'. Comparing with == after only stripping the ends made
these unequal.
This normalizes all whitespace in the fallback comparison and fixes the
currently failing test_latex_notation_math stock-percent case on main.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fixes a currently failing test on main:
tests/test_all.py::test_latex_notation_math(the stock-percent case) fails on current main.Root cause
When a LaTeX answer cannot be converted to a sympy expression (e.g.
28 \%),grader.verifyfalls back to comparing the extracted strings. The same answer reaches that fallback with or without internal whitespace depending on where it was extracted from:$28 \%$→"28 \%"\boxed{28\%}→"28\%"The fallback only stripped the ends and compared with
==, so these compared unequal — the goldThe final answer is $28 \%$. ...vs the prediction$\boxed{28\%}$case in the test suite returnsFalsealthough both answers are identical.Fix
Normalize all internal whitespace in the fallback string comparison (
"".join(s.split())). Empty/whitespace-only strings still never match. No sympy comparison path is affected.Tests
tests/test_whitespace_fallback.py: equal-after-normalization, still-rejects-different, still-rejects-empty.tests/test_all.pygoes from 1 failed / 197 passed → 198 passed; full suite (excluding the large corpus files) 232 passed, no regressions.Note: I couldn't find CI workflows in the repo, which is presumably why this red test went unnoticed — happy to split a CI addition into a separate PR if wanted.