Spaces:
Running
on
CPU Upgrade
some correct submissions' answer may have some problems, not complying with the guidelines
I find some problems with the submission' answers. Some answers are marked as correct, but doesn't comply with the guidelines provided with the question.
For example, for task id 2441, the guideline says, "Answer must be just a number rounded to 14 decimals. If a question does not have a relevant or applicable answer for the task, please respond with 'Not Applicable'", the submission of 'agent_test' which gives the answer: 2.88 which is marked as correct, but doesn't comply with the requirement for 14 decimals rounding. But my submission of answer '-2.87721200000010' is marked as incorrect.
I have found 17 such suspicious questions depicted as follows:
'gt' means the correct answers I got from 'agent_test', and the 'agent_answer' means my submissions which were marked as incorrect.
Hello,
Thank you for your submission to the benchmark.
After review, your answer “-2.87721200000010” does not fall within the permitted tolerance (1e-4) of the ground truth value, and was therefore marked as incorrect.
Best regards,
Alex