[BFCL] Standardize TEST_CATEGORY Among eval_runner.py and openfunctions_evaluation.py #506
Conversation
Good point. Will address this in a different PR. |
|
Hey @HuanzhiMao and @CharlieJCJ I don't think this is a great idea for the following reason. So we start by showing how to install the dependencies, and then in the middle we go into a long extempore on the different flags and then we come back with a list of commands to run. This ain't helpful back and forth? Maybe we retain the previous structure where after each command we show the options so users have the context on the command. Thoughts? |
How about we move the flags section to the bottom of the README, just like how all the available model names are displayed at the end? |
|
I have solved this issue here. |
I don't think this solves it right? We should present the flags when the user cares about the command, not super down. Folks read and execute code from READMEs linearly, and making them jump around isn't a good idea imo. |
As mentioned in #506, this PR make the warning messages more informative for user to know action items when aggregating leaderboard results. --------- Co-authored-by: CharlieJCJ <charliechengjieji@berkeley.edu>
…ns_evaluation.py (ShishirPatil#506) There are inconsistencies between the `test_category` argument that's used by `eval_checker/eval_runner.py` and `openfunctions_evaluation.py`. This PR partially addresses ShishirPatil#501 and ShishirPatil#502. --------- Co-authored-by: Shishir Patil <30296397+ShishirPatil@users.noreply.github.com>
…#517) As mentioned in ShishirPatil#506, this PR make the warning messages more informative for user to know action items when aggregating leaderboard results. --------- Co-authored-by: CharlieJCJ <charliechengjieji@berkeley.edu>
There are inconsistencies between the
test_categoryargument that's used byeval_checker/eval_runner.pyandopenfunctions_evaluation.py.This PR partially addresses #501 and #502.