Where do you use Phoenix
Self-hosted
What version of Phoenix are you using?
main (9212a42)
What happened?
Two warnings in phoenix.client experiments are wrong:
run_experiment: "Only X out of Y expected runs were completed successfully" never fires when a task fails, because the runs re-fetched from the server include errored runs and all of them are counted.resume_evaluation: the warning compares completed evaluations against runs × all evaluators, but only each run's missing evaluations are re-run. Re-running one missing evaluation successfully prints "Only 1 out of 2 incomplete evaluations were completed successfully."
Expected: (1) warns when any run failed; (2) warns only when an attempted evaluation failed.
Additional information
packages/phoenix-client/src/phoenix/client/resources/experiments/__init__.py, sync and async (actual_runs = len(task_runs); total_processed * len(evaluators_by_name)).
Where do you use Phoenix
Self-hosted
What version of Phoenix are you using?
main (9212a42)
What happened?
Two warnings in
phoenix.clientexperiments are wrong:run_experiment: "Only X out of Y expected runs were completed successfully" never fires when a task fails, because the runs re-fetched from the server include errored runs and all of them are counted.resume_evaluation: the warning compares completed evaluations against runs × all evaluators, but only each run's missing evaluations are re-run. Re-running one missing evaluation successfully prints "Only 1 out of 2 incomplete evaluations were completed successfully."Expected: (1) warns when any run failed; (2) warns only when an attempted evaluation failed.
Additional information
packages/phoenix-client/src/phoenix/client/resources/experiments/__init__.py, sync and async (actual_runs = len(task_runs);total_processed * len(evaluators_by_name)).