[microTVM] Rework evaluate_model_accuracy into a more generic helper function #12539
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently,
evaluate_model_accuracy
addresses only a very specific use case - when one only cares about the average accuracy and runtime of a host-driven AOT model. This prevents users from implementing workarounds to bugs like #12538, and means we redundantly return both model predictions and the accuracy.This PR changes
evaluate_model_accuracy
inpython/tvm/micro/testing/evaluation.py
intopredict_labels_aot
. This function has the type signature:Unlike
evaluate_model_accuracy
,predict_labels_aot
does not take as input the true labels of the data. Instead, it returns an iterator of(prediction, runtime)
tuples, and lets users do what they want with these.cc @alanmacd @gromero @mehrdadh