Support device detection with new benchmark suite #13182

pzread · 2023-04-20T04:45:45Z

IREE benchmark tool automatically detects the device info of benchmark device and filter the benchmarks. This change maps the detected info to device architecture enum of new benchmark suite.

For unknown architecture, instead of failing directly, print a warning and skip the benchmarks. This allows users to use the tools with partially unknown hardware (e.g. If users want to run GPU benchmarks but don't care about the CPU benchmarks, we shouldn't fail simply because their CPU is not in the supported list). This is actually the current intended behavior for x86_64 (if uarch is unknown, we don't fail but no CPU benchmarks will be run) but without any warning.

In the case that we should fail if hardware is mismatched to the benchmarks (e.g. on CI), the force mode option will be added in a follow-up change (#13198).

As a side effect, some tests are updated to test with new benchmark suite as we change some code to use new path by default.

This enables the detection of mobile phones' CPU/GPU for Android benchmark tool with new benchmark suite #13176

First 3 commits are the major changes. Each commit message describes their goals.

github-actions · 2023-04-20T06:21:09Z

Abbreviated Benchmark Summary

@ commit a3449ae47ab17b896b99ab336441c486908b2050 (vs. base f2fc7c5bb95a5e59f540d201474e53563f109f96)

No improved or regressed benchmarks 🏖️

No improved or regressed compilation metrics 🏖️

For more information:

Source Workflow Run

iree-github-actions-bot · 2023-04-20T07:59:32Z

Abbreviated Android Benchmark Summary

@ commit 177a3da54b9cfb17a1c0bdb489b443aa2204e28c (vs. base 5363ea3f0ae3af7f016ee4da4ee48f49f869dd99)

Regressed Latencies 🚩

Benchmark Name	Average Latency (ms)	Median Latency (ms)	Latency Standard Deviation (ms)
PoseNet [fp32] (TFLite) full-inference,experimental-flags with IREE-Vulkan @ Pixel-6-Pro (GPU-Mali-G78)	15.861 (vs. 14.645, 8.30%↑)	15.862	0.061

Improved Latencies 🎉

Benchmark Name	Average Latency (ms)	Median Latency (ms)	Latency Standard Deviation (ms)
PoseNet [fp32] (TFLite) big-core,full-inference,default-flags with IREE-LLVM-CPU-Sync @ Pixel-4 (CPU-ARMv8.2-A)	219.793 (vs. 275.624, 20.26%↓)	192.784	37.174

For more information:

pzread · 2023-04-21T20:18:22Z

Reorg the commits and gave each commit a clear commit message, which should help the code review.

pzread · 2023-04-26T18:33:22Z

Kindly ping

GMNGeoffrey · 2023-05-01T23:57:02Z

Sorry for the delay here, I've been swamped and summiting all last week. I'm looking now, but just from reading the description, I think I'd prefer that fail be the default and auto-detect the special flag. That way it's always clear that autodetection is happening and we don't get silent failures/skips

build_tools/benchmarks/common/benchmark_definition.py

GMNGeoffrey · 2023-05-02T00:27:16Z

build_tools/benchmarks/common/benchmark_driver.py

+    if cpu_target_arch is None:
+      print("WARNING: Detected unsupported CPU architecture in "
+            f'"{self.device_info}", CPU benchmarking is disabled.')
+      cpu_target_arch = "unknown"


Rather than letting magic string filtering take care of this here, should we have an explicit case for running no cpu benchmarks?

I realized that we probably just want to provide a list of the included architectures to the filter_benchmarks_for_category below. Refactored to avoid the magic string and simplify the interface.

pzread · 2023-05-02T16:14:44Z

#13198

SG. But I think we can address this in #13198

build_tools/benchmarks/common/benchmark_suite.py

This field indicates the version of the benchmark suite and help the following control flow make decision.

Later each DeviceArchitecture has a name and implements __str__. Using that instead of crafting one here.

ScottTodd · 2023-05-03T17:37:37Z

build_tools/benchmarks/common/benchmark_suite_test.py

  def test_load_from_run_configs(self):
    model_tflite = common_definitions.Model(


This is failing on macOS: https://github.com/openxla/iree/actions/runs/4868571511/jobs/8682175725#step:7:2529

I'm looking into it.

Should be fixed with #13390

IREE benchmark tool automatically detects the device info of benchmark device and filter the benchmarks. This change maps the detected info to device architecture enum of new benchmark suite.

pzread added benchmarks:cuda Run default CUDA benchmarks benchmarks:x86_64 Run default x86_64 benchmarks labels Apr 20, 2023

pzread force-pushed the bench-android-tool-migrate-arch branch 6 times, most recently from ff54f00 to 8b90d01 Compare April 20, 2023 05:41

pzread mentioned this pull request Apr 20, 2023

Support new benchmark suite in Android benchmark tool #13176

Merged

pzread changed the title ~~Support host detection with new benchmark suite~~ Support device detection with new benchmark suite Apr 20, 2023

pzread added the (deprecated) buildkite:benchmark-android Deprecated. Please use benchmarks:android-* label Apr 20, 2023

pzread marked this pull request as ready for review April 20, 2023 08:21

pzread requested review from GMNGeoffrey and antiagainst as code owners April 20, 2023 08:21

pzread mentioned this pull request Apr 20, 2023

Migrate to GitHub actions #9855

Closed

42 tasks

pzread force-pushed the bench-android-tool-migrate-arch branch from 4e56e94 to 95c1b17 Compare April 21, 2023 16:38

pzread removed the (deprecated) buildkite:benchmark-android Deprecated. Please use benchmarks:android-* label Apr 21, 2023

pzread force-pushed the bench-android-tool-migrate-arch branch 3 times, most recently from e4f506a to a024b3a Compare April 21, 2023 20:15

pzread force-pushed the bench-android-tool-migrate-arch branch 4 times, most recently from 306e283 to 069af70 Compare April 25, 2023 17:37

pzread added the (deprecated) buildkite:benchmark-android Deprecated. Please use benchmarks:android-* label Apr 26, 2023

GMNGeoffrey reviewed May 2, 2023

View reviewed changes

pzread removed the (deprecated) buildkite:benchmark-android Deprecated. Please use benchmarks:android-* label May 2, 2023

pzread force-pushed the bench-android-tool-migrate-arch branch from 069af70 to 9a50800 Compare May 2, 2023 15:13

pzread requested a review from GMNGeoffrey May 2, 2023 16:14

pzread force-pushed the bench-android-tool-migrate-arch branch from a76d51e to 2b1920b Compare May 2, 2023 16:16

GMNGeoffrey approved these changes May 2, 2023

View reviewed changes

build_tools/benchmarks/common/benchmark_suite.py Outdated Show resolved Hide resolved

pzread added the (deprecated) buildkite:benchmark-android Deprecated. Please use benchmarks:android-* label May 2, 2023

Jerry Wu added 7 commits May 2, 2023 22:48

Add the field legacy_suite to BenchmarkSuite.

fb46109

This field indicates the version of the benchmark suite and help the following control flow make decision.

Use the "name" of DeviceArchitecture instead of crafting one.

1fdb349

Later each DeviceArchitecture has a name and implements __str__. Using that instead of crafting one here.

Add new map and logic of device detcetion for the new suite

6cca4d2

Rename and add comments

9c0c273

Fix case

4d1b0d8

Refactor architecture filter

8f08639

Fix typo

eb885ed

pzread force-pushed the bench-android-tool-migrate-arch branch from aced3be to eb885ed Compare May 2, 2023 22:50

Fix python import path of other tools

177a3da

pzread merged commit c7a99d5 into iree-org:main May 3, 2023

ScottTodd reviewed May 3, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support device detection with new benchmark suite #13182

Support device detection with new benchmark suite #13182

pzread commented Apr 20, 2023 •

edited

Loading

github-actions bot commented Apr 20, 2023 •

edited

Loading

iree-github-actions-bot commented Apr 20, 2023 •

edited

Loading

pzread commented Apr 21, 2023

pzread commented Apr 26, 2023

GMNGeoffrey commented May 1, 2023

GMNGeoffrey May 2, 2023

pzread May 2, 2023

pzread commented May 2, 2023 •

edited

Loading

ScottTodd May 3, 2023

pzread May 3, 2023

pzread May 3, 2023 •

edited

Loading

		def test_load_from_run_configs(self):
		model_tflite = common_definitions.Model(

Support device detection with new benchmark suite #13182

Support device detection with new benchmark suite #13182

Conversation

pzread commented Apr 20, 2023 • edited Loading

github-actions bot commented Apr 20, 2023 • edited Loading

Abbreviated Benchmark Summary

iree-github-actions-bot commented Apr 20, 2023 • edited Loading

Abbreviated Android Benchmark Summary

Regressed Latencies 🚩

Improved Latencies 🎉

pzread commented Apr 21, 2023

pzread commented Apr 26, 2023

GMNGeoffrey commented May 1, 2023

GMNGeoffrey May 2, 2023

Choose a reason for hiding this comment

pzread May 2, 2023

Choose a reason for hiding this comment

pzread commented May 2, 2023 • edited Loading

ScottTodd May 3, 2023

Choose a reason for hiding this comment

pzread May 3, 2023

Choose a reason for hiding this comment

pzread May 3, 2023 • edited Loading

Choose a reason for hiding this comment

pzread commented Apr 20, 2023 •

edited

Loading

github-actions bot commented Apr 20, 2023 •

edited

Loading

iree-github-actions-bot commented Apr 20, 2023 •

edited

Loading

pzread commented May 2, 2023 •

edited

Loading

pzread May 3, 2023 •

edited

Loading