Generate CMake rules to download and import models

[pzread]

Add a module to generate CMake rules from the python-defined benchmarks.

The follow-up changes will generate the rules for compilation and run.

Here is an example of the generated CMake rules:

# Fetch the model from "https://storage.googleapis.com/iree-model-artifacts/mobilenet_v2_1.0_224.tflite"
add_custom_command(
  OUTPUT "${_MODEL_ARTIFACTS_DIR}/7d45f8e5-bb5e-48d0-928d-8f125104578f_mobilenet_v2.tflite"
  COMMAND
    "${Python3_EXECUTABLE}" "${IREE_ROOT_DIR}/build_tools/scripts/download_file.py"
    "https://storage.googleapis.com/iree-model-artifacts/mobilenet_v2_1.0_224.tflite" -o "${_MODEL_ARTIFACTS_DIR}/7d45f8e5-bb5e-48d0-928d-8f125104578f_mobilenet_v2.tflite"
  DEPENDS
    "${IREE_ROOT_DIR}/build_tools/scripts/download_file.py"
  COMMENT "Downloading https://storage.googleapis.com/iree-model-artifacts/mobilenet_v2_1.0_224.tflite"
)
add_custom_target(
    "${_PACKAGE_NAME}_model-7d45f8e5-bb5e-48d0-928d-8f125104578f"
  DEPENDS
    "${_MODEL_ARTIFACTS_DIR}/7d45f8e5-bb5e-48d0-928d-8f125104578f_mobilenet_v2.tflite"
)

# Import the TFLite model "${_MODEL_ARTIFACTS_DIR}/7d45f8e5-bb5e-48d0-928d-8f125104578f_mobilenet_v2.tflite"
iree_import_tflite_model(
  TARGET_NAME "${_PACKAGE_NAME}_iree-import-model-7d45f8e5-bb5e-48d0-928d-8f125104578f"
  SOURCE "${_MODEL_ARTIFACTS_DIR}/7d45f8e5-bb5e-48d0-928d-8f125104578f_mobilenet_v2.tflite"
  OUTPUT_MLIR_FILE "${_IREE_ARTIFACTS_DIR}/7d45f8e5-bb5e-48d0-928d-8f125104578f_mobilenet_v2/mobilenet_v2.mlir"
)

[ScottTodd]

(I might have missed this in an earlier review / design doc)
Are the generated CMake rules going to be checked in, or just materialized when building benchmarks?

[pzread]

I haven't fully decided yet. But I believe it's possible to generate and include a CMakefile during the configuration? I'll prefer not to check in the generated CMakefile.

[ScottTodd]

Hmm.. seems tricky but possible: https://stackoverflow.com/questions/52556785/code-generator-generating-its-own-cmake-files-and-targets

[GMNGeoffrey]

Hmm that seems a bit scary into deep CMake land, which is not a nice land

[ScottTodd]

Zooming out a bit, I'm wondering if CMake is the really right tool for building / running benchmarks. This moves one step away from CMake (generating rules via Python, instead of adding more logic to the rules themselves or authoring them manually), but what could this look like if we took another step in the same direction?

What is CMake (a build system / build system generator) actually providing for us here, and would other scripts / tools / build systems do a better job?

• listing benchmark artifacts to build (CMake targets)
• listing benchmarks to run
• running commands (download files, invoke importer tools, invoke compiler tools, pushing files to devices, running benchmarks, collecting results, uploading results)

Not really provided:

• Enumerating attached devices (e.g. choosing to run benchmarks on 1 of 3 connected Android devices)
• Comparing results across devices / benchmark sessions / historical data

Moving away from CMake entirely seems like a stretch, but here's another idea: what if we made the benchmarks their own CMake project (or even their own repository)? We'd have more flexibility there with defining benchmark-specific options, including dependencies, and organizing the file structure.

[pzread]

I'm only planning to use CMake to import and build the artifacts, since those have dependencies on the tools built by the build system. The benchmark list will be expressed in JSON and read by benchmark tools or other CI tools.

I agree it might be a good idea to have a separate project for benchmarks, but I don't see lots of benefits to do that for now, especially in the separate benchmark project, we need to write some extra CMake rules to find the import/compiler tools from the main IREE project. But it should be fairly easy to move these codes out when we need to.

[ScottTodd]

I'm only planning to use CMake to import and build the artifacts, since those have dependencies on the tools built by the build system.

That's a very weak dependency though - the tools are often built separately.

especially in the separate benchmark project, we need to write some extra CMake rules to find the import/compiler tools from the main IREE project

Finding a few files doesn't seem particularly tricky? This function isn't doing much:

iree/build_tools/cmake/iree_macros.cmake

Line 144 in a456db6

function(iree_get_executable_path OUTPUT_PATH_VAR EXECUTABLE)

For local iteration, it doesn't seem that unreasonable to me to require first building the tools, then running a benchmark script with the path to the tools. I don't think having the benchmarks and the compiler tools in the same CMake project adds that much value.

[ScottTodd]

(I think the mechanics of this PR itself are fine, but I'm trying to think through the design space to see if there's a more flexible / easier to maintain configuration that we could be building towards)

[GMNGeoffrey]

I think using CMake to manage dependencies between things like tflite -> mlir -> vmfb is good, let's not reinvent dependency management. But I think I agree with Scott that this could be a separate CMake project (doesn't need to be in this PR). The separate project could accept a path to an IREE install directory, just as we do for our crosscompile builds. In fact, it could still import the iree_macros.cmake, I think and so could reuse functions without actually being the same project. OTOH, what do we actually gain from the separate CMake project? We could make all benchmark options dependent options, so they're hidden unless you enable benchmarks, if the concern is just with that. Although I guess if they're a separate project they don't need to be as well namespaced, especially since it's a project we wouldn't need to craft so that someone else could use it in their own build.

[ScottTodd]

what do we actually gain from the separate CMake project?

• a simpler "core" project
• a dedicated space to focus on benchmarking (script and target namespacing, source file tree structure, build file tree structure, etc.)
• enforced looser coupling between the source build, binary tools, and benchmarks

If we had a standalone iree-org/iree-benchmarking / iree-org/iree-performance / iree-org/iree-perf / iree-org/perf repository that just used the public APIs, files, scripts, etc. then it would be easier to come in from outside the project and make changes, or reference what that project has for running custom benchmarks. The current setup pretty tightly couples the specific benchmark workloads, the tools themselves, and the scripts used to orchestrate benchmarking all within the core IREE project.

A separate CMake project in the same repository offers some of those benefits, but I think it's useful to consider where the design could eventually end up. Running benchmarks on presubmit using tools and benchmark definitions from the same repository is a good argument to keep them in the same repository, but maybe once we have more programs in the test suite that will get tricky.

[pzread]

I think most of the advantages will come from having a separate repository. I can expect that things will get tricky when we start adding more larger benchmarks (or even the benchmarks of other ML frameworks), especially right now everything is under build_tools/benchmarks. The larger benchmarks might include some unwanted deps to the core IREE repo (and again we need to use more flags to disable them).

But the new repository introduces the works of a separate CI and problems of sync between benchmarks and the core IREE repo (like how to make sure the changes on both sides won't break each other, since we want to run benchmarks for each commit, not just the nightly).

Nevertheless, I feel like this topic is stretching a lot from this PR, should we open a dedicated issue for it? : ) To me the current implementation isn't harmful for now and isn't hard to move out from IREE (mostly just refactoring some CMake rules). Just need to make sure to do the required refactoring/separation before introducing the new benchmarks.

[ScottTodd]

Nevertheless, I feel like this topic is stretching a lot from this PR, should we open a dedicated issue for it? : ) To me the current implementation isn't harmful for now and isn't hard to move out from IREE (mostly just refactoring some CMake rules).

SGTM, thanks for the discussion so far :)

[pzread]

Filed the issue #10244

[ScottTodd]

Nevertheless, I feel like this topic is stretching a lot from this PR, should we open a dedicated issue for it? : ) To me the current implementation isn't harmful for now and isn't hard to move out from IREE (mostly just refactoring some CMake rules).

SGTM, thanks for the discussion so far :)