@koparasy @slabasan @alizalisan

I took a quick look at the PR and makes sense overall. Thanks for testing it @alizalisan.

Couple of things before we can merge:

• LLVM function names that get passed to Caliper (eg _Z3fooRKs instead of just foo) look confusing and don't match the original function name. Let's see if we can do better.

• Caliper should not be "required" library. Many users may not need low-level perf counter details on their workflow and may just want the timeline traces. So, we need to add in an option/flag that determines if PFA should be built with Caliper for users who want something fine grained. With multiple workflow components eg in MuMMI or AHA Moles, we don't need this detail all the time and simple timelines are sufficient.

• We need to update the CI to match these build changes (we need to build caliper in the CI tests)

• We need to add detailed docs:

  • Explain the need and use case for integration with Caliper,
  • Add information on how to build caliper first (point to repo etc),
  • Add an example on how to use this with an application (eg CALI_CONFIG=runtime-report ./smoketest or CALI_CONFIG=runtime-report,output=test.cali ./smoketest or something with cali-query) and show the corresponding generated output, and
  • Explain how we can conduct end-to-end analysis of a workflow with a combination of PFA files and .cali files. For example, showcasing that we can get a quick big picture with the 800 configs of AMS with PFA and then dive deeper into 2-3 of those in detail with Caliper perf counters is a good use case to explain. We need to be clear in explaining that we don't want to end up with a data overload and that with a workflow with many configurations and components, we don't always want a low-level analysis for all configs. That is too much data (multiple metrics, time, cpu/mem usage, several perf counters) that we can't reasonably analyze at a function level or visualize.
  • Explain that using Caliper is only a C/C++ feature and not applicable to python workflows or hybrid workflows.