pmc_perf

[jpietarilagraham]

When I run omniperf analyze I get this error:

tool: error: Profling Error: Cannot find pmc_perf.csv in 

But when I look at my mi200 output directory, I see

mi200/perfmon:
pmc_perf.txt  SQ_IFETCH_LEVEL.txt  SQ_INST_LEVEL_LDS.txt  SQ_INST_LEVEL_SMEM.txt  SQ_INST_LEVEL_VMEM.txt  SQ_LEVEL_WAVES.txt

So it looks like my profile command created a .txt instead of a .csv. What can I do now?

[coleramos425]

Hi Jonathan, thanks for submitting the question.

In Omniperf, there are two parts to a "successful" profiling run.

1. Generating rocprof input files (.txt)
   Based on the command line options given, Omniperf generates a set of optimal input files (.txt) to feed rocprofiler. These are saved to workloads/name/mi200/perfmon/

2. Querying rocprofiler to generate output (.csv)
   Input files (.txt) tell rocprofiler which counters to collect. Application is run, counters are collected, output data is saved to a .csv with the exact same filename in workloads/name/mi200/

Looks to me like you're getting past step (1) which is good, but you may be missing step (2) output (.csv). Is that correct?

[jpietarilagraham]

Yes. That's what happened. I should have looked at the slurm output first. Thanks!

[jpietarilagraham]

OK. My first attempt was killed after 2 hours. I've now been running for over 10 hours and the simulation itself took less than half an hour. Still not getting to step 2. Top shows I'm running python3. This seems a little long to me, but I'll check again in the morning. Thanks!

[IanBogle]

The profile step can take a while if you run it on a full-sized problem. rocprof has to run the simulation about 17 times to gather the complete set of counters. Merging the counters to produce the pmc_perf.csv can also take some time, depending on how many kernels are in your simulation. If possible, reducing the problem size or number of simulation steps can help, if the profiling data would still be representative of your use case.

[jpietarilagraham]

Thanks! Eventually, it did complete, but my problem was entirely something different.