Performance testing

[paulf81]

I was hoping to start a quick discussion on performance testing. Let's say also this applies to both FLORIS and FLASC. Some of the improvements we consider are aimed at performance but I'm not always positive the best way to assess that performance. I know some NREL codes have automatic performance testing, but I wasn't sure the lift it would take to bring that to FLORIS/FLASC, or if we even want that. So wanted to take a pulse of what people thought of having some standard performance tests in these codes?

@rafmudaf @bayc @ejsimley @Bartdoekemeijer @misi9170 @RHammond2

[rafmudaf]

FWIW I have a script that I was using to get an idea of the performance changes over time. I haven't updated it in a while, but the plots throughout the development of v3 are in the docs: https://nrel.github.io/floris/code_quality.html. I'll work on adding at least all of the releases to that chart.

Long term, it would be nice to have some automation to display the performance in a pull request on command. For example, I'd like to be able to add comment to a pull request to run these scripts and have the results added to another comment.

[rafmudaf]

I had added something like an automated performance reporting system for OpenFAST through GitHub Actions, but I found that the results varied greatly between jobs even if nothing in the code changed. For that project at least, the variability of GitHub runners was too high to get meaningful information on performance. It may not be an issue with FLORIS since this is less computationally intensive, but it is worth considering that we don't have much control of the hardware resources available on GitHub.

[paulf81]

that makes a lot of sense, I was thinking this was trickier to do, thanks for these thoughts, I was even thinking an example of how to time things could be useful to have, but then I find it difficult to do before/after comparisons unless the timing script exists in both branches

[Bartdoekemeijer]

That's really interesting @rafmudaf. I can imagine even with the exact same machine, the timing of the exact same code 12 months apart may even be substantially different due to driver updates, other applications consuming resources in the background, or even thermal throttling. Would there be value to doing performance tests over a number of commits in a single batch, so on the same machine under the same circumstances?

In pseudo-code,

for i in range(10):  # Perform benchmark test for last 10 commits
    git checkout HEAD~$i  # Check i commits back from the latest
    python performance_tests.py benchmark_results_$i.txt # Benchmark this commit

This would not necessarily have to be evaluated by GitHub Actions, but instead can be something that we do ourselves internally between larger releases and we can manually publish the results on the repo?

[paulf81]

I like that idea

[rafmudaf]

As another point of consideration, I think we talk of performance at two scales: high level parallelization of many data points (i.e. atmospheric conditions) and low level vectorization or parallelization of a single data point. If we develop performance tests, it would be good to test at both resolutions.
Is there any other "type" of performance that I'm missing?

[rafmudaf]

Check out this discussion topic: #293

[rafmudaf]

The results are out of date, at this point, but it starts to document how to do this sort of profiling. I've found this to be pretty challenging to do at scale in the past, but this is certainly an area that could use a better process and better documentation.

[paulf81]

That's helpful, yeah, I can now see this will be a challenge, maybe something worthy the next code sprint?

[RHammond2]

I think what would be helpful here, and I know Raf you've got some of this started, and I did a bit of building on that in my one of my PRs, is to understand where things are meaningfully speeding up. Because we have three separate considerations: wind directions, wind speeds, and number of turbines; and, the combinations of these, I think we'd want to know how something is impacting run time. For instance, is the 20% speedup on a somewhat trivial case and only 5% something more computationally intensive, or vice versa?

However, I'll also admit that this adds quite a bit of extra computational requirements (especially if we include GCH) to a CI pipeline if it's implemented there, or for someone to run all of these cases on their now. So I'd probably lean towards Paul's suggestion of including this in a code sprint with two separate sessions 1) to have a focus group figure out what should be prioritized for the tests and how should they be presented so that everyone can easily figure out performance changes and then 2) have one or two people implement the solutions for a CI pipeline and reporting process for a PR.

[paulf81]

That sounds good to me, to include digging into this work as a focused activity of the next code spring

[paulf81]

hi @rafmudaf just following up on our discussion this morning, I think we should test a few things, here is my idea:

• Smaller farm: 3x3 turbines (or smaller?)
  • Testing the full reinitialize -> calc_wake path
  • Testing reinitialize only
  • Testing calls to just calc_wake
• Larger farm: (maybe 50 turbines?)
  • Testing the full reinitialize -> calc_wake path
  • Testing reinitialize only
  • Testing calls to just calc_wake

[rafmudaf]

Thanks @paulf81! An additional point for performance characterization is to consider the scaling across each dimension of the flow field arrays (wind direction, wind speed, n turbines). So you've proposed identifying the impact of scaling the n-turbines dimension, and we could expand to testing the other two.

Thinking through the question that would be answered by each of these suggestions, here's what I understand:

• Testing the full reinitialize -> calc_wake path - How does a change to the code the impact on the full system performance? Is this equal to the sum of the next two?
• Testing reinitialize only - How does a change to the code impact the memory initialization and preprocessing such as rotations, gridding, etc?
• Testing calls to just calc_wake - How does a change to the code impact the performance of the wake models and solvers?

@paulf81 Is this correct and is there any to add? Also, seeing that the test cases might be the same for each size of n-turbines, it seems we might be able to define a set of attributes to apply to all sets of configuration (dimension scaling or other use cases).

ping @bayc if you have more to add here.

[paulf81]

I think that's all right, and I agree with you, if it doesn't break the bank, testing on the other scaling dimensions of ws/wd would be really great! Thank you @rafmudaf !