Force inline par_for_inner

[jdolence]

PR Summary

This PR just changes KOKKOS_INLINE_FUNCTION to KOKKOS_FORCEINLINE_FUNCTION for all of our par_for_inner overloads.

In a downstream code, we found that a particular loop was failing to vectorize when using the par_for_inner that corresponds to a single simd for loop, whereas just using the raw simd for loop directly resulted in vectorization (and a 40% speedup of the whole code!). Changing INLINE to FORCEINLINE on the par_for_inner resolves this, suggesting that the compiler was making the obnoxious choice not to inline and then (presumably) failing to vectorize what I can only guess it thought was a function call.

PR Checklist

• Code passes cpplint
• New features are documented.
• Adds a test for any bugs fixed. Adds tests for new features.
• Code is formatted
• Changes are summarized in CHANGELOG.md
• CI has been triggered on Darwin for performance regression tests.
• Docs build
• (@lanl.gov employees) Update copyright on changed files

[pdmullen]

I approve, although I expect you may get some pushback from others in the collab 😉

[lroberts36]

LGTM. Amazed that this change provides a 40% speedup.

[jdolence]

I approve, although I expect you may get some pushback from others in the collab 😉

@pgrete any objections?

[Yurlungur]

This is one of the wackier failures of the compiler I've seen.

[Yurlungur]

This seems like a very trivial change in keeping with the original intent, so I'm pressing the button

[pgrete]

IIRC the original motivation why we went away from force inline (what we had originally) to just inline was that the (now legacy) Intel compiler was not able to compile the code any more.
Might be worth double checking.

[Yurlungur]

IIRC the original motivation why we went away from force inline (what we had originally) to just inline was that the (now legacy) Intel compiler was not able to compile the code any more. Might be worth double checking.

@jdolence did you run your tests of this performance with legacy intel?