Question (but Possible Feature): OperationSetupAttribute?

[Mike-E-angelo]

Hello Community,

I was re-reading #730 and was curious why there isn't an OperationSetupAttribute? Is there something that is keeping additional code from being executed with each operation? Besides performance, that is. 😆

Thank you for any consideration/insight you can provide here.

[adamsitnik]

Hi @Mike-EEE

IterationSetup works like OperationSetupAttribute. We decided to change the behavior of IterationSetup because as Stephen has suggested this is what the users were expecting from the beginning.

We also execute many operations in a single iteration by default to:

1. minimize the noise (one bad sample amongst many other samples in single iteration is not going to have a huge impact on the final result)
2. allow for accurate nano-benchmarks (Stopwatch.Start() + Stopwatch.Stop() would take more than the single nano-benchmark invocation and it would spoil the results)

[Mike-E-angelo]

Ah ok @adamsitnik thank you for your time here. Perhaps I am confused, then. From looking at the remarks of #730, it is my understanding (and experience, which is what made me look this up in the first place 😆) that the IterationSetup attribute limits execution to 1op per iteration. Is this not correct? What I am looking for is to be able to run as many ops as possible per iteration and run some code beforehand that is not measured. This line and this line (same difference) are good examples.

Based on my experience yesterday, I was using WithUnrollFactor(16) and the code in IterationSetup seemed to only execute once per iteration, leading to reported times that dropped from ms to ns. I could have had something misconfigured, however.

FWIW I was in favor of #730 as it made sense to me as well. However, after reading this page it cleared up a lot for me in terms of how this works now.

[adamsitnik]

that the IterationSetup attribute limits execution to 1op per iteration.

this is correct

run some code beforehand that is not measured.

would you like to run this code before every benchmark invocation? or before every iteration that would execute the benchmark many times?

However, after reading this page it cleared up a lot for me in terms of how this works now.

awesome! 👍

[Mike-E-angelo]

would you like to run this code before every benchmark invocation? or before every iteration that would execute the benchmark many times?

So in my case, I would like to reset the stream that I am testing before any code is run. This has to occur every time before the operation is invoked or else I get the drastically different results as noted above. Essentially, the code runs on a finished stream which means no data which means no compute. :)

[adamsitnik]

Now I understand.

If your operation takes milliseconds or more, you should just use [IterationSetup].
If your operation takes nanoseconds there is no way of doing this. Your benchmark should contain the stream.Postion = 0 and include the price of it. If you compare few different approaches of solving given problem all of them should include the same "artificial overhead".

[Mike-E-angelo]

OK... that's what I was thinking. Is there a technical barrier that keeps from creating and using an OperationSetupAttribute? To me, it makes sense to have a setup for both phases so that I have the ability to not only set up functionality per iteration (or grouping of operations), but also to cleanly separate out any functionality that I do not want to be recorded in the individual benchmarks (or singular operations).

Now that I have a better understanding of the pipeline this makes sense to me.

[adamsitnik]

@Mike-EEE so today we have:

[GlobalSetup]
[IterationSetup]

and you would like to have:

[GlobalSetup]
[IterationSetup]
[OperationSetup]

? We just assumed that everybody wanted the IterationSetup to work like OperationSetup and this is why we did not introduce a new one.

[Mike-E-angelo]

So you want to run an open source project. 😂

Hey, it's your model. I think it's pretty rad and it makes sense. The ask here isn't for me per se, but for those who might want to follow your model as such. In my case, I do want that and I would definitely take advantage of both scenarios now that I understand what they actually mean. 😁

But, that said I can understand the frustration here in saying one thing and then asking for another.

The big thing for me is that while the IterationSetup "acts" like an OperationSetup, the amount of data that it provides now is nothing like what a "true" OperationSetup would actually provide. That is (if I understand correctly) if I use IterationSetup I am only getting one point of data per iteration, whereas with a true OperationSetup I would get n-points of data, depending on how many operations were fired in the iteration.

If you can talk me out of this data gap then I for one would feel better about walking down from this.

[adamsitnik]

When it comes to OperationSetup for nano-benchmarks it would hurt our accuracy. It would introduce cache misses and it could definitely affect the generated assembly code (like assembly registers usage). Our every iteration consists of MANY calls to the benchmarked method. We want to keep this code "tight", without any interruptions.

For non-nano-benchmarks the [IterationSetup] should be enough. For nano-benchmarks OperationSetup is impossible or almost impossible when we want to have really good accuracy.