runtime stats for application versions #3356
Comments
|
This concept could be useful where task runtimes are solely or predominantly determined by application version. But for many projects, task runtime is determined by the data being analysed - sometimes deterministically, sometimes with a strong degree of randomness. I'm surprised to see you quoting SETI in this context: the SETI project comes into the 'deterministic variablility' class (varies by angle range and search parameters). Has that been taken into account by your algorithm? |
|
That is true, it would only be useful if the average runtimes were consistent for a given batch. It would be a moving average, so it would adjust over time as batches changed. Also note that randomness is not a problem, as long as the mean of the runtimes makes sense. If the underlying distribution of runtimes is multi-modal then the mean would not be a good measure of anything. I actually was not quoting SETI in my original comment. I mentioned that "SETIguy" has already made the required changes, for no other reason than to give him credit, as he is the one who did the work. However, the fact that he was interested in this tells me that there might be an application for it at SETI, but I will let him speak to that. |
|
The version in the runtime_stats branch, it uses the same average as the PFC code, (an exponential running average) For the version of this currently running on the SETI@home Beta, I'm using a running median with roughly the same time constant as the exponential average, but it is far more stable, and does not require outlier exclusion except in the case that you could get several thousand outliers in a row. We could include that in this PR or have a separate PR. Of course, neither handles the case of a widely bifurcated distribution, but nothing would short of custom code to recognize that case (ala VLAR at S@H). |
|
Another approach (used, for example, in generating the web page comparing
GPU models)
is to enumerate a few thousand recently-completed and compute their stats.
This also lets you show histograms etc.
…On Tue, Oct 29, 2019 at 10:44 AM SETIguy ***@***.***> wrote:
The version in the runtime_stats branch, it uses the same average as the
PFC code, (an exponential running average) For the version of this
currently running on the ***@***.*** Beta, I'm using a running median with
roughly the same time constant as the exponential average, but it is far
more stable, and does not require outlier exclusion except in the case that
you could get several thousand outliers in a row. We could include that in
this PR or have a separate PR.
Of course, neither handles the case of a widely bifurcated distribution,
but nothing would short of custom code to recognize that case (ala VLAR at
***@***.***).
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#3356?email_source=notifications&email_token=AAHQVAMDBICDM565AKTBZBTQRBY6ZA5CNFSM4JGDOJI2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECROLSQ#issuecomment-547546570>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAHQVAPRKNZIGDJQS72XBNTQRBY6ZANCNFSM4JGDOJIQ>
.
|
|
SETI@home has a separate reason for wanting this that would build this
code. The number of processors (or fraction of a processor) used by GPU
and multithreaded apps is currently an app constant or workunit create time
computation. It's not tracked anywhere. The computation used with GPU
apps is always wrong (because of variation in GPU models, driver versions,
opencl versions). Having run time and CPU time in the app_version and
host_app_version tables will allow n_cpus to be computed on a per
host_app_version basis for use in the scheduler, allow proper app choice
based upon the app efficiency, and will allow proper credit to be
calculated in the validator for multithreaded/multiprocessing apps. I
haven't submitted an issue for this (or looked to see if one already exists.
…On Tue, Oct 29, 2019 at 3:38 AM RichardHaselgrove ***@***.***> wrote:
This concept could be useful where task runtimes are solely or
predominantly determined by application version.
But for many projects, task runtime is determined by the data being
analysed - sometimes deterministically, sometimes with a strong degree of
randomness.
I'm surprised to see you quoting SETI in this context: the SETI project
comes into the 'deterministic variablility' class (varies by angle range
and search parameters). Has that been taken into account by your algorithm?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#3356?email_source=notifications&email_token=ACS5ZMVH6JZJAOOA55LI5G3QRAHA3A5CNFSM4JGDOJI2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECQAMZA#issuecomment-547358308>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACS5ZMSD4UTFXT56WWDRTNTQRAHA3ANCNFSM4JGDOJIQ>
.
--
Eric Korpela
korpela@ssl.berkeley.edu
AST:7731^29u18e3
|
|
@davidpanderson |
It would be nice to keep track of average run times and cpu times for each application version in the database.
This information could then be displayed directly in apps.php (and possibly other places). This would give desirable information for both project developers and users. For example, you could compare average run times between 32 bit app versions and 64 bit app versions; or between a CPU version and a GPU version.
To implement this, 4 new fields would need to be added to both the app_version table and the host_app_version table:
runtime_nruntime_avgcputime_ncputime_avgThese would then be tracked in the validator. Impact to server performance would be negligible.
SETIguy created a branch called runtime_stats, where the above solution was implemented as a proof of concept. I have been using it for several months in a private test server and it works very well. Here is a screenshot showing the runtimes in apps.php:
The text was updated successfully, but these errors were encountered: