Adding scripts and custom providers (dotnet#172)

docs/README.md

@@ -15,6 +15,8 @@
 |-------|------------|
 |**[Storing results](storing_results.md)** | Storing results locally or in a SQL Server database.
 |**[Using different .NET versions](dotnet_versions.md)** | Benchmarking with different .NET versions.
+|**[Collecting event counters](event_counters.md)** | Collecting predefined and custom event counters.
+|**[Post-processing results](post_processing.md)** | Adding custom results and running scripts.
 
 
 ## Reference documentation

docs/event_counters.md

@@ -0,0 +1,159 @@
+## Description
+
+This guide shows how to collect standard and custom dotnet event counters.
+
+Crank is able to record any predefined set of event counters, the same way [dotnet-counters](https://docs.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-counters) does, but without the need of spawning a different sidecar process. Another advantage over dotnet-counters is that it can also be configured to record custom event counters and apply build custom statistics out of it (max, min, percentiles, ...).
+
+## Prerequisites
+
+1. You should have followed the [Getting Started](getting_started.md) tutorial, and have `crank` and `crank-agent` tools available.
+
+## Collecting event counters
+
+The following command line will run a benchmark and record the event counters exposed by the `System.Runtime` provider.
+
+```
+crank --config /crank/samples/hello/hello.benchmarks.yml --scenario hello --profile local --application.counterProviders System.Runtime --chart
+```
+
+```
+| application                   |            |                                     |
+| ----------------------------- | ---------- | ----------------------------------- |
+| CPU Usage (%)                 | 38         |           ▃▇▇▇▆▆▆▇▆▆▆▅█▇▇█▇▇▇▅▃     |
+| Raw CPU Usage (%)             | 300.46     |           ▃▇▇▇▆▆▆▇▆▆▆▅█▇▇█▇▇▇▆▃     |
+| Working Set (MB)              | 89         | ▃▃▃▃▃▃▃▃▃▃▆▇████████████████████    |
+| Build Time (ms)               | 5,358      |                                     |
+| Start Time (ms)               | 564        |                                     |
+| Published Size (KB)           | 86,753     |                                     |
+| .NET Core SDK Version         | 5.0.100    |                                     |
+| Max CPU Usage (%)             | 35         |            ▂█▇█▆▆██▆█▅▇▇▇███▇█▇▅    |
+| Max Working Set (MB)          | 93         | ▃▃▃▃▃▃▃▃▃▃▃▅▇▇█████████████████████ |
+| Max GC Heap Size (MB)         | 39         |            ▄▅▆▆▁▂▄▆▅▇▂▁▂▂▃▂▂█▅▆▂▂▂▂ |
+| Max Number of Gen 0 GCs / min | 3.00       |             ▅▅▅█▅▅▅▅▅█▅▅▅▅▅▅▃▅▅▅    |
+| Max Number of Gen 1 GCs / min | 1.00       |             █    █                  |
+| Max Number of Gen 2 GCs / min | 1.00       |                  █                  |
+| Max Time in GC (%)            | 1.00       |                             █       |
+| Max Gen 0 Size (B)            | 192        |             ███████████████████████ |
+| Max Gen 1 Size (B)            | 3,272,160  |                 █                   |
+| Max LOH Size (B)              | 134,392    |             ███████████████████████ |
+| Max Allocation Rate (B/sec)   | 72,372,048 |            ▂▇███▇██▇█▇▇▇██▇▇▇▆▆▅    |
+| Max GC Heap Fragmentation     | NaN        |                                     |
+| # of Assemblies Loaded        | 97         | ▇▇▇▇▇▇▇▇▇▇▇████████████████████████ |
+| Max Exceptions (#/s)          | 1,019      |                 █              █    |
+| Max Lock Contention (#/s)     | 34         |                 █▁▁ ▁  ▁  ▁ ▁▁▁█    |
+| Max ThreadPool Threads Count  | 23         | ▁▁▁▁▁▁▁▁▁▁▁▆▆▆▆▆██▆▆█▆▆▇▆▆▇▆▇▆▇▇▇▇▇ |
+| Max ThreadPool Queue Length   | 101        |            █  ▁▃   ▁     ▁  ▁       |
+| Max ThreadPool Items (#/s)    | 191,763    |            ▁▆██▇▆██▇█▇▇▇████▇▆▆▅    |
+| Max Active Timers             | 0          |                                     |
+| IL Jitted (B)                 | 168,412    | ▁▂▂▂▂▂▂▂▂▂▂▃▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇███ |
+| Methods Jitted                | 1,927      | ▁▂▂▂▂▂▂▂▂▂▂▃▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇███ |
+```
+
+The chart on the last column are created from all the data points over the benchmark.
+
+## List of pre-defined counter providers:
+
+The following providers are pre-defined in crank:
+
+- System.Runtime
+- Microsoft-AspNetCore-Server-Kestrel
+- Microsoft.AspNetCore.Hosting
+- System.Net.Http
+- Microsoft.AspNetCore.Http.Connections
+- Grpc.AspNetCore.Server
+- Grpc.Net.Client
+- Npgsql
+
+## Adding custom providers
+
+The providers are defined in the same config files that contain the scenarios of job definitions, under the `counters` section.
+
+Here is an example defining a subset of the `System.Runtime` counters:
+
+```yml
+counters:
+- provider: System.Runtime
+  values: 
+  - name: cpu-usage
+    measurement: runtime-counter/cpu-usage
+    description: Percentage of time the process has utilized the CPU (%)
+
+  - name: working-set
+    measurement: runtime-counter/working-set
+    description: Amount of working set used by the process (MB)
+```
+
+Using this format, you can add the definition of a counter you are exposing, and all its values will be recorded as measurements.
+
+## Building results out of measurements
+
+When an event counter is recorded, it creates timestamped measurements. There can be multiple data points of the same measurement. To build a result that is displayed in the summary table, and saved in results files, you need to define a __result__ that will desribe how to transform the set of measurements in a single value, usually the max.
+
+In any configuration file, results can be defined under the `results` section like this:
+
+```yml
+results:
+
+# System.Runtime counters
+- name: runtime-counter/cpu-usage
+  measurement: runtime-counter/cpu-usage
+  description: Max CPU Usage (%)
+  format: n0
+  aggregate: max
+  reduce: max
+  
+- name: runtime-counter/working-set
+  measurement: runtime-counter/working-set
+  description: Max Working Set (MB)
+  format: n0
+  aggregate: max
+  reduce: max
+```
+
+Multiple results can be computed for the same measurements. The following example adds a new result that computes the 95th of the working set:
+
+```yml
+- name: runtime-counter/working-set/95
+  measurement: runtime-counter/working-set
+  description: Max Working Set (MB)
+  format: n0
+  aggregate: percentile95
+  reduce: max
+```
+
+- The `aggregate` operation defines how to group the results that come from a single source.
+- The  `reduce` operation defines how to group the aggregated results from different sources. This is less usual as it requires a job to run on multiple nodes, for instance when splitting a web load on multiple clients.
+
+### Available operations
+
+The following operations are available by default:
+
+- max
+- min
+- last
+- first
+- all
+- avg
+- sum
+- median
+- count
+- delta
+- percentile99
+- percentile95
+- percentile90
+- percentile75
+- percentile50
+
+## Defining custom operations
+
+Any configuration file can add custom operations using javascript, in the `defaultScript` section.
+
+The following example defines the `max` operation:
+
+```yml
+defaultScripts:
+  - |
+    function max(values) {
+      return Math.max(...values);
+    }
+```

docs/post_processing.md

@@ -0,0 +1,76 @@
+## Description
+
+This guide shows how to execute custom scripts over the results and measurements.
+
+Once a benchmark is finished, the results (compured values our of measurements) and the data points (timestamped measurements) can be saved locally or in a database. However it might be necessary to compute custom results, alter existing ones, or even add custom properties.
+
+Crank provides the ability to define custom post-processing scripts that can be invoked before the results are stored, and reused across runs.
+
+## Prerequisites
+
+1. You should have followed the [Getting Started](getting_started.md) tutorial, and have `crank` and `crank-agent` tools available.
+
+## Defining custom scripts
+
+Custom scripts are defined in the `scripts` section of a configuration file.
+In a script, the `benchmarks` property is available for read/write access, and represent the JSon document as it would be save on disk with the option `--output [filename]`.
+
+The following configuration snippet demonstrates how to add a custom property to the `properties` element of the results:
+
+```yml
+scripts:
+  add_current_time: |
+    benchmarks.properties["time"] = new Date().toISOString();
+```
+
+Custom scripts are invoked from the command line with the `--script` option like so:
+
+```
+crank --config /crank/samples/hello/hello.benchmarks.yml --scenario hello --profile local --script add_current_time
+```
+
+Multiple scripts can be invoked from the command line by invoking the argument multiple times.
+All the script are executed in the same JavaScript context, such that top level variables are shared across script invocations.
+
+## Defining global scripts
+
+A global script is one that is executed automatically when a configuration file is loaded. It can be useful when named scripts need to share some common function or variables.
+The following script creates a function to compute percentiles of a set of values.
+
+__percentile.config.yml__
+
+```yml
+defaultScript:
+  - |
+    console.log("this section is loaded by default and before named scripts")
+
+    function percentile(items, th) {
+      var ordered = items.sort((a, b) => a - b); // by default sort() uses ordinal comparison
+      index = Math.max(0, Math.round(ordered.length * th / 100) - 1);
+      return ordered[index];
+    }
+```
+
+By running a benchmark with the `--config percentile.config.yml` the text _this section is loaded by default and before named scripts_ would be displayed, and the `percentile()` function would be available for any other scripts that are invoked.
+
+## Adding custom results
+
+Some values might be the results of results coming from different source. The following examples show how to add a new result to the __application__ job by using one that is in the __load__ job.
+
+```yml
+  add_allocations_per_request: |
+    var allocations = benchmarks.jobs.application.results["runtime-counter/alloc-rate"]
+    var rps = benchmarks.jobs.load.results["wrk/requests"];
+    benchmarks.jobs.application.results["alloc-per-request"] = allocations / rps;
+```
+
+## Logging
+
+A custom `console` object is made available and support the following methods:
+
+- log(message)
+- info(message)
+- warn(message)
+- error(message)
+
+These methods will use different colors to render the message, respectively default, green, yellow and red.

samples/scripts/scripts.benchmarks.yml

@@ -0,0 +1,37 @@
+defaultScripts:
+  - |
+    console.log("this section is loaded by default and before named scripts")
+
+    function percentile(items, th) {
+      var ordered = items.sort((a, b) => a - b); // by default sort() uses ordinal comparison
+      index = Math.max(0, Math.round(ordered.length * th / 100) - 1);
+      return ordered[index];
+    }
+
+scripts:
+
+  # displays a message using all different colors
+  say_hello: |
+    console.log("hello")
+    console.info("world")
+    console.warn("this is")
+    console.error("scary")
+
+  # records the current date and time as a custom property
+  add_current_time: |
+    benchmarks.properties["time"] = new Date().toISOString();
+
+  # calculats the allocations per request by using results from application and load
+  add_allocations_per_request: |
+    var allocations = benchmarks.jobs.application.results["runtime-counter/alloc-rate"]
+    var rps = benchmarks.jobs.load.results["wrk/requests"];
+    benchmarks.jobs.application.results["alloc-per-request"] = allocations / rps;
+
+  # computes the 95th percentile of the working set from all the data points (measurements)
+  add_p95_memory: |
+    var memories = benchmarks.jobs.application.measurements[0]
+      .filter(m => m.name == "benchmarks/working-set")
+      .map(x => x.value);
+    var value = percentile(memories, 95);
+    console.info(`p95 of working set is ${value}`);
+    benchmarks.jobs.application.results["benchmarks/working-set/p95"] = value;