Find Heap Bound

[jackkoenig]

This refactors benchmark/scripts/benchmark_cold_compile.py a little bit and adds an additional script for measuring information about FIRRTL runs: benchmark/scripts/find_heap_bound.py

The basic idea is that you provide a starting heap size (passed via -Xmx), a minimum step size, and JVM arguments and the script will do a binary search to find the minimum allowable heap size. There are sensible defaults for starting heap size and minimum step.

Note that this isn't specific to FIRRTL, you can use this for any java program. The only java specific aspect is that it's using -Xmx.

Use

Help text

$ ./benchmark/scripts/find_heap_bound.py -h
usage: find_heap_bound.py [-h] [--verbose] [--start-size START_SIZE]
                          [--min-step MIN_STEP] [--java JAVA]
                          [--timeout-factor TIMEOUT_FACTOR]
                          args [args ...]

positional arguments:
  args                  Arguments to JVM, include classpath and main

optional arguments:
  -h, --help            show this help message and exit
  --verbose, -v         Increase verbosity level (cumulative) (default: 0)
  --start-size START_SIZE
                        Starting heap size (default: 4G)
  --min-step MIN_STEP   Minimum heap size step (default: 100M)
  --java JAVA           Java executable to use (default: java)
  --timeout-factor TIMEOUT_FACTOR
                        Multiple of wallclock time of first successful run
                        that counts as a timeout, runs over this time count as
                        a fail (default: 4.0)

Example

./benchmark/scripts/find_heap_bound.py --start-size 16G -- -cp utils/bin/firrtl.jar firrtl.stage.FirrtlMain -i regress/Rob.fir -o out -X verilog
Xmx,Max RSS (MiB),Wall Clock (s),User Time (s),System Time (s)
16G,2128,27.64,60.26,5.33
8G,2195,29.08,61.75,4.92
4G,1759,27.67,62.17,4.36
2G,1349,29.29,68.5,4.1
1G,1200,23.02,70.16,2.24
500M,765,27.86,101.59,3.76
400M,674,27.26,111.51,3.91
300M,-,-,-,-
$ echo "
> Xmx,Max RSS (MiB),Wall Clock (s),User Time (s),System Time (s)
> 16G,2128,27.64,60.26,5.33
> 8G,2195,29.08,61.75,4.92
> 4G,1759,27.67,62.17,4.36
> 2G,1349,29.29,68.5,4.1
> 1G,1200,23.02,70.16,2.24
> 500M,765,27.86,101.59,3.76
> 400M,674,27.26,111.51,3.91
> 300M,-,-,-,-
> " | column -t -s,
Xmx   Max RSS (MiB)  Wall Clock (s)  User Time (s)  System Time (s)
16G   2128           27.64           60.26          5.33
8G    2195           29.08           61.75          4.92
4G    1759           27.67           62.17          4.36
2G    1349           29.29           68.5           4.1
1G    1200           23.02           70.16          2.24
500M  765            27.86           101.59         3.76
400M  674            27.26           111.51         3.91
300M  -              -               -              -

This is super useful for examining the tradeoff between heap size, runtime (wall clock) and user time which shows how much extra GC pressure there is.

Future work

It might be nice to search for heap sizes based on different optimization criteria (the current one being "works"). In particular, I think a target user time or perhaps ratio of user time / wall clock time would help find better bounds than just "does it work".

Contributor Checklist

• Did you add Scaladoc to every public function/method?
• Did you update the FIRRTL spec to include every new feature/behavior?
• Did you add at least one test demonstrating the PR?
• Did you delete any extraneous printlns/debugging code?
• Did you specify the type of improvement?
• Did you state the API impact?
• Did you specify the code generation impact?
• Did you request a desired merge strategy?
• Did you add text to be included in the Release Notes for this change?

Type of Improvement

• new feature/API (for developers)

API Impact

None

Backend Code Generation Impact

None

Desired Merge Strategy

• Rebase: You will rebase the PR onto master and it will be merged with a merge commit.

Release Notes

Add script for measuring heap allocation vs. performance and memory use

Reviewer Checklist (only modified by reviewer)

• Did you add the appropriate labels?
• Did you mark the proper milestone (1.2.x, 1.3.0, 1.4.0) ?
• Did you review?
• Did you check whether all relevant Contributor checkboxes have been checked?
• Did you mark as Please Merge?

[jackkoenig]

Also protip, if you paste a CSV into Excel, and then do the whole "Text to Columns" thing, you can copy paste that table into Github and it will autoformat as a table:

Xmx	Max RSS (MiB)	Wall Clock (s)	User Time (s)	System Time (s)
16G	2128	27.64	60.26	5.33
8G	2195	29.08	61.75	4.92
4G	1759	27.67	62.17	4.36
2G	1349	29.29	68.5	4.1
1G	1200	23.02	70.16	2.24
500M	765	27.86	101.59	3.76
400M	674	27.26	111.51	3.91
300M	-	-	-	-

Not sure if this works with GSuite

[seldridge]

Looks good.

Minor comments throughout.

[jackkoenig]

@seldridge I have incorporated your feedback and rebased on master. I also added another command-line argument --timeout-factor (default 4.0) that allows the script to pick a timeout for java runs since sometimes the JVM can thrash when it's very close to the limit. I had a run of this run all night because of this issue so I figured a timeout is important.