Leaderboard implementation of tabular Q #141
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
facebook-github-bot
added
the
CLA Signed
|
The development branch has a new dependency that the version on pip does not have, so the module is missing. Run Cheers, |
|
Thanks, it works! Funny to call a package |
|
I'm facing this error during the evaluation: Not sure why this is happening, also the iteration seems to slow down. Is there a way to identify the issue? @ChrisCummins |
|
this seems to happen consistently at 30th evaluations. |
Yep, that's a strange one. The fact that it happens consistently on the 30th evaluation implies that it's a problem with the third benchmark (the benchmarks are always evaluted in the same order, and the default Can you confirm that you're running this via bazel on the latest development branch? Cheers, |
* [llvm] Temporarily disable polybench Mitigates facebookresearch#55. * [validation] Add a flakiness retry loop around validation. Add a retry loop around the granular individual validation callbacks for cBench-v1. Mitigates facebookresearch#144. * [validation] Catch timeouts in retry loop. * [docs/faq] "I updated with 'git pull' and not it doesn't work * [tests] Extend timeout on datasets test. * [tests] Update regression tests. * [tests] Reduce validation regression test retry counts. * Call env.reset() just after creation - fixes facebookresearch#150 * [docs] Update LLVM actions table. * Add a target to rename the manylinux file. * Force UTF-8 on README decoding. * [util] Improve runfiles docstrings. * [llvm] Remove LLVM binaries from wheel. This patch removes the LLVM binaries from the shipped wheel. This is to reduce the package size to be under the 100MB default maximum imposed by PyPi. Instead of shipping the files in the wheel, the LLVM binaries are downloaded from an archive hosted by Facebook when needed. The circumstances for needing them are: (1) starting an LLVM service, (2) attempting to resolve the path to an LLVM binary. * Defer evaluation of cBench runtime data directory. * [tests] Remove tests that overwrite site data path. These no longer work now that site data requires LLVM binaries to be present. * Release v0.1.5. * Add a fast path check for downloaded LLVM files. * [tests] Use full URI for benchmark. * Correct retry count in error message. * [env] Include last error on init failure. * [env] Add a special error message for UNKNOWN errors. * [rpc] Allow loglines() when logs directory does not exist. * [rpc] Include service logs in error message on init failure. * [rpc] Include final error message on retry loop failure. * [rpc] Add decoded signal name on init error. * [llvm] Replace DCHECK() with Status error. * [tests] Remove tests that interfere with site data path. Site data directory is now a pre-requisite of the LLVM environment and cannot be moved. * [tests] Fix caught exception type. * [llvm] Add a check for runfile requirement. * [llvm] Add a file existing check. * [rpc] Disable logs buffering on debugging runs. * [tests] Fix error message comparison tests * [bin/manual_env] Update prompt after reset(). Running `reset()` with no benchmark set will select a random program, so the prompt must be updated. * [tests] Add workaround for prompt issue. * Release v0.1.6. This release focuses on hardening the LLVM environments, providing improved semantics validation, and improving the datasets. Many thanks to @JD-at-work, @bwasti, and @mostafaelhoushi for code contributions. - [llvm] Added a new `cBench-v1` dataset which changes the function attributes of the IR to permit inlining. `cBench-v0` is deprecated and will be removed no earlier than v0.1.6. - [llvm] Removed 15 passes from the LLVM action space: `-bounds-checking`, `-chr`, `-extract-blocks`, `-gvn-sink`, `-loop-extract-single`, `-loop-extract`, `-objc-arc-apelim`, `-objc-arc-contract`, `-objc-arc-expand`, `-objc-arc`, `-place-safepoints`, `-rewrite-symbols`, `-strip-dead-debug-info`, `-strip-nonlinetable-debuginfo`, `-structurizecfg`. Passes are removed if they are: irrelevant (e.g. used only debugging), if they change the program semantics (e.g. inserting runtimes bound checking), or if they have been found to have nondeterministic behavior between runs. - Extended `env.step()` so that it can take a list of actions that are all performed in a single batch. This improve efficiency. - Added default reward spaces for `CompilerEnv` that are derived from scalar observations (thanks @bwasti!) - Added a new Q learning example (thanks @JD-at-work!). - *Deprecation:* The next release v0.1.5 will introduce a new datasets API that is easier to use and more flexible. In preparation for this, the `Dataset` class has been renamed to `LegacyDataset`, the following dataset operations have been marked deprecated: `activate()`, `deactivate()`, and `delete()`. The `GetBenchmarks()` RPC interface method has also been marked deprecated.. - [llvm] Improved semantics validation using LLVM's memory, thread, address, and undefined behavior sanitizers. - Numerous bug fixes and improvements. * [tests] Add temporary workaround for flaky init benchmark. * Add missing copyright header to make_specs.py. * [util] Force string type in truncate(). * [bin/service]: Fix reporting of observation space shape. * [bin/service]: Fix reporting of observation space shape. * [util] Force string type in truncate(). * [llvm] Add an InstCount observation space. This adds new observation spaces that expose the -instcount pass values. The -instcount pass counts the number of instructions of each type in a program, along with the total number of instructions, total number of blocks, and total number of functions. There are four new observation spaces: `InstCount`, which returns the feature vector as a numpy array, `InstCountDict`, which returns the values as a dictionary of named features, and `InstCountNorm` and `InstCountNormDict`, which are the same as above but the counts are instead normalized to the total number of instructions in the program. Example usage: >>> import gym >>> import compiler_gym >>> env = gym.make("llvm-v0") >>> env.observation_space = "InstCountDict" >>> env.reset("cBench-v0/crc32") {'TotalInstsCount': 196, 'TotalBlocksCount': 29, 'TotalFuncsCount': 13, 'RetCount': 5, 'BrCount': 24, 'SwitchCount': 0, 'IndirectBrCount': 0, 'InvokeCount': 0, 'ResumeCount': 0, 'UnreachableCount': 0, 'CleanupRetCount': 0, 'CatchRetCount': 0, 'CatchSwitchCount': 0, 'CallBrCount': 0, 'FNegCount': 0, 'AddCount': 5, 'FAddCount': 0, 'SubCount': 0, 'FSubCount': 0, 'MulCount': 0, 'FMulCount': 0, 'UDivCount': 0, 'SDivCount': 0, 'FDivCount': 0, 'URemCount': 0, 'SRemCount': 0, 'FRemCount': 0, 'ShlCount': 0, 'LShrCount': 3, 'AShrCount': 0, 'AndCount': 3, 'OrCount': 1, 'XorCount': 8, 'AllocaCount': 24, 'LoadCount': 51, 'StoreCount': 38, 'GetElementPtrCount': 5, 'FenceCount': 0, 'AtomicCmpXchgCount': 0, 'AtomicRMWCount': 0, 'TruncCount': 1, 'ZExtCount': 5, 'SExtCount': 0, 'FPToUICount': 0, 'FPToSICount': 0, 'UIToFPCount': 0, 'SIToFPCount': 0, 'FPTruncCount': 0, 'FPExtCount': 0, 'PtrToIntCount': 0, 'IntToPtrCount': 0, 'BitCastCount': 0, 'AddrSpaceCastCount': 0, 'CleanupPadCount': 0, 'CatchPadCount': 0, 'ICmpCount': 10, 'FCmpCount': 0, 'PHICount': 0, 'CallCount': 13, 'SelectCount': 0, 'UserOp1Count': 0, 'UserOp2Count': 0, 'VAArgCount': 0, 'ExtractElementCount': 0, 'InsertElementCount': 0, 'ShuffleVectorCount': 0, 'ExtractValueCount': 0, 'InsertValueCount': 0, 'LandingPadCount': 0, 'FreezeCount': 0} The InstCount observation spaces are quick to compute and lightweight. They have similar computational complexity as Autophase. Fixes facebookresearch#149. * [ci] Enable test workflows on Python 3.9. Issue facebookresearch#162. * Bump grpcio from 1.34 to 1.36. Issue facebookresearch#162. * Bump bazel requirement to 4.0.0. This is required to build grpcio 1.36.0. Issue facebookresearch#162. * [ci] Reverse order of sudo in setup. Issue facebookresearch#162. * Add libjpeg-dev to list of required linux packages. This to enable compiling Pillow from source on Python 3.9. Issue facebookresearch#162. * Bump the gym dependency to 0.18.0. Issue facebookresearch#162. * [examples] Fix initialization of temporary directory variable. * Add zlib to macOS dependencies. This is to fix compilation of Pillow using Python 3.9. Issue facebookresearch#162. * [readme] Recommend python 3.9 for conda environments. Issue facebookresearch#162. * [ci] Use python 3.9 for continuous integration jobs. Issue facebookresearch#162. * [setup.py] Add a list of supported python versions * [setup.py] Bump development status to Alpha. * [README] Use non-sudo instructions for linux setup. * [README] Simplify table of contents. This adds <!-- omit from toc --> annotations to some of the minor subheadings to keep the table of contents as simple as possible. This uses the "Markdown All in One" plugin for VSCode to automatically keep the table of contents up to date: https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one#table-of-contents * [README] Use syntax highlighting for installation instructions. * [README] Small tweak to wording. * [README] Use -U in pip install example. * [README] Don't use '$' prefix on shell commands. It makes it harder to copy and paste the commands. * [README] Add explicit "proceed to all platforms" below. * Add missing load() of bazel rules. * [leaderboard] Move leaderboard utility into compiler_gym namespace. This adds a compiler_gym.leaderboard module that contains the LLVM codesize leaderboard helper code. New API docs provide improved explanation of how to use it. Issue facebookresearch#158. * [leaderboard] Rename --logfile to --results_logfile. This is to break the duplicate flag error from //tests/benchmarks:parallelization_load_test. * [leaderboard] Make it clear that users can set observation spaces. Issue facebookresearch#142. * [CONTRIBUTING] Improve leaderboard submission instructions. Re-order the file so that leaderboard submissions appear directly below pull requests. Then provide more details about the submission review process. * [leaderboard] Rename LLVM codesize to instruction count. Be clear that this leaderboard evaluates performance at reducing the instruction count of LLVM-IR, not the binary codesize. * Add leaderboard package as a dependency of //compiler_gym. * [CONTRIBUTING] Use random-agent PR as example for leaderboard. * leaderboard implementation * fails due to env selection * fails at 60th evaluation * Rebase Tabular Q leaderboard on latest development. * Add load() for bazel symbol. Co-authored-by: Bram Wasti <bwasti@fb.com> Co-authored-by: Jiadong Guo <jdguo@fb.com>
ChrisCummins
changed the title
|
LGTM, thanks for this @JD-ETH! Feel free to merge when happy. Cheers, |
bwasti
added a commit
to bwasti/CompilerGym
that referenced
this pull request
Aug 3, 2021
* leaderboard implementation * fails due to env selection * fails at 60th evaluation * Update to WIP Tabular Q leaderboard submission (facebookresearch#1) * [llvm] Temporarily disable polybench Mitigates facebookresearch#55. * [validation] Add a flakiness retry loop around validation. Add a retry loop around the granular individual validation callbacks for cBench-v1. Mitigates facebookresearch#144. * [validation] Catch timeouts in retry loop. * [docs/faq] "I updated with 'git pull' and not it doesn't work * [tests] Extend timeout on datasets test. * [tests] Update regression tests. * [tests] Reduce validation regression test retry counts. * Call env.reset() just after creation - fixes facebookresearch#150 * [docs] Update LLVM actions table. * Add a target to rename the manylinux file. * Force UTF-8 on README decoding. * [util] Improve runfiles docstrings. * [llvm] Remove LLVM binaries from wheel. This patch removes the LLVM binaries from the shipped wheel. This is to reduce the package size to be under the 100MB default maximum imposed by PyPi. Instead of shipping the files in the wheel, the LLVM binaries are downloaded from an archive hosted by Facebook when needed. The circumstances for needing them are: (1) starting an LLVM service, (2) attempting to resolve the path to an LLVM binary. * Defer evaluation of cBench runtime data directory. * [tests] Remove tests that overwrite site data path. These no longer work now that site data requires LLVM binaries to be present. * Release v0.1.5. * Add a fast path check for downloaded LLVM files. * [tests] Use full URI for benchmark. * Correct retry count in error message. * [env] Include last error on init failure. * [env] Add a special error message for UNKNOWN errors. * [rpc] Allow loglines() when logs directory does not exist. * [rpc] Include service logs in error message on init failure. * [rpc] Include final error message on retry loop failure. * [rpc] Add decoded signal name on init error. * [llvm] Replace DCHECK() with Status error. * [tests] Remove tests that interfere with site data path. Site data directory is now a pre-requisite of the LLVM environment and cannot be moved. * [tests] Fix caught exception type. * [llvm] Add a check for runfile requirement. * [llvm] Add a file existing check. * [rpc] Disable logs buffering on debugging runs. * [tests] Fix error message comparison tests * [bin/manual_env] Update prompt after reset(). Running `reset()` with no benchmark set will select a random program, so the prompt must be updated. * [tests] Add workaround for prompt issue. * Release v0.1.6. This release focuses on hardening the LLVM environments, providing improved semantics validation, and improving the datasets. Many thanks to @JD-at-work, @bwasti, and @mostafaelhoushi for code contributions. - [llvm] Added a new `cBench-v1` dataset which changes the function attributes of the IR to permit inlining. `cBench-v0` is deprecated and will be removed no earlier than v0.1.6. - [llvm] Removed 15 passes from the LLVM action space: `-bounds-checking`, `-chr`, `-extract-blocks`, `-gvn-sink`, `-loop-extract-single`, `-loop-extract`, `-objc-arc-apelim`, `-objc-arc-contract`, `-objc-arc-expand`, `-objc-arc`, `-place-safepoints`, `-rewrite-symbols`, `-strip-dead-debug-info`, `-strip-nonlinetable-debuginfo`, `-structurizecfg`. Passes are removed if they are: irrelevant (e.g. used only debugging), if they change the program semantics (e.g. inserting runtimes bound checking), or if they have been found to have nondeterministic behavior between runs. - Extended `env.step()` so that it can take a list of actions that are all performed in a single batch. This improve efficiency. - Added default reward spaces for `CompilerEnv` that are derived from scalar observations (thanks @bwasti!) - Added a new Q learning example (thanks @JD-at-work!). - *Deprecation:* The next release v0.1.5 will introduce a new datasets API that is easier to use and more flexible. In preparation for this, the `Dataset` class has been renamed to `LegacyDataset`, the following dataset operations have been marked deprecated: `activate()`, `deactivate()`, and `delete()`. The `GetBenchmarks()` RPC interface method has also been marked deprecated.. - [llvm] Improved semantics validation using LLVM's memory, thread, address, and undefined behavior sanitizers. - Numerous bug fixes and improvements. * [tests] Add temporary workaround for flaky init benchmark. * Add missing copyright header to make_specs.py. * [util] Force string type in truncate(). * [bin/service]: Fix reporting of observation space shape. * [bin/service]: Fix reporting of observation space shape. * [util] Force string type in truncate(). * [llvm] Add an InstCount observation space. This adds new observation spaces that expose the -instcount pass values. The -instcount pass counts the number of instructions of each type in a program, along with the total number of instructions, total number of blocks, and total number of functions. There are four new observation spaces: `InstCount`, which returns the feature vector as a numpy array, `InstCountDict`, which returns the values as a dictionary of named features, and `InstCountNorm` and `InstCountNormDict`, which are the same as above but the counts are instead normalized to the total number of instructions in the program. Example usage: >>> import gym >>> import compiler_gym >>> env = gym.make("llvm-v0") >>> env.observation_space = "InstCountDict" >>> env.reset("cBench-v0/crc32") {'TotalInstsCount': 196, 'TotalBlocksCount': 29, 'TotalFuncsCount': 13, 'RetCount': 5, 'BrCount': 24, 'SwitchCount': 0, 'IndirectBrCount': 0, 'InvokeCount': 0, 'ResumeCount': 0, 'UnreachableCount': 0, 'CleanupRetCount': 0, 'CatchRetCount': 0, 'CatchSwitchCount': 0, 'CallBrCount': 0, 'FNegCount': 0, 'AddCount': 5, 'FAddCount': 0, 'SubCount': 0, 'FSubCount': 0, 'MulCount': 0, 'FMulCount': 0, 'UDivCount': 0, 'SDivCount': 0, 'FDivCount': 0, 'URemCount': 0, 'SRemCount': 0, 'FRemCount': 0, 'ShlCount': 0, 'LShrCount': 3, 'AShrCount': 0, 'AndCount': 3, 'OrCount': 1, 'XorCount': 8, 'AllocaCount': 24, 'LoadCount': 51, 'StoreCount': 38, 'GetElementPtrCount': 5, 'FenceCount': 0, 'AtomicCmpXchgCount': 0, 'AtomicRMWCount': 0, 'TruncCount': 1, 'ZExtCount': 5, 'SExtCount': 0, 'FPToUICount': 0, 'FPToSICount': 0, 'UIToFPCount': 0, 'SIToFPCount': 0, 'FPTruncCount': 0, 'FPExtCount': 0, 'PtrToIntCount': 0, 'IntToPtrCount': 0, 'BitCastCount': 0, 'AddrSpaceCastCount': 0, 'CleanupPadCount': 0, 'CatchPadCount': 0, 'ICmpCount': 10, 'FCmpCount': 0, 'PHICount': 0, 'CallCount': 13, 'SelectCount': 0, 'UserOp1Count': 0, 'UserOp2Count': 0, 'VAArgCount': 0, 'ExtractElementCount': 0, 'InsertElementCount': 0, 'ShuffleVectorCount': 0, 'ExtractValueCount': 0, 'InsertValueCount': 0, 'LandingPadCount': 0, 'FreezeCount': 0} The InstCount observation spaces are quick to compute and lightweight. They have similar computational complexity as Autophase. Fixes facebookresearch#149. * [ci] Enable test workflows on Python 3.9. Issue facebookresearch#162. * Bump grpcio from 1.34 to 1.36. Issue facebookresearch#162. * Bump bazel requirement to 4.0.0. This is required to build grpcio 1.36.0. Issue facebookresearch#162. * [ci] Reverse order of sudo in setup. Issue facebookresearch#162. * Add libjpeg-dev to list of required linux packages. This to enable compiling Pillow from source on Python 3.9. Issue facebookresearch#162. * Bump the gym dependency to 0.18.0. Issue facebookresearch#162. * [examples] Fix initialization of temporary directory variable. * Add zlib to macOS dependencies. This is to fix compilation of Pillow using Python 3.9. Issue facebookresearch#162. * [readme] Recommend python 3.9 for conda environments. Issue facebookresearch#162. * [ci] Use python 3.9 for continuous integration jobs. Issue facebookresearch#162. * [setup.py] Add a list of supported python versions * [setup.py] Bump development status to Alpha. * [README] Use non-sudo instructions for linux setup. * [README] Simplify table of contents. This adds <!-- omit from toc --> annotations to some of the minor subheadings to keep the table of contents as simple as possible. This uses the "Markdown All in One" plugin for VSCode to automatically keep the table of contents up to date: https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one#table-of-contents * [README] Use syntax highlighting for installation instructions. * [README] Small tweak to wording. * [README] Use -U in pip install example. * [README] Don't use '$' prefix on shell commands. It makes it harder to copy and paste the commands. * [README] Add explicit "proceed to all platforms" below. * Add missing load() of bazel rules. * [leaderboard] Move leaderboard utility into compiler_gym namespace. This adds a compiler_gym.leaderboard module that contains the LLVM codesize leaderboard helper code. New API docs provide improved explanation of how to use it. Issue facebookresearch#158. * [leaderboard] Rename --logfile to --results_logfile. This is to break the duplicate flag error from //tests/benchmarks:parallelization_load_test. * [leaderboard] Make it clear that users can set observation spaces. Issue facebookresearch#142. * [CONTRIBUTING] Improve leaderboard submission instructions. Re-order the file so that leaderboard submissions appear directly below pull requests. Then provide more details about the submission review process. * [leaderboard] Rename LLVM codesize to instruction count. Be clear that this leaderboard evaluates performance at reducing the instruction count of LLVM-IR, not the binary codesize. * Add leaderboard package as a dependency of //compiler_gym. * [CONTRIBUTING] Use random-agent PR as example for leaderboard. * leaderboard implementation * fails due to env selection * fails at 60th evaluation * Rebase Tabular Q leaderboard on latest development. * Add load() for bazel symbol. Co-authored-by: Bram Wasti <bwasti@fb.com> Co-authored-by: Jiadong Guo <jdguo@fb.com> * Add JD's tabular-q leaderboard submission * updated smoke test and readme Co-authored-by: Jiadong Guo <jdguo@fb.com> Co-authored-by: Chris Cummins <chrisc.101@gmail.com> Co-authored-by: Bram Wasti <bwasti@fb.com> Co-authored-by: Chris Cummins <cummins@fb.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
@ChrisCummins I am encountering this error during compilation, and I have been using the most recent development branch and up-to-date pip package of compiler gym.
Decided to make it a PR so that you have context as well.