Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Redo benchmarks #1320

Open
RyanGlScott opened this issue Jan 27, 2022 · 1 comment
Open

Redo benchmarks #1320

RyanGlScott opened this issue Jan 27, 2022 · 1 comment
Labels
design needed We need to specify precisely what we want

Comments

@RyanGlScott
Copy link
Contributor

Attempting to build bench:cryptol-bench fails with:

$ cabal build bench:cryptol-bench
Build profile: -w ghc-8.10.7 -O1
In order, the following will be built (use -v for more details):
 - cryptol-2.12.0.99 (bench:cryptol-bench) (first run)
Preprocessing benchmark 'cryptol-bench' for cryptol-2.12.0.99..
Building benchmark 'cryptol-bench' for cryptol-2.12.0.99..
[1 of 1] Compiling Main             ( bench/Main.hs, /home/rscott/Documents/Hacking/Haskell/cryptol/dist-newstyle/build/x86_64-linux/ghc-8.10.7/cryptol-2.12.0.99/b/cryptol-bench/build/cryptol-bench/cryptol-bench-tmp/Main.o )

bench/Main.hs:151:23: error:
    Not in scope: ‘C.evalPrim’
    Module ‘Cryptol.Eval.Concrete’ does not export ‘evalPrim’.
    |
151 |       let ?evalPrim = C.evalPrim
    |                       ^^^^^^^^^^

bench/Main.hs:172:57: error:
    Not in scope: ‘S.sbvEval’
    Module ‘Cryptol.Eval.SBV’ does not export ‘sbvEval’.
    |
172 |     bench name $ whnfIO $ fmap force E.runEval evOpts $ S.sbvEval $ do
    |                                                         ^^^^^^^^^

bench/Main.hs:173:23: error:
    Not in scope: ‘S.evalPrim’
    Module ‘Cryptol.Eval.SBV’ does not export ‘evalPrim’.
    |
173 |       let ?evalPrim = S.evalPrim
    |                       ^^^^^^^^^^

bench/Main.hs:174:27: error:
    Not in scope: data constructor ‘S.SBV’
    Module ‘Cryptol.Eval.SBV’ does not export ‘SBV’.
    |
174 |       env' <- E.evalDecls S.SBV (M.allDeclGroups menv) mempty
    |                           ^^^^^

bench/Main.hs:175:13: error:
    Not in scope: type constructor or class ‘S.Value’
    Perhaps you meant ‘C.Value’ (imported from Cryptol.Eval.Concrete)
    Module ‘Cryptol.Eval.SBV’ does not export ‘Value’.
    |
175 |       (e :: S.Value) <- E.evalExpr S.SBV env' texpr
    |             ^^^^^^^

bench/Main.hs:175:36: error:
    Not in scope: data constructor ‘S.SBV’
    Module ‘Cryptol.Eval.SBV’ does not export ‘SBV’.
    |
175 |       (e :: S.Value) <- E.evalExpr S.SBV env' texpr
    |                                    ^^^^^

It's possible that #928 is the culprit.

@qsctr
Copy link
Contributor

qsctr commented Aug 25, 2022

After a conversation with @robdockins it seems like the existing benchmarks have bit-rotted so much that it's probably a better choice to delete them and start over, and they were not really fully achieving what we wanted them to do anyways. But we need to spend some time thinking about what exactly we want to measure, what is the best way to measure it, and how it would fit into CI (e.g. would we make CI fail on a performance regression?).

@qsctr qsctr changed the title cryptol-bench fails to build Redo benchmarks Aug 25, 2022
@qsctr qsctr added the design needed We need to specify precisely what we want label Aug 25, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
design needed We need to specify precisely what we want
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants