Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(cmd/gno): perform type checking when calling linter #1730

Open
wants to merge 70 commits into
base: master
Choose a base branch
from

Conversation

thehowl
Copy link
Member

@thehowl thehowl commented Mar 4, 2024

Depends on (in order):

  1. feat(stdlibs): remove support for linkedType in native bindings #1700
  2. feat(gno.land): add go type checking to keeper + tx simulation in gnokey #1702

This PR uses the type checker added in #1702 to perform Gno type checking when calling gno lint. Additionally, it adds validation of gno.mod indirectly (the parsed gno mod is used to determine if a package is a draft, and if so skip type checking).

Because gno lint uses the TestStore, the resulting MemPackages may contain redefinitions, for overwriting standard libraries like AssertOriginCall. I changed the type checker to filter out the redefinitions before they reach the Go type checker.

Further improvements, which can be done after this:

@thehowl thehowl self-assigned this Mar 4, 2024
@thehowl thehowl requested a review from jaekwon as a code owner March 4, 2024 18:40
@thehowl thehowl requested review from leohhhn and a team as code owners May 13, 2024 11:33
@github-actions github-actions bot added 📦 🌐 tendermint v2 Issues or PRs tm2 related 📦 ⛰️ gno.land Issues or PRs gno.land package related labels May 13, 2024
@thehowl thehowl requested a review from moul May 13, 2024 17:36
DIGIX666 pushed a commit to kazai777/gno that referenced this pull request May 15, 2024
…key (gnolang#1702)

Split from gnolang#1695 for ease of reviewing. Merge order:

1. gnolang#1700 
2. gnolang#1702 (this one!)
3. gnolang#1695 \
   gnolang#1730

This PR removes `TranspileAndCheckMempkg` in favour of performing the
type checking it was supposed to do using `go/types` with a custom
importer. This importer works together with Gno's `Store`, and can as
such be used to type check Gno packages without ever writing a single
file to disk. It is important to note that by "Go type check" I mean a
variety of compile-time checks the Go compiler performs; in fact, this
is much more powerful than running "gofmt" as we are currently doing.

Additionally, it adds a new flag to gnokey, `-simulate`, to control
transaction simulation before committing a transaction. See [this issue
comment](gnolang#1702 (comment))

Resolves gnolang#1661.

## Reviewing notes

- transpiler.TranspileAndCheckMempkg has been removed from the gnokey
client and gnoclient, in favour of having this step be performed on the
vm keeper. This paves the way for clients to not have to include the
entire GnoVM, which I call a win.
- Stdlib io had a precompiling error due to an unused variable
(`remaining`); I updated it to the latest code on Go's standard
libraries.
- `Store` changes
- `Store` has been changed to have its `getPackage` method work by
detecting import cycles, without risking race conditions (the current
implementation is not thread-safe). This is done by creating a new
store, `importerStore`, which contains the previously imported paths in
the current chain. Cyclic imports are still (correctly) detected in the
tests.
- `GetMemPackage` has been changed to return nil when a package cannot
be found. This matches its behaviour with `GetMemFile`, which already
did this when the file does not exist.
- `GetMemPackage`, if a package is not found in the store, now attempts
retrieving it using Store.GetPackage first. The underlying reason is
that the Gno importer for the type checker needs to access the source of
the standard libraries; however, these are never in any transaction and
are not executed "per se" when the blockchain start. As a consequence,
they may not exist within the Store; as a solution, when using
GetMemPackage, we ensure that a package does not exist by checking if
GetPackage does not retrieve it through getMemPackage and save it.
Copy link
Member

@zivkovicmilos zivkovicmilos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for adding this functionality 🙏

I've left a few comments, mostly nitpicks, I think we should be good to go 🚀

examples/gno.land/r/demo/art/gnoface/gno.mod Show resolved Hide resolved
gnovm/cmd/gno/lint.go Show resolved Hide resolved
gnovm/cmd/gno/lint.go Show resolved Hide resolved
gnovm/cmd/gno/lint.go Outdated Show resolved Hide resolved
gnovm/cmd/gno/lint.go Show resolved Hide resolved
gnovm/cmd/gno/lint.go Outdated Show resolved Hide resolved
gnovm/pkg/gnolang/go2gno.go Outdated Show resolved Hide resolved
gnovm/pkg/gnolang/go2gno.go Outdated Show resolved Hide resolved
@thehowl
Copy link
Member Author

thehowl commented May 30, 2024

From review meeting, we decided to add this functionality on gno test, following comments on #2050 (comment) .

return false, fmt.Errorf("unexpected error type: %T", err)
}
}
return true, nil
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is the situation that errorsFound is true while err is nil?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

None, really: further above, we check that tcErr == nil. So, at this point, there will always be at least an error that was printed.

Comment on lines +219 to +223
// XXX: package ending with `_test` is not supported yet
if strings.HasSuffix(mfile.Name, "_test.gno") && !strings.HasSuffix(string(n.PkgName), "_test") {
// Keep only test files
testfiles.AddFiles(n)
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This code block should be moved before gno.ParseFile(...) line 214.
I saw somewhere else in this PR that there is also some code to match against *_filetest.gno. Should it be added here, or better, get rid of special treatment for *_filetest.gno. Why should it be different from *_test.gno?

endsWith(file.Name, []string{"_test.gno", "_filetest.gno"}) {
continue // skip spurious file.
strings.HasSuffix(file.Name, "_test.gno") ||
strings.HasSuffix(file.Name, "_filetest.gno") {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we have some special treatment for *_filetest.gno? Could these files just not be *_test.gno?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🥲

Unfortunately, we have filetests and we have them in examples code; and as an official, documented feature of gno test. (see gno test -h). We have previously discussed this at the Rouen retreat. These are of the same format of the tests we have in gnovm/tests/files, similar to the ones you originally created for Yaegi. And I agree; they should probably not exist for anyone who isn't a language developer.

Alas, it's out of scope of this PR to tackle this; so we have to ignore those as well.

@zivkovicmilos
Copy link
Member

@thehowl

All good on my side 🫡

Please check the leftover comms and conflicts, and we're good to merge 🚀

thehowl added a commit that referenced this pull request Jun 19, 2024
Merge order:

1. #1700 
2. #1702
3. #1695 (this one!) -- review earlier ones first, if they're still
open!

This PR modifies the Gno transpiler (fka precompiler) to use Gno's
standard libraries rather than Go's when performing transpilation. This
creates the necessity to transpile Gno standard libraries, and as such
support their native bindings. And it removes the necessity for a
package like `stdshim`, and a mechanism like `stdlibWhitelist`.

- Fixes #668. Fixes #1865.
- Resolves #892.
- Part of #814. 
- Makes #1475 / #1576 possible without using hacks like `stdshim`.

cc/ @leohhhn @tbruyelle, as this relates to your work

## Why?

- This PR enables us to perform Go type-checking across the board, and
not use Go's standard libraries in transpiled code. This enables us to
_properly support our own standard libraries_, such as `std` but any
others we might want or need.
- It also paves the way further to go full circle, and have Gno code be
transpiled to Go, and then have "compilable" gno code

## Summary of changes

- The transpiler has been thoroughly refactored.
- The biggest change is described above: instead of maintaing the import
paths like `"strconv"` and `"math"` the same (so using Gno's stdlibs in
Gno, and Go's in Go), the import paths for standard libraries is now
also updated to point to the Gno standard libraries.
- Native functions are handled by removing their definitions when
transpiling, and changing their call expressions where appropriate. This
links the transpiled code directly to their native counterparts.
  - This removes the necessity for `stdlibWhitelist`. 
- As a consequence, `stdshim` is no longer needed and has been removed.
- Test files are still not "strictly checked": they may reference
stdlibs with no matching source, and will not be tested when running
with `--gobuild`. This is because packages like `fmt` have no
representation in Gno code; they only exist as injections in
`tests/imports.go`. I'll fix this eventually :)
- The CLI (`gno transpile`) has been changed to reflect the above
changes.
- Flag `--skip-fmt` has been removed (the result of transpile is always
formatted, anyway), and `--gofmt-binary` too, obviously. `gno transpile`
does not perform validation, but will gladly provide helpful validation
with the `--gobuild` flag.
- There is another PR that adds type checking in `gno lint`, without
needing to run through the transpilation step first:
#1730
- It now works by default by looking at "packages" rather than
individual files. This is necessary so that when performing `transpile`
on the `examples` directory, we can skip those where the gno.mod marks
the module as draft. These modules make use of packages like "fmt",
which because they don't have an underlying gno/go source, cannot be
transpiled.
- Running with `-gobuild` now handles more errors correctly; ie., all
errors not previously captured by the `errorRe` which only matches those
pertaining to a specific file/line.
  - `gnoFilesFromArgs` was unused and as such deleted
- `gnomod`'s behaviour was slightly changed.
- I am of the opinion that `gno mod download` should not precompile what
it downloads; _especially_ to gather the dependencies it has. I've
changed it so that it does a `OnlyImports` parse of the file it
downloads to fetch additional dependencies

Misc:

- `Makefile` now contains a recipe to calculate the coverage for
`gnovm/cmd/gno`, and also view it via the HTML interface. This is needed
as it has a few extra steps (which @gfanton already previously added in
the CI).
- Realms r/demo/art/gnoface and r/x/manfred_outfmt have been marked as
draft, as they depend on packages which are not actually present in the
Gno standard libraries.
  - The transpiler now ignores draft packages by default.
- `ReadMemPackage` now also considers Go files. This is meant to have
on-chain the code for standard libraries like `std` which have native
bindings. We still exclude Go code if it's not in a standard library.
- `//go:build` constraints have been removed from standard libraries, as
go files can only have one and we already add our own when transpiling

## Further improvements

after this PR

- Scope understanding in `transpiler` (so call expressions are not
incorrectly rewritten)
- Correctly transpile gno.mod

---------

Co-authored-by: Antonio Navarro Perez <antnavper@gmail.com>
Co-authored-by: Miloš Živković <milos.zivkovic@tendermint.com>
gfanton pushed a commit to gfanton/gno that referenced this pull request Jul 23, 2024
Merge order:

1. gnolang#1700 
2. gnolang#1702
3. gnolang#1695 (this one!) -- review earlier ones first, if they're still
open!

This PR modifies the Gno transpiler (fka precompiler) to use Gno's
standard libraries rather than Go's when performing transpilation. This
creates the necessity to transpile Gno standard libraries, and as such
support their native bindings. And it removes the necessity for a
package like `stdshim`, and a mechanism like `stdlibWhitelist`.

- Fixes gnolang#668. Fixes gnolang#1865.
- Resolves gnolang#892.
- Part of gnolang#814. 
- Makes gnolang#1475 / gnolang#1576 possible without using hacks like `stdshim`.

cc/ @leohhhn @tbruyelle, as this relates to your work

## Why?

- This PR enables us to perform Go type-checking across the board, and
not use Go's standard libraries in transpiled code. This enables us to
_properly support our own standard libraries_, such as `std` but any
others we might want or need.
- It also paves the way further to go full circle, and have Gno code be
transpiled to Go, and then have "compilable" gno code

## Summary of changes

- The transpiler has been thoroughly refactored.
- The biggest change is described above: instead of maintaing the import
paths like `"strconv"` and `"math"` the same (so using Gno's stdlibs in
Gno, and Go's in Go), the import paths for standard libraries is now
also updated to point to the Gno standard libraries.
- Native functions are handled by removing their definitions when
transpiling, and changing their call expressions where appropriate. This
links the transpiled code directly to their native counterparts.
  - This removes the necessity for `stdlibWhitelist`. 
- As a consequence, `stdshim` is no longer needed and has been removed.
- Test files are still not "strictly checked": they may reference
stdlibs with no matching source, and will not be tested when running
with `--gobuild`. This is because packages like `fmt` have no
representation in Gno code; they only exist as injections in
`tests/imports.go`. I'll fix this eventually :)
- The CLI (`gno transpile`) has been changed to reflect the above
changes.
- Flag `--skip-fmt` has been removed (the result of transpile is always
formatted, anyway), and `--gofmt-binary` too, obviously. `gno transpile`
does not perform validation, but will gladly provide helpful validation
with the `--gobuild` flag.
- There is another PR that adds type checking in `gno lint`, without
needing to run through the transpilation step first:
gnolang#1730
- It now works by default by looking at "packages" rather than
individual files. This is necessary so that when performing `transpile`
on the `examples` directory, we can skip those where the gno.mod marks
the module as draft. These modules make use of packages like "fmt",
which because they don't have an underlying gno/go source, cannot be
transpiled.
- Running with `-gobuild` now handles more errors correctly; ie., all
errors not previously captured by the `errorRe` which only matches those
pertaining to a specific file/line.
  - `gnoFilesFromArgs` was unused and as such deleted
- `gnomod`'s behaviour was slightly changed.
- I am of the opinion that `gno mod download` should not precompile what
it downloads; _especially_ to gather the dependencies it has. I've
changed it so that it does a `OnlyImports` parse of the file it
downloads to fetch additional dependencies

Misc:

- `Makefile` now contains a recipe to calculate the coverage for
`gnovm/cmd/gno`, and also view it via the HTML interface. This is needed
as it has a few extra steps (which @gfanton already previously added in
the CI).
- Realms r/demo/art/gnoface and r/x/manfred_outfmt have been marked as
draft, as they depend on packages which are not actually present in the
Gno standard libraries.
  - The transpiler now ignores draft packages by default.
- `ReadMemPackage` now also considers Go files. This is meant to have
on-chain the code for standard libraries like `std` which have native
bindings. We still exclude Go code if it's not in a standard library.
- `//go:build` constraints have been removed from standard libraries, as
go files can only have one and we already add our own when transpiling

## Further improvements

after this PR

- Scope understanding in `transpiler` (so call expressions are not
incorrectly rewritten)
- Correctly transpile gno.mod

---------

Co-authored-by: Antonio Navarro Perez <antnavper@gmail.com>
Co-authored-by: Miloš Živković <milos.zivkovic@tendermint.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
📦 🌐 tendermint v2 Issues or PRs tm2 related 📦 ⛰️ gno.land Issues or PRs gno.land package related 📦 🤖 gnovm Issues or PRs gnovm related 🧾 package/realm Tag used for new Realms or Packages.
Projects
Status: In Progress
Status: No status
Status: In Review
Development

Successfully merging this pull request may close these issues.

7 participants