Advanced queries via Go code

[mvdan]

Simple queries can be done as a pipeline, such as finding all unnecessary uses of [:] on strings:

$ gogrep -x '$x[:]' -a 'type(string)'

However, this quickly breaks down if any non-linear logic is required. For example, what if we wanted to catch all unnecessary uses of [:], including both strings and slices? You might imagine a more complex syntax and language, like:

$ gogrep -x '$x[:]' -a 'type(string) OR is(slice)'

However, this would still be a limited subset of all the logic that you might want in your query. Another common example is variations on -g, such as contains at least 2 of X pattern, as opposed to contains at least 1 of X pattern.

Instead of complicating the query language further, we should leverage the Go language, which all gogrep users should already be familiar with. Our unnecessary [:] example above could be something like:

stmts := gogrep.FindAll(input, `$x[:]`)
stmts = gogrep.Filter(stmts, gogrep.AnyOf(
        gogrep.Type("string"),
        gogrep.Kind("slice"),
))
gogrep.Output(stmts, "should drop [:] on strings and slices")

All the features that are available via the command line would be available via this API, as the command line would be implemented using the API after all. For example, to make the code perform a rewrite instead of giving a warning, we could have:

stmts := gogrep.FindAll(input, `$x[:]`)
stmts = gogrep.Filter(stmts, gogrep.AnyOf(
        gogrep.Type("string"),
        gogrep.Kind("slice"),
))
stmts = gogrep.Substitute(stmts, "$x")
gogrep.WriteBack(stmts)

One of the important features of using Go code is the ability to compose commands in more interesting ways, such as our use of gogrep.AnyOf above. But it would also be possible to perform changes and filters directly, as one can range over the stmts list of type []ast.Stmt.

We could even add ways to simplify the writing of simpler pipe-like queries in Go code, such as:

gogrep.Pipe(input,
    gogrep.FindAll("$x[:]"),
    gogrep.Filter(gogrep.AnyOf(gogrep.Type("string"), gogrep.Kind("slice"))),
    gogrep.Substitute("$x"),
    gogrep.WriteBack,
)

The first step is to move the logic out of the main package, so that it can be imported as a Go package. I would also like to come up with a package name that is shorter than gogrep, as one will have to write it everywhere (assuming dot-imports are to be avoided). Perhaps gg for its initials, or gr for "grep".

/cc @rogpeppe @kytrinyx @quasilyte

[quasilyte]

mvdan.cc/go/ast/grep or mvdan.cc/go/grep. :)

Thanks for working on this.
It can be very helpful in tools that one may want to write for one-off analysis of code.

[mvdan]

I'm starting work on this; the refactor will be large, and will probably take me another day or two before it's usable and pushed to master.

@quasilyte once done, it's probably going to allow you to stop vendoring this module in your project :) The tool's UI will also change, as I've never really liked the grep-like command line UI based on flags.

I'm of course being influenced a bit by your design in go-ruleguard, though my high-level idea has always been about running the user's Go code directly, which I think is different from your design.

[mvdan]

I know I said "another day or two", but holidays and all.

The refactor is pretty much done, but I've spent some extra time tweaking the API before I publish it all. I'm pretty happy with it at this point, so I'll probably publish it over the weekend, with a big fat warning about it being a work in progress.

[quasilyte]

Will there be a way to use user-provided types info and other kinds of data that are needed by gogrep?

It would make it easier to integrate it inside linters that want to use patterns inside some of their checks.

[mvdan]

Probably not with the first design, but we can look into lower-level APIs once that first design is done.

[quasilyte]

Any updates on this?

I'll check out the recent commits and release (I believe there was one).
I could add an example of "advanced query" if the gogrep as a library is ready. And then we can close this issue, perhaps?

[mvdan]

Sorry for the silence. I got stuck on two user-facing problems, and I haven't found good solutions, so I struggle to replace master with the rewrite.

1. How to make the nls package composable enough. I can imagine multiple ways to make the DSL more powerful, but they all feel clunky and overly complex.

2. What UX to show to the command line. Currently, the first argument is for gogrep files, and all remaining arguments are source code to search. But we could also mimic tests, with foo_grep.go files that check a package and any sub-packages.

[mvdan]

I've pushed what I have as of a couple of weeks ago on a separate branch, to avoid breaking master while it's still not ready. See rewrite.