Skip to content

Commit

Permalink
Change fmt/lint/check docs with latest changes (#17133)
Browse files Browse the repository at this point in the history 
Changing the docs to keep the "lowest-bar-to-entry" example but compatible with the new changes. I don't _love_ it, but I'm also not feeling very creative.

Fixes #16865
  • Loading branch information
@thejcannon
thejcannon authored Oct 19, 2022
1 parent b1f0598 commit be00ed5
Show file tree
Hide file tree
Showing 4 changed files with 141 additions and 99 deletions.
64 changes: 63 additions & 1 deletion docs/markdown/Writing Plugins/common-plugin-tasks/plugin-upgrade-guide.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,68 @@ updatedAt: "2022-07-25T20:02:17.695Z"
2.15
----

### `lint` and `fmt` schema changes

In order to accomplish several goals (namely targetless formatters and unifying the implementation of `lint`)
`lint` and `fmt` have undergone a drastic change of their plugin API.

#### 1. `Lint<Targets|Files>Request` and `FmtTargetsRequest` now require a `tool_subsystem` class attribute.

Instead of the `name` class attribute, `Lint<Targets|Files>Request` and `FmtTargetsRequest` require
subclasses to provide a `tool_subsystem` class attribute with a value of your tool's `Subsystem` subclass.

#### 2. Your tool subsystem should have a `skip` option.

Although not explictly not required by the engine to function correctly, `mypy` will complain if the
subsystem type provided to `tool_subsystem` doesn't have a `skip: SkipOption` option registered.

Otherwise, you can `# type: ignore[assignment]` on your `tool_subsystem` declaration.

#### 3. The core goals now use a 2-rule approach

Fmt:

In order to support targetless formatters, `fmt` needs to know which _files_ you'll be operating on.
Therefore the plugin API for `fmt` has forked into 2 rules:

1. A rule taking `<RequestType>.PartitionRequest` and returning a `Partitions` object. This is sometimes referred to as the "partitioner" rule.
2. A rule taking `<RequestType>.SubPartition` and returning a `FmtResult`. This is sometimes referred to as the "runner" rule.

This way `fmt` can serialize tool runs that operate on the same file(s) while parallelizing tool runs
that don't overlap.

(Why are targetless formatters something we want to support? This allows us to have `BUILD` file formatters,
formatters like `Prettier` running on your codebase *without* boilerplate targets, as well as Pants
doing interesting deprecation fixers on its own files)

The partitioner rule gives you all the matching files (or `FieldSet`s depending on which class you
subclassed) and you'll return a mapping from `<key>` to files (called a Partition).
The `<key>` can be anything passable at the rule boundary and is given back to you in your runner rule.
The partitioner rule gives you an opportunity to perform expensive `Get`s once for the entire run,
to partition the inputs based on metadata to simplify your runner, and to have a place for easily
skipping your tool if requested.

The runner rule will mostly remain unchanged, aside from the request type (`<RequestType>.SubPartition`),
which now has a `.files` property.

If you don't require any `Get`s or metadata for your tool in your partitioner rule,
Pants has a way to provide a "default" implementation. In your `FmtRequest` subclass,
set the `partitioner_type` class variable to `PartitionerType.DEFAULT_SINGLE_PARTITION` and only
provide a runner rule.

-----

Lint:

Lint plugins are almost identical to format plugins, except in 2 ways:

1. Your partitioner rule still returns a `Partitions` object, but the element type can be anything.
2. `<RequestType>.SubPartition` has a `.elements` field instead of `.files`.

-----

As always, taking a look at Pants' own plugins can also be very enlightening.

### `EnvironmentName` is now required to run processes, get environment variables, etc

Pants 2.15 introduces the concept of ["Target Environments"](doc:environments), which allow Pants to execute processes in remote or local containerized environments (using Docker), and to specify configuration values for those environments.
Expand Down Expand Up @@ -175,7 +237,7 @@ on `FieldSet`'s mechanisms for matching targets and getting field values.
Rather than directly subclassing `GenerateToolLockfileSentinel`, we encourage you to subclass
`GeneratePythonToolLockfileSentinel` and `GenerateJvmToolLockfileSentinel`. This is so that we can
distinguish what language a tool belongs to, which is used for options like
`[python].resolves_to_constraints_file` to validate which resolve names are recognized.
`[python].resolves_to_constraints_file` to validate which resolve names are recognized.

Things will still work if you do not make this change, other than the new options not recognizing
your tool.
Expand Down
71 changes: 32 additions & 39 deletions docs/markdown/Writing Plugins/common-plugin-tasks/plugins-fmt-goal.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,18 +6,24 @@ hidden: false
createdAt: "2020-07-01T04:52:28.820Z"
updatedAt: "2022-04-27T18:37:11.334Z"
---
In Pants, every formatter is (typically) also a linter, meaning that if you can run a tool with `./pants fmt`, you can run the same tool in check-only mode with `./pants lint`. Start by skimming [Add a linter](doc:plugins-lint-goal) to familiarize yourself with how linters work.
In Pants, every formatter is also a linter, meaning that if you can run a tool with `./pants fmt`,
you can run the same tool in check-only mode with `./pants lint`.
Start by skimming [Add a linter](doc:plugins-lint-goal) to familiarize yourself with how linters work.

This guide assumes that you are running a formatter that already exists outside of Pants as a stand-alone binary, such as running Black or Prettier.

If you are instead writing your own formatting logic inline, you can skip Step 1. In Step 4, you will not need to use `Process`.

1. Install your formatter
-------------------------
# 1. Install your formatter

There are several ways for Pants to install your formatter. See [Installing tools](doc:rules-api-installing-tools). This example will use `ExternalTool` because there is already a pre-compiled binary for shfmt.
There are several ways for Pants to install your formatter. See [Installing tools](doc:rules-api-installing-tools).
This example will use `ExternalTool` because there is already a pre-compiled binary for shfmt.

You will also likely want to register some options, like `--config`, `--skip`, and `--args`. Options are registered through a [`Subsystem`](doc:rules-api-subsystems). If you are using `ExternalTool`, this is already a subclass of `Subsystem`. Otherwise, create a subclass of `Subsystem`. Then, set the class property `options_scope` to the name of the tool, e.g. `"shfmt"` or `"prettier"`. Finally, add options from `pants.option.option_types`.
You will also likely want to register some options, like `--config`, `--skip`, and `--args`.
Options are registered through a [`Subsystem`](doc:rules-api-subsystems).
If you are using `ExternalTool`, this is already a subclass of `Subsystem`.
Otherwise, create a subclass of `Subsystem`. Then, set the class property `options_scope` to the
name of the tool, e.g. `"shfmt"` or `"prettier"`. Finally, add options from `pants.option.option_types`.

```python
from pants.core.util_rules.external_tool import ExternalTool
Expand Down Expand Up @@ -61,8 +67,7 @@ class Shfmt(ExternalTool):
return f"./shfmt_{self.version}_{plat_str}"
```

2. Set up a `FieldSet` and `FmtRequest`
---------------------------------------
# 2. Set up a `FieldSet` and `FmtTargetsRequest`

As described in [Rules and the Target API](doc:rules-api-and-target-api), a `FieldSet` is a way to tell Pants which `Field`s you care about targets having for your plugin to work.

Expand All @@ -89,36 +94,18 @@ Then, hook this up to a new subclass of `FmtRequest`.
```python
from pants.core.goals.fmt import FmtRequest

class ShfmtRequest(FmtRequest):
class ShfmtRequest(FmtTargetsRequest):
field_set_type = ShfmtFieldSet
name = "shfmt"
tool_subsystem = Shfmt
```

Finally, register your new `FmtRequest` with a [`UnionRule`](doc:rules-api-unions) so that Pants knows your formatter exists:
# 3. Create `fmt` rules

```python
from pants.engine.unions import UnionRule

...

def rules():
return [
*collect_rules(),
UnionRule(FmtRequest, ShfmtRequest),
]
```

3. Create `fmt` rules
---------------------

You will need a rule for `fmt` which takes the `FmtRequest` from step 3 (e.g. `ShfmtRequest`) as a parameter and returns a `FmtResult`.
You will need a rule for `fmt` which takes the `FmtTargetsRequest.SubPartition` from step 3 (e.g. `ShfmtRequest`) as a parameter and returns a `FmtResult`.

```python
@rule(desc="Format with shfmt", level=LogLevel.DEBUG)
async def shfmt_fmt(request: ShfmtRequest, shfmt: Shfmt, platform: Platform) -> FmtResult:
if shfmt.skip:
return FmtResult.skip(formatter_name=request.name)

async def shfmt_fmt(request: ShfmtRequest.SubPartition, shfmt: Shfmt, platform: Platform) -> FmtResult:
download_shfmt_get = Get(
DownloadedExternalTool,
ExternalToolRequest,
Expand Down Expand Up @@ -157,23 +144,30 @@ async def shfmt_fmt(request: ShfmtRequest, shfmt: Shfmt, platform: Platform) ->
argv=argv,
input_digest=input_digest,
output_files=request.snapshot.files,
description=f"Run shfmt on {pluralize(len(request.field_sets), 'file')}.",
description=f"Run shfmt on {pluralize(len(request.snapshot.files), 'file')}.",
level=LogLevel.DEBUG,
)

result = await Get(ProcessResult, Process, process)
output_snapshot = await Get(Snapshot, result.output_digest)
return FmtResult.create(request, result, output_snapshot)
return await FmtResult.create(request, result, output_snapshot)
```

The `FmtRequest` has properties `.field_sets` and `.snapshot`, which store collections of the `FieldSet`s defined in step 2, and their sources. Each `FieldSet` corresponds to a single target. Pants will have already validated that there is at least one valid `FieldSet`, so you can expect `ShfmtRequest.field_sets` to have 1-n `FieldSet` instances.

If you have a `--skip` option, you should check if it was used at the beginning of your `fmt` and `lint` rules and, if so, to early return an empty `LintResults()` and return `FmtResult.skip()`.
The `FmtRequest.SubPartition` has `.snapshot`, which stores the list of files and the `Digest` for each source file.

If you used `ExternalTool` in step 1, you will use `Get(DownloadedExternalTool, ExternalToolRequest)` to ensure that the tool is fetched.

Use `Get(Digest, MergeDigests)` to combine the different inputs together, such as merging the source files and downloaded tool.

At the bottom of your file, tell Pants about your rules:

```python
def rules():
return [
*collect_rules(),
*ShfmtRequest.rules(partitioner_type=PartitionerType.DEFAULT_SINGLE_PARTITION),
]
```

Finally, update your plugin's `register.py` to activate this file's rules. Note that we must register the rules added in Step 2, as well.

```python pants-plugins/shell/register.py
Expand All @@ -184,9 +178,8 @@ def rules():
return [*shfmt.rules()]
```

Now, when you run `./pants fmt ::` or `./pants lint ::`, your new formatter should run.
Now, when you run `./pants fmt ::` or `./pants lint ::`, your new formatter should run.

5. Add tests (optional)
-----------------------
# 4. Add tests (optional)

Refer to [Testing rules](doc:rules-api-testing).
100 changes: 43 additions & 57 deletions docs/markdown/Writing Plugins/common-plugin-tasks/plugins-lint-goal.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,16 +6,20 @@ hidden: false
createdAt: "2020-07-01T04:51:55.583Z"
updatedAt: "2022-04-07T14:48:36.791Z"
---
This guide assumes that you are running a linter that already exists outside of Pants as a stand-alone binary, such as running Shellcheck, Pylint, Checkstyle, or ESLint.
This guide assumes that you are running a linter that already exists outside of Pants as a stand-alone binary, such as running Shellcheck, Pylint, Checkstyle, or ESLint.

If you are instead writing your own linting logic inline, you can skip Step 1. In Step 3, you will not need to use `Process`. You may find Pants's [`regex-lint` implementation](https://github.com/pantsbuild/pants/blob/main/src/python/pants/backend/project_info/regex_lint.py) helpful for how to integrate custom linting logic into Pants.

1. Install your linter
----------------------
# 1. Install your linter

There are several ways for Pants to install your linter. See [Installing tools](doc:rules-api-installing-tools). This example will use `ExternalTool` because there is already a pre-compiled binary for Shellcheck.
There are several ways for Pants to install your linter. See [Installing tools](doc:rules-api-installing-tools).
This example will use `ExternalTool` because there is already a pre-compiled binary for Shellcheck.

You will also likely want to register some options, like `--config`, `--skip`, and `--args`. Options are registered through a [`Subsystem`](doc:rules-api-subsystems). If you are using `ExternalTool`, this is already a subclass of `Subsystem`. Otherwise, create a subclass of `Subsystem`. Then, set the class property `options_scope` to the name of the tool, e.g. `"shellcheck"` or `"eslint"`. Finally, add options from `pants.option.option_types`.
You will also likely want to register some options, like `--config`, `--skip`, and `--args`.
Options are registered through a [`Subsystem`](doc:rules-api-subsystems).
If you are using `ExternalTool`, this is already a subclass of `Subsystem`.
Otherwise, create a subclass of `Subsystem`. Then, set the class property `options_scope` to the name
of the tool, e.g. `"shellcheck"` or `"eslint"`. Finally, add options from `pants.option.option_types`.

```python
from pants.core.util_rules.external_tool import ExternalTool
Expand All @@ -27,6 +31,7 @@ class Shellcheck(ExternalTool):
"""A linter for shell scripts."""

options_scope = "shellcheck"
name = "ShellCheck"
default_version = "v0.8.0"
default_known_versions = [
"v0.8.0|macos_arm64 |e065d4afb2620cc8c1d420a9b3e6243c84ff1a693c1ff0e38f279c8f31e86634|4049756",
Expand Down Expand Up @@ -55,23 +60,14 @@ class Shellcheck(ExternalTool):

```

Lastly, register your Subsystem with the engine:
# Set up a `FieldSet` and `LintTargetsRequest`

```python
from pants.engine.rules import collect_rules

...

def rules():
return collect_rules()
```

2. Set up a `FieldSet` and `LintRequest`
----------------------------------------

As described in [Rules and the Target API](doc:rules-api-and-target-api), a `FieldSet` is a way to tell Pants which `Field`s you care about targets having for your plugin to work.
As described in [Rules and the Target API](doc:rules-api-and-target-api), a `FieldSet` is a way to
tell Pants which `Field`s you care about targets having for your plugin to work.

Usually, you should add a subclass of the `Sources` field to the class property `required_fields`, such as `BashSources` or `PythonSources`. This means that your linter will run on any target with that sources field or a subclass of it.
Usually, you should add a subclass of the `Sources` field to the class property `required_fields`,
such as `BashSources` or `PythonSources`.
This means that your linter will run on any target with that sources field or a subclass of it.

Create a new dataclass that subclasses `FieldSet`:

Expand Down Expand Up @@ -99,46 +95,31 @@ from pants.core.goals.lint import LintTargetsRequest

class ShellcheckRequest(LintTargetsRequest):
field_set_type = ShellcheckFieldSet
name = "shellcheck"
tool_subsystem = Shellcheck
```

Finally, register your new `LintTargetsRequest ` with a [`UnionRule`](doc:rules-api-unions) so that Pants knows your linter exists:
# 3. Create a rule for your linter logic

```python
from pants.engine.unions import UnionRule

...

def rules():
return [
*collect_rules(),
UnionRule(LintTargetsRequest, ShellcheckRequest),
]
```

3. Create a rule for your linter logic
--------------------------------------

Your rule should take as a parameter the `LintRequest` from step 2 and the `Subsystem` (or `ExternalTool`) from step 1. It should return `LintResults`:
Your rule should take as a parameter `ShellcheckRequest.SubPartition` and the
`Subsystem` (or `ExternalTool`) from step 1 (a `SubPartition` is an object containing a subset of all
the matched field sets for your tool). It should return a `LintResult`:

```python
from pants.engine.rules import rule
from pants.core.goals.lint import LintTargetsRequest, LintResult, LintResults
from pants.core.goals.lint import LintResult

...

@rule
async def run_shellcheck(
request: ShellcheckRequest, shellcheck: Shellcheck
) -> LintResults:
return LintResults(linter_name=request.name)
) -> LintResult:
return LintResult.create(...)
```

The `LintTargetsRequest ` has a property called `.field_sets`, which stores a collection of the `FieldSet`s defined in step 2. Each `FieldSet` corresponds to a single target. Pants will have already validated that there is at least one valid `FieldSet`, so you can expect `LintRequest.field_sets` to have 1-n `FieldSet` instances.

The rule should return `LintResults`, which is a collection of multiple `LintResult` objects. Normally, you will only have one single `LintResult`. Sometimes, however, you may want to group your targets in a certain way and return a `LintResult` for each group, such as grouping Python targets by their interpreter compatibility.

If you have a `--skip` option, you should check if it was used at the beginning of your rule and, if so, to early return an empty `LintResults()`.
The `ShellcheckRequest.SubPartition` instance has a property called `.elements`, which in this case,
stores a collection of the `FieldSet`s defined in step 2. Each `FieldSet` corresponds to a single target.
Pants will have already validated that there is at least one valid `FieldSet`.

If you used `ExternalTool` in step 1, you will use `Get(DownloadedExternalTool, ExternalToolRequest)` to install the tool.

Expand Down Expand Up @@ -177,11 +158,8 @@ from pants.util.strutil import pluralize

@rule
async def run_shellcheck(
request: ShellcheckRequest, shellcheck: Shellcheck, platform: Platform
) -> LintResults:
if shellcheck.skip:
return LintResults([], linter_name=request.name)

request: ShellcheckRequest.SubPartition, shellcheck: Shellcheck, platform: Platform
) -> LintResult:
download_shellcheck_request = Get(
DownloadedExternalTool,
ExternalToolRequest,
Expand All @@ -190,7 +168,7 @@ async def run_shellcheck(

sources_request = Get(
SourceFiles,
SourceFilesRequest(field_set.sources for field_set in request.field_sets),
SourceFilesRequest(field_set.sources for field_set in request.elements),
)

# If the user specified `--shellcheck-config`, we must search for the file they specified with
Expand Down Expand Up @@ -229,15 +207,24 @@ async def run_shellcheck(
*sources.snapshot.files,
],
input_digest=input_digest,
description=f"Run Shellcheck on {pluralize(len(request.field_sets), 'file')}.",
description=f"Run Shellcheck on {pluralize(len(request.elements), 'file')}.",
level=LogLevel.DEBUG,
),
)
result = LintResult.from_fallible_process_result(process_result)
return LintResults([result], linter_name=request.name)
return LintResult.create(request, process_result)

```

At the bottom of your file, tell Pants about your rules:

```python
def rules():
return [
*collect_rules(),
ShellcheckRequest.rules(partitioner_type=PartitionerType.DEFAULT_SINGLE_PARTITION),
]
```

Finally, update your plugin's `register.py` to activate this file's rules.

```python pants-plugins/bash/register.py
Expand All @@ -250,7 +237,6 @@ def rules():

Now, when you run `./pants lint ::`, your new linter should run.

4. Add tests (optional)
-----------------------
# 4. Add tests (optional)

Refer to [Testing rules](doc:rules-api-testing).
Loading

0 comments on commit be00ed5

Please sign in to comment.