Add Error format support, and JSON output option #11396

tusharsadhwani · 2021-10-27T19:21:26Z

Description

Resolves #10816

The changes this PR makes are relatively small.
It currently:

Adds an --output option to mypy CLI
Adds a ErrorFormatter abstract base class, which can be subclassed to create new output formats
Adds a MypyError class that represents the external format of a mypy error.
Adds a check for --output being 'json', in which case the JSONFormatter is used to produce the reported output.

Demo:

$ mypy mytest.py              
mytest.py:2: error: Incompatible types in assignment (expression has type "str", variable has type "int")
mytest.py:3: error: Name "z" is not defined
Found 2 errors in 1 file (checked 1 source file)

$ mypy mytest.py --output=json
{"file": "mytest.py", "line": 2, "column": 4, "severity": "error", "message": "Incompatible types in assignment (expression has type \"str\", variable has type \"int\")", "code": "assignment"}
{"file": "mytest.py", "line": 3, "column": 4, "severity": "error", "message": "Name \"z\" is not defined", "code": "name-defined"}

A few notes regarding the changes:

I chose to re-use the intermediate ErrorTuples created during error reporting, instead of using the more general ErrorInfo class, because a lot of machinery already exists in mypy for sorting and removing duplicate error reports, which produces ErrorTuples at the end. The error sorting and duplicate removal logic could perhaps be separated out from the rest of the code, to be able to use ErrorInfo objects more freely.
ErrorFormatter doesn't really need to be an abstract class, but I think it would be better this way. If there's a different method that would be preferred, I'd be happy to know.
The --output CLI option is, most probably, not added in the correct place. Any help in how to do it properly would be appreciated, the mypy option parsing code seems very complex.
The ability to add custom output formats can be simply added by subclassing the ErrorFormatter class inside a mypy plugin, and adding a name field to the formatters. The mypy runtime can then check through the __subclasses__ of the formatter and determine if such a formatter is present.
The "checking for the name field" part of this code might be appropriate to add within this PR itself, instead of hard-coding JSONFormatter. Does that sound like a good idea?

…output-json

intgr · 2021-10-28T14:06:37Z

mypy/error_formatter.py

+        return json.dumps({
+            'file': file,
+            'line': line,
+            'column': column,


line, column seems short-sighed.

Maybe startLine, startColumn, which would leave room to later add endLine, endColumn. This information is useful for IDEs to know how what span to highlight.

Yeah, good point.
But mypy currently only does line and column, and it might be very long before spans are added in. It could be argued that the change to startLine and startColumn can be made when the feature exists.

Changing the field names later will break tools though. And startLine, startColumn would already be accurate right now, because mypy currently points out the start of the span.

I'll leave this convention for the separate --output=sarif format.

intgr · 2021-10-28T15:31:57Z

I'd like to make the case again for the existing JSON-based standard: Static Analysis Results Interchange Format (SARIF)

While that does not preclude support for a custom JSON format as well, perhaps if mypy were to support SARIF, there would be no need for a custom format?

Pros of SARIF:

There is no need to reinvent the wheel.
Can already integrate with existing tools like GitHub code scanning, Visual Studio, VS Code
Every IDE shouldn't have to implement its own mypy-specific integration. This is exemplified by the situation in IntelliJ IDEA/PyCharm: there are two 3rd party mypy-specific plugins, both of which have major problems that aren't getting fixed. [1]

Clearly the momentum isn't there to provide good mypy-specific IDE integrations. Adding mypy support for SARIF, and letting IDEs take care of parsing SARIF and implementing a good UI on top, seems like a far more sustainable option.

Cons:

SARIF is clearly infected with "design by committee", the data structures are verbose and deeply nested. However, advanced features are optional, a minimal implementation is straightforward, see example below.
No streaming output; all results would have to be collected before the JSON output can be serialized.

[1] The plugin ratings are 3.4 and 2.9 out of 5 😖 https://plugins.jetbrains.com/search?search=mypy

Example minimal mypy SARIF output

{
  "version": "2.1.0",
  "$schema": "https://schemastore.azurewebsites.net/schemas/json/sarif-2.1.0.json",
  "runs": [
    {
      "tool": {
        "driver": {
          "name": "mypy",
          "version": "0.910"
        }
      },
      "results": [
        {
          "ruleId": "assignment",
          "level": "error",
          "message": {
            "text": "Incompatible types in assignment (expression has type \"str\", variable has type \"int\")"
          },
          "locations": [
            {
              "physicalLocation": {
                "artifactLocation": {
                  "uri": "mytest.py"
                },
                "region": {
                  "startLine": 2,
                  "startColumn": 4
                }
              }
            }
          ]
        },
        {
          "ruleId": "name-defined",
          "level": "error",
          "message": {
            "text": "Name \"z\" is not defined"
          },
          "locations": [
            {
              "physicalLocation": {
                "artifactLocation": {
                  "uri": "mytest.py"
                },
                "region": {
                  "startLine": 3,
                  "startColumn": 4
                }
              }
            }
          ]
        }
      ]
    }
  ]
}

nvuillam · 2021-12-08T17:42:56Z

+1 for SARIF format, let's use a common format instead of a mypy custom one ! :)

tusharsadhwani · 2021-12-08T18:46:07Z

I'm willing to turn it into SARIF (basing it on the json snippet provided by @intgr), if that's what I need to get this into mypy 😄

@ethanhs @TH3CHARLie what do you think?

intgr · 2021-12-08T18:57:48Z

After sitting on it a little, I feel that there's room for more than one machine-readable format. The simpler json-line-based format is probably better for simple tools that only care about mypy. I'm willing to implement the SARIF part myself.

But I shouldn't be the one to decide that, I'm just an occasional lurker here. No mypy developers have chipped in yet.

nvuillam · 2021-12-08T19:07:58Z

I'm just a tourist here, but i'm currently activating SARIF output for all linters of MegaLinter, and having native SARIF output is a great benefit for linters ^^ ( + the Github CodeQL that natively understands SARIF format ^^ )

Some other python linters already have SARIF output, like bandit , maybe there is some code to reuse to manage the format ?

tusharsadhwani · 2021-12-08T19:09:57Z

@nvuillam this PR introduces an ErrorFormatter class. By the time this PR is finalized, even if it doesn't use SARIF you will be able to define your own ErrorFormatter class in a plugin probably, and tell it to output the SARIF format, it'll be really easy.

Pylint has the same setup.

nvuillam · 2021-12-08T19:13:05Z

@tusharsadhwani thank but... I don't have the bandwidth to implement SARIF output on all linters that do not manage it yet 😅

intgr · 2021-12-08T19:47:19Z

I think one property that machine-readable formats should have is: if one error causes multiple lines of output, then that should appear as one result item rather than multiple.

So for example with mypy --show-error-context

_local/multi_error.py: note: In function "foo":
_local/multi_error.py:5:9: error: No overload variant of "get" of "Mapping" matches argument type "str"  [call-overload]
_local/multi_error.py:5:9: note: Possible overload variants:
_local/multi_error.py:5:9: note:     def get(self, key: Any) -> Any
_local/multi_error.py:5:9: note:     def [_T] get(self, key: Any, default: Union[Any, _T]) -> Union[Any, _T]
Found 1 error in 1 file (checked 1 source file)

should maybe be output as

{
  "file": "_local/multi_error.py",
  "line": 5,
  "column": 8,
  "severity": "error",
  "context": "In function \"foo\":",
  "message": "No overload variant of \"get\" of \"Mapping\" matches argument type \"str\"",
  "hint": "Possible overload variants:\n    def get(self, key: Any) -> Any\n    def [_T] get(self, key: Any, default: Union[Any, _T]) -> Union[Any, _T]",
  "code": "call-overload"
}

But again, maybe that shouldn't be a blocker for this PR, getting machine-readable output can be useful even when findings aren't accurately grouped.

The negative line/column numbers right now seem awkward though:

{"file": "_local/multi_error.py", "line": -1, "column": -1, "severity": "note", "message": "In function \"foo\":", "code": null}

tusharsadhwani · 2021-12-08T19:49:05Z

That's definitely a bug.

The mypy code that generates this output was ridiculously coupled. I'll take a look at if this can be fixed.

TomMD · 2022-01-14T20:46:44Z

It has been many months as the project has hoped for a best solution over an existing solution. Can we merge this and accept future improvement using --output serif if someone comes along and implements it?

tusharsadhwani · 2022-01-19T09:30:14Z

@intgr I did it properly this time. Hints should be fixed now.

sehyun-hwang · 2022-02-22T07:44:00Z

--output json option gets ignored after some usage with mypy.api.run_dmypy . Pasing json option to dmypy CLI worked fine, but invoking run_dmypy function in a Python script returns plain string output after some usage. Can someone help me troubleshoot this?

tusharsadhwani · 2022-02-22T07:51:39Z

@sehyun-hwang sure thing. If you could provide a reproducible example I can look into it right away.

sehyun-hwang · 2022-02-23T01:59:15Z

@tusharsadhwani Thank you! I tried linting jobs in a container, and was unable to reproduce it. The problem occurs only when my IDE invokes dmypy, so I'm suspecting this has to do with concurrent execution. Do you see a chance where --output json option gets ignored when multiple clients are connected, or clients are threaded or running in a child process?

sehyun-hwang · 2022-02-23T02:28:51Z

I found that the first file linted output a json format, but from the second one it goes back to the string format. Can you try to reproduce this?

centos@www /m/tax-automation (99-basic-types) [1]> dmypy check -- batch_api.py
{"file": "batch_api.py", "line": 5, "column": 0, "message": "Cannot find implementation or library stub for module named \"LAMBDA_CONFIG\"", "hint": "See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports", "code": "import"}
{"file": "batch_api.py", "line": 6, "column": 0, "message": "Cannot find implementation or library stub for module named \"common.utils\"", "hint": "", "code": "import"}
centos@www /m/tax-automation (99-basic-types) [1]> dmypy check -- foo.py
foo.py:2: error: Name "bar" is not defined
foo.py:5: error: Argument 1 to "foo" has incompatible type "str"; expected "int"```

tusharsadhwani · 2022-02-23T06:30:32Z

@sehyun-hwang This seems like a dmypy bug. It seems that dmypy ignores cli flags or config file flags in certain cases.

Here's a file structure:

$ tree .
.
├── a.py
├── b.py
└── mypy.ini

And mypy.ini contains the following:

[mypy]
strict = True

a.py and b.py both contain:

def foo(): pass

This way mypy a.py won't show errors but mypy --strict a.py will.
When alternating between dmypy check a.py and dmypy check b.py, at some point dmypy stops showing strict output. For the same reason it might be forgetting the --output flag being set as well.

sehyun-hwang · 2022-02-23T07:18:16Z

#11396 (comment)

@intgr Could you take a look at the comment by @tusharsadhwani ?

intgr · 2022-02-23T10:10:52Z

I'm afraid I can't help much with this. Just to clarify, I think my name shows up in the "reviewers" list only because I left a code comment about this PR earlier. I'm not a maintainer or reviewer in an official capacity and I don't have much knowledge of mypy's internals.

I'm just a user interested in consuming structured output from mypy (and potentially adding SARIF support later on). That's why I shared my opinions in this comment thread.

tusharsadhwani · 2024-05-09T21:19:23Z

@JelleZijlstra CI is taken care of.

JelleZijlstra · 2024-05-09T21:20:21Z

I'm not sure we'd accept support for other output formats in mypy itself, since we'd take on additional maintenance load. If this PR is merged, such formats could be supported in external tools that consume mypy's JSON output.

JelleZijlstra · 2024-05-09T21:20:43Z

@JelleZijlstra CI is taken care of.

Thanks, I'll take a look soon.

2bndy5 · 2024-05-09T21:58:34Z

Yeah, the scope of this project has nothing to do with integrating various git servers' CI runners. That would be a rabbit hole of tech debt.

JelleZijlstra · 2024-05-09T23:48:12Z

mypy/errors.py

+                # No error tuple found for this hint. Ignoring it
+                continue
+
+            if error.hint == "":


Could we make hint into a list of strings instead?

It's often that a single hint is wrapped into multiple lines. For internal representaiton we can keep it as a list of strings but for the user I think it makes most sense to display it as a single string.

JelleZijlstra · 2024-05-09T23:48:38Z

mypy/errors.py

+            error_location = (file_path, line, column)
+            error = latest_error_at_location.get(error_location)
+            if error is None:
+                # No error tuple found for this hint. Ignoring it


Why not instead generate a MypyError with some field that lets us indicate that this is a note?

JelleZijlstra · 2024-05-09T23:49:00Z

mypy/options.py

        self.force_uppercase_builtins = False
        self.force_union_syntax = False

+        # Sets custom output format
+        self.output = ""


Would None be a better default?

JelleZijlstra · 2024-05-09T23:50:58Z

test-data/unit/outputjson.test

+foo('42')
+
+def bar() -> None: ...
+bar('42')


Add a test case with reveal_type(); let's make sure that shows up in the output.

intgr · 2024-05-10T10:08:37Z

I'm not sure we'd accept support for other output formats in mypy itself, since we'd take on additional maintenance load. If this PR is merged, such formats could be supported in external tools that consume mypy's JSON output.

If mypy will only support 1 machine-parsable output format, then please choose a standard or de facto recognized format. Don't create yet another tool-specific mypy output format that everybody then has to write another set of adapters for.

For example, tools already exist to convert SARIF into GitLab "CodeQuality" reports https://gitlab.com/ahogen/sarif-to-codequality.

github-actions · 2024-05-10T10:10:04Z

According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅

tusharsadhwani · 2024-05-10T10:28:37Z

FWIW I prefer current output as it is leaner and it's trivial for any tool to translate it to SARIF (pyright rejected the idea of adding SARIF for the same reason).

But I will add a SARIF formatter in the same PR if maintainers agree to it.

JelleZijlstra · 2024-05-10T13:00:01Z

test-data/unit/outputjson.test

 foo('42')

 def bar() -> None: ...
 bar('42')
 [out]
-{"file": "main", "line": 12, "column": 0, "message": "No overload variant of \"foo\" matches argument type \"str\"", "hint": "Possible overload variants:\n    def foo() -> None\n    def foo(x: int) -> None", "code": "call-overload"}
-{"file": "main", "line": 15, "column": 0, "message": "Too many arguments for \"bar\"", "hint": null, "code": "call-arg"}
+{"file": "main", "line": 12, "column": 12, "message": "Revealed type is \"Overload(def (), def (x: builtins.int))\"", "hint": null, "code": "misc", "is_note": true}


Could we change this to "severity": "note", mirroring mypy's Python code (and severity error for the errors)? That will also make it easier to extend to other kinds of severity in the future.

dgutson · 2024-05-10T14:12:46Z

FWIW I prefer current output as it is leaner and it's trivial for any tool to translate it to SARIF (pyright rejected the idea of adding SARIF for the same reason).

which I just commented.

But I will add a SARIF formatter in the same PR if maintainers agree to it.

Otherwise it will require yet another converter-to-sarif tool. There is plenty of support already in IDEs, linters aggregators, and other analysis tools. If you think that SARIF could be leaner, then I do propose you to participate in its committee (I'm not being cinic, I mean it, you may have a good idea), make your proposal, and discuss it. That would be the right approach so the whole static analysis community gets benefited.

JelleZijlstra · 2024-05-10T15:35:40Z

If I merge this PR, I take on the responsibility (at least in my mind) for triaging and reviewing future fixes to this area. I'm OK with doing that for this simple JSON format; I'm not OK with having to maintain compliance with the external SARIF spec. If another maintainer wants to add SARIF support, I'd welcome it, since clearly it will help some users, but I'm not interested in taking ownership of it.

github-actions · 2024-05-10T19:03:48Z

According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅

JelleZijlstra · 2024-05-10T19:03:54Z

I think this is close to ready.

We'll also want to document the JSON format in the mypy documentation, but that can wait for a separate PR.

tusharsadhwani · 2024-05-10T19:34:02Z

@dgutson Thanks for bringing this PR to my attention again :)

Otherwise it will require yet another converter-to-sarif tool

I just made and verified a scipt to run mypy with SARIF output, dropping it here incase it is useful to someone: https://gist.github.com/tusharsadhwani/5fd7a3c32d73f10fcb86e8deac4b60cf

dgutson · 2024-05-10T22:53:30Z

@dgutson Thanks for bringing this PR to my attention again :)

Otherwise it will require yet another converter-to-sarif tool

I just made and verified a scipt to run mypy with SARIF output, dropping it here incase it is useful to someone: https://gist.github.com/tusharsadhwani/5fd7a3c32d73f10fcb86e8deac4b60cf

could you please add it to the source tree? Otherwise it will be lost in a PR comment :)
Moreover, adding it as part of this PR would make sense for me.

tusharsadhwani · 2024-05-10T23:03:19Z

I think Jelle has made his stance clear. I'd rather not have this feature delayed potentially a few more months over that.

Although I don't mind SARIF support, I'm not particularly looking for it, and neither are the maintainers super stoked about it yet, it seems.

At this moment I think it'd be best to make a different issue for SARIF support. Feel free to use the gist as a basis for the PR against that issue.

patrick91 · 2024-05-27T15:07:40Z

@JelleZijlstra sorry for always being that person, but is this released?

Was just working on some automated tests for Strawberry and having JSON output would be make it much easier (we use that for pyright as well)

AlexWaygood · 2024-05-27T15:12:33Z

@JelleZijlstra sorry for always being that person, but is this released?

not yet released, but you can track the progress of the next release in #17285. It should include this PR.

JelleZijlstra · 2024-05-27T23:34:47Z

General tip: you can click on the commit page to see whether a commit has been included in a release. In this case, the commit (35fbd2a) shows no releases. A random earlier commit (304997b) shows that the commit is included in a number of releases:

Documents the new option added by #11396 --------- Co-authored-by: Shantanu <12621235+hauntsaninja@users.noreply.github.com>

tusharsadhwani added 7 commits October 24, 2021 19:54

Add -O/--output CLI option

393820c

Initial formatter setup

282bd28

Make error_formatter an optional argument

ccda5b0

Fix type annotation

c849a77

Fix whitespace

fd2feab

Remove whitespace

b188001

Merge branch 'master' of https://github.com/tusharsadhwani/mypy into …

51c1acc

…output-json

intgr reviewed Oct 28, 2021

View reviewed changes

tushar-deepsource and others added 3 commits January 19, 2022 13:01

Merge branch 'python:master' into output-json

9177dab

Add hint property to errors

bc5ceac

Fix lint issues

9d29ab0

97littleleaf11 requested a review from JukkaL February 22, 2022 07:46

JelleZijlstra self-requested a review May 9, 2024 21:20

JelleZijlstra reviewed May 9, 2024

View reviewed changes

address comments

e0e6896

tusharsadhwani requested a review from JelleZijlstra May 10, 2024 10:37

JelleZijlstra reviewed May 10, 2024

View reviewed changes

use severity

a0dc6d1

JelleZijlstra approved these changes May 10, 2024

View reviewed changes

JelleZijlstra merged commit 35fbd2a into python:master May 10, 2024
18 checks passed

tusharsadhwani deleted the output-json branch May 10, 2024 23:45

tsuyoshicho mentioned this pull request May 11, 2024

feat: JSON Output support tsuyoshicho/action-mypy#142

Open

This was referenced Jul 31, 2024

Document --output=json CLI option #17611

Merged

Add a new output format option to print errors as GitHub Actions Workflow commands #17612

Open

hauntsaninja added a commit that referenced this pull request Sep 27, 2024

Document --output=json CLI option (#17611)

1995155

Documents the new option added by #11396 --------- Co-authored-by: Shantanu <12621235+hauntsaninja@users.noreply.github.com>

Add Error format support, and JSON output option #11396

Add Error format support, and JSON output option #11396

Conversation

tusharsadhwani commented Oct 27, 2021 • edited Loading

Description

Demo:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

intgr Oct 28, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

intgr commented Oct 28, 2021 • edited Loading

nvuillam commented Dec 8, 2021

tusharsadhwani commented Dec 8, 2021

intgr commented Dec 8, 2021

nvuillam commented Dec 8, 2021

tusharsadhwani commented Dec 8, 2021 • edited Loading

nvuillam commented Dec 8, 2021

intgr commented Dec 8, 2021

tusharsadhwani commented Dec 8, 2021

TomMD commented Jan 14, 2022

tusharsadhwani commented Jan 19, 2022

sehyun-hwang commented Feb 22, 2022

tusharsadhwani commented Feb 22, 2022

sehyun-hwang commented Feb 23, 2022

sehyun-hwang commented Feb 23, 2022

tusharsadhwani commented Feb 23, 2022

sehyun-hwang commented Feb 23, 2022

intgr commented Feb 23, 2022 • edited Loading

tusharsadhwani commented May 9, 2024

JelleZijlstra commented May 9, 2024

JelleZijlstra commented May 9, 2024

2bndy5 commented May 9, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

intgr commented May 10, 2024 • edited Loading

github-actions bot commented May 10, 2024

tusharsadhwani commented May 10, 2024

Choose a reason for hiding this comment

dgutson commented May 10, 2024

JelleZijlstra commented May 10, 2024

github-actions bot commented May 10, 2024

JelleZijlstra commented May 10, 2024

tusharsadhwani commented May 10, 2024

dgutson commented May 10, 2024

tusharsadhwani commented May 10, 2024

patrick91 commented May 27, 2024 • edited Loading

AlexWaygood commented May 27, 2024 • edited Loading

JelleZijlstra commented May 27, 2024

tusharsadhwani commented Oct 27, 2021 •

edited

Loading

intgr Oct 28, 2021 •

edited

Loading

intgr commented Oct 28, 2021 •

edited

Loading

tusharsadhwani commented Dec 8, 2021 •

edited

Loading

intgr commented Feb 23, 2022 •

edited

Loading

intgr commented May 10, 2024 •

edited

Loading

patrick91 commented May 27, 2024 •

edited

Loading

AlexWaygood commented May 27, 2024 •

edited

Loading