Add support for pulling configuration from pyproject.toml files #10219

TheCleric · 2021-03-17T05:22:24Z

Description

This PR will add support to mypy for end users specifying configuration in a pyproject.toml file. . I also updated the documentation to indicate that this support has been added, along with some guidelines on adapting the ini configurations to the toml format.

Test Plan

I've added tests as best I could to mirror existing tests that use a mypy.ini file, but with a pyrproject.toml file, to ensure the configuration is being pulled correctly with the desired results.

TheCleric · 2021-03-17T17:37:40Z

Looking into the test failures. I didn't see these on my machine because I was using a venv and I think they got skipped.

…s for TOML parsing

TheCleric · 2021-03-19T01:46:01Z

Might need some help from someone with more Travis CI experience than I. It seems to be throwing strange python segmentation faults, which I am not sure where they are coming from.

JukkaL · 2021-03-19T12:08:55Z

It looks like you may have encountered a mypyc bug (mypyc is a compiler we use to compile mypy to C extensions). I'll try to debug this, since debugging mypyc issues can be non-trivial. Sorry about that!

TheCleric · 2021-03-19T14:40:15Z

It looks like you may have encountered a mypyc bug (mypyc is a compiler we use to compile mypy to C extensions). I'll try to debug this, since debugging mypyc issues can be non-trivial. Sorry about that!

Thanks so much!

mypy/config_parser.py

JukkaL · 2021-03-19T16:27:37Z

After fixing the above issue, compiled tests still fail for me. Here's what I see:

/Users/jukka/src/mypy/test-data/unit/check-custom-pyproject-plugin.test:557:
/Users/jukka/src/mypy/mypy/test/testcheck.py:138: in run_case
    self.run_case_once(testcase, ops, step)
/Users/jukka/src/mypy/mypy/test/testcheck.py:177: in run_case_once
    options = parse_options(original_program_text, testcase, incremental_step)
/Users/jukka/src/mypy/mypy/test/helpers.py:386: in parse_options
    targets, options = process_options(flag_list, require_targets=False)
mypy/main.py:855: in process_options
    ???
mypy/config_parser.py:200: in parse_config_file
    ???
mypy/config_parser.py:306: in parse_section
    ???
E   TypeError: str or None object expected; got float

This actually looks like it could be a legitimate error that mypyc was able to catch through strict runtime type checking. The type for argument section: Mapping[str, str] seems incorrect when using toml, as the value could be float, not just str.

TheCleric · 2021-03-19T16:28:53Z

This actually looks like it could be a legitimate error that mypyc was able to catch through strict runtime type checking. The type for argument section: Mapping[str, str] seems incorrect when using toml, as the value could be float, not just str.

Thanks, I'll fix!

TheCleric · 2021-03-20T02:15:34Z

@JukkaL any clue on the strange failures on 3.5.1 with mypyc? These seem to pass everywhere else (including locally on 3.5.1), but for some reason the output prints out of order on these.

TheCleric · 2021-04-01T14:40:53Z

I think I'm narrowing down the issues with 3.5. TOML doesn't use an OrderedDict by default like config_parser does, so I'm working on overriding that. In Python 3.6+ dicts are ordered by default which is why this is the only version with an issue. Will have a fix soon.

JelleZijlstra · 2021-04-01T15:45:28Z

We might drop support for 3.5 soon (#9950), so maybe it's not worth worrying about? I'd also be OK with not supporting pyproject.toml on 3.5 while we wait to confirm that we can fully drop support for 3.5.

TheCleric · 2021-04-01T15:47:21Z

We might drop support for 3.5 soon (#9950), so maybe it's not worth worrying about? I'd also be OK with not supporting pyproject.toml on 3.5 while we wait to confirm that we can fully drop support for 3.5.

I'd be fine either way. I'll defer to those who may know more than I. 😄

JukkaL · 2021-04-01T15:49:17Z

I'm fine with not supporting pyproject.toml on Python 3.5. In the next public release we can declare Python 3.5 as deprecated (even though Python 3.5 will likely still be mostly supported). Just give a clear error messages if somebody tries to use it on Python 3.5.

TheCleric · 2021-04-01T15:51:05Z

I'm fine with not supporting pyproject.toml on Python 3.5. In the next public release we can declare Python 3.5 as deprecated (even though Python 3.5 will likely still be mostly supported). Just give a clear error messages if somebody tries to use it on Python 3.5.

It actually works fine in Python 3.5, it's just one of the tests is failing because the order we store the data in the dict is variable since they aren't ordered. But the actual functionality is fine.

TheCleric · 2021-04-24T19:37:03Z

@JukkaL I think other than the boolean func I left a comment on above, all the requested changes have been made.

JukkaL

Thanks for the updates! This is almost ready to merge. It would be good to have more tests (but not as many as there were originally). Using ValueError in the boolean value parser seems okay.

JukkaL · 2021-04-30T16:28:37Z

test-data/unit/cmdline.pyproject.test

+-- messages and 1 if there are.
+
+-- Directories/packages on the command line
+-- ----------------------------------------


Could you add back some successful test cases as well that would cover all the essential features? It looks like all the test cases test failure modes.

Please add additional test cases to check-*.test files, since they are much faster than command-line tests.

Sorry, got a little overzealous in the removal. Will do.

TheCleric · 2021-04-30T19:45:41Z

Thanks for the updates! This is almost ready to merge. It would be good to have more tests (but not as many as there were originally). Using ValueError in the boolean value parser seems okay.

Alright @JukkaL, I tried to find a happy medium between too few and too many tests. 😆

Let me know if I got a better balance this time.

dciborow · 2021-04-30T21:53:58Z

as black is against supporting tox.ini, this is extra important to me to eventually have all of our configurations in a single spot...

psf/black#2172

Right now we can not configure both black and mypy into the same file as black lacks support for tox.ini

@JelleZijlstra , watching is closely.

(But am going to stand by the fact I think black should make the reverse effort here to support ini files as well, particularly if this does not gain traction again)

JukkaL

Thanks for the final batch of updates! Looks good now.

See python/mypy#10219

xtofl

Thanks for this functionality! I'm looking forward to having just one config file in my project.

I didn't find any errors, but I think some code may be expressed a bit nicer.

Kind regards

xtofl · 2021-05-12T12:20:39Z

docs/source/config_file.rst

+    # mypy global options:
+
+    [tool.mypy]
+    python_version = "2.7"


Is it wise to have deprecated python versions in new code?

xtofl · 2021-05-12T12:24:20Z

mypy/config_parser.py

@@ -34,6 +40,14 @@ def parse_version(v: str) -> Tuple[int, int]:
    return major, minor


+def try_split(v: Union[str, Sequence[str]], split_regex: str = '[,]') -> List[str]:


what is it called 'try_' for? This name implies the splitting may fail. Should it be just 'split'? Or 'polymorphic_split'?

xtofl · 2021-05-12T12:32:27Z

mypy/config_parser.py

+
+# Reuse the ini_config_types and overwrite the diff
+toml_config_types = ini_config_types.copy()  # type: Final[Dict[str, _INI_PARSER_CALLABLE]]
+toml_config_types.update({


Would it be possible to use the | operator?

tom_config_types = ini_config_types | { 'python_version': lambda s: parse_version(str(s)), ... }

Mypy works with 3.5+ and this change would break everything on <3.9

xtofl · 2021-05-12T12:33:47Z

mypy/config_parser.py

@@ -121,14 +162,27 @@ def parse_config_file(options: Options, set_strict_flags: Callable[[], None],
    else:
        config_files = tuple(map(os.path.expanduser, defaults.CONFIG_FILES))

-    parser = configparser.RawConfigParser()
+    config_parser = configparser.RawConfigParser()


This name change looks unnecessary and adds confusion: it differs only by whitespace.

xtofl · 2021-05-12T12:53:02Z

mypy/config_parser.py

+                if 'mypy' not in toml_data:
+                    continue
+                toml_data = OrderedDict({'mypy': toml_data['mypy']})
+                parser = destructure_overrides(toml_data)  # type: MutableMapping[str, Any]


This logic complicates the loop body; would it be hard to extract a method? I think the responsibilities are better separated with something like:

def create_parser(config_file): if is_toml(config_file): return toml_parser(config_file), toml_config_types else: return ini_parser(config_file), ini_config_types ... try: parser, config_types = create_parser(config_file) except (toml.TomlDecodeError, configparser.Error, ConfigTOMLValueError) as err: ...

xtofl · 2021-05-12T12:59:50Z

mypy/config_parser.py

        for k, v in updates.items():
            setattr(options, k, v)
        options.report_dirs.update(report_dirs)

    for name, section in parser.items():
        if name.startswith('mypy-'):
-            prefix = '%s: [%s]: ' % (file_read, name)
+            prefix = get_prefix(file_read, name)


Actually, once we know we're parsing a TOML file, we know what get_prefix is going to do. As such, this can also become part of the return values of the proposed create_parser function.

xtofl · 2021-05-12T13:42:31Z

mypy/config_parser.py

+    return filemame.lower().endswith('.toml')
+
+
+def destructure_overrides(toml_data: "OrderedDict[str, Any]") -> "OrderedDict[str, Any]":


I had a hard time guessing what 'destructure' meant. Would 'overrides_by_module' be a better name? This reflects what is achieved, not what the code is doing.

xtofl · 2021-05-12T13:49:55Z

mypy/config_parser.py

+            if old_config_name not in result:
+                result[old_config_name] = module_overrides
+            else:
+                for new_key, new_value in module_overrides.items():


Does allowing 'accidental but equivalent' double entries bring more convenience, or would they rather hide a dormant configuration error?

[style] Also, since this is the exceptional case, shouldn't it stand out as such?

if old_config_name in result: raise ConfigTOMLValueError("...") result[old_config_name] = module_overrides

xtofl · 2021-05-12T13:55:18Z

mypy/config_parser.py

+    if isinstance(value, bool):
+        return value
+    if not isinstance(value, str):
+        value = str(value)


How is None handled here?

xtofl · 2021-05-12T14:01:18Z

mypy/config_parser.py

+    if not is_toml(filename):
+        return ", ".join("[mypy-%s]" % module for module in modules)
+
+    return "module = ['%s']" % ("', '".join(sorted(modules)))


This looks a bit weird. The quotes can be applied directly:

quoted = '"{}"'.format return "module = [%s]" % ", ".join(map(quoted, sorted(modules)))

TheCleric added 3 commits March 17, 2021 00:45

Add gitignore entries for pyenv

8059a6d

Add support for pulling configuration from pyproject.toml files

b86c6c0

Allow pyproject.toml with no mypy section

6435a9d

TheCleric and others added 9 commits March 17, 2021 14:55

Move pyproject tests to their own test

9f70b29

Further delineate pyproject tests

da965ec

Escape special chars with single quotes

fe201b5

Update docs for pyproject.toml

76d6b6c

Fix some (the rest🤞) of the pyproject tests

6618f2c

Remove unnessecary .test changes

76e1197

Restore config_parser functions to original typing and create new one…

34f246b

…s for TOML parsing

Helps if you get the types right BEFORE commiting 🤦

df8db18

3.5 doesn't have typing.Type?

94de3d5

JukkaL reviewed Mar 19, 2021

View reviewed changes

mypy/config_parser.py Outdated Show resolved Hide resolved

JukkaL mentioned this pull request Mar 19, 2021

Investigate crash in compiled mypy tests mypyc/mypyc#813

Closed

TheCleric added 2 commits March 19, 2021 14:33

Deepcopy segfaults mypyc

60a4558

Fix typing issue found by mypyc

0278242

JelleZijlstra mentioned this pull request Apr 1, 2021

Drop support for running on Python 3.5 #9950

Closed

Use OrderedDict to preserve option orders in python 3.5

771964c

TheCleric added 5 commits April 23, 2021 22:15

Convert TOML parse ValueError to custom ValueError

b7e02e6

Add tests to ensure new errors are caught

0a44423

Ensure toml load stays an OrderedDict on copy

8f0e6d2

Strip pyproject testing down now that we know it works

3416337

Fix some mistyping

ddb3922

TheCleric added 3 commits April 24, 2021 15:37

Merge branch 'master' into feature/pyproject_toml_support

6e5dc11

Merge branch 'master' into feature/pyproject_toml_support

74c29e7

Merge branch 'master' into feature/pyproject_toml_support

3249676

JukkaL reviewed Apr 30, 2021

View reviewed changes

TheCleric added 2 commits April 30, 2021 12:57

Merge branch 'master' into feature/pyproject_toml_support

c7110fa

Flesh out some feature tests of the pyproject.toml config

b4f7d79

JelleZijlstra mentioned this pull request Apr 30, 2021

Support Configuration via tox.ini psf/black#2172

Closed

johnthagen mentioned this pull request May 1, 2021

Migrate mypy config to pyproject.toml johnthagen/python-blueprint#23

Closed

TheCleric added 2 commits May 2, 2021 00:39

Fix double to single quotes for test paths

d9d091e

Merge branch 'master' into feature/pyproject_toml_support

742059f

JukkaL approved these changes May 5, 2021

View reviewed changes

JukkaL merged commit fdeedd6 into python:master May 5, 2021

branchvincent mentioned this pull request May 5, 2021

move mypy to supported section carlosperate/awesome-pyproject#43

Merged

7 tasks

Gobot1234 added a commit to Gobot1234/steam.py that referenced this pull request May 5, 2021

Drop mypy.ini in favour of pyproject.toml

5802c58

See python/mypy#10219

eganjs mentioned this pull request May 11, 2021

Add support for reading config from pyproject.toml PyCQA/flake8#1332

Closed

xtofl reviewed May 12, 2021

View reviewed changes

TheCleric deleted the feature/pyproject_toml_support branch May 13, 2021 00:23

hukkin mentioned this pull request Jul 14, 2021

Switch from toml to tomli for TOML v1 support #10824

Merged

IMBlues mentioned this pull request Jul 16, 2021

use-pyprojects-toml-as-project-tool IMBlues/IMBlues.github.io#4

Open

CAM-Gerlach mentioned this pull request Jul 25, 2021

Mypy now supports pyproject.toml flying-sheep/awesome-python-packaging#6

Closed

weibullguy mentioned this pull request Jul 26, 2021

Migrate mypy Configuration from setup.cfg to pyproject.toml ReliaQualAssociates/ramstk#614

Closed

1 task

nstarman mentioned this pull request Oct 27, 2021

pre-commit isort astropy/astropy#12276

Closed

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for pulling configuration from pyproject.toml files #10219

Add support for pulling configuration from pyproject.toml files #10219

TheCleric commented Mar 17, 2021

TheCleric commented Mar 17, 2021

TheCleric commented Mar 19, 2021

JukkaL commented Mar 19, 2021

TheCleric commented Mar 19, 2021

JukkaL commented Mar 19, 2021

TheCleric commented Mar 19, 2021

TheCleric commented Mar 20, 2021

TheCleric commented Apr 1, 2021

JelleZijlstra commented Apr 1, 2021

TheCleric commented Apr 1, 2021

JukkaL commented Apr 1, 2021

TheCleric commented Apr 1, 2021

TheCleric commented Apr 24, 2021

JukkaL left a comment

JukkaL Apr 30, 2021

TheCleric Apr 30, 2021

TheCleric commented Apr 30, 2021

dciborow commented Apr 30, 2021 •

edited

Loading

JukkaL left a comment

xtofl left a comment

xtofl May 12, 2021

xtofl May 12, 2021

xtofl May 12, 2021

Gobot1234 May 12, 2021

xtofl May 12, 2021

xtofl May 12, 2021

xtofl May 12, 2021

xtofl May 12, 2021

xtofl May 12, 2021

xtofl May 12, 2021

xtofl May 12, 2021

		@@ -34,6 +40,14 @@ def parse_version(v: str) -> Tuple[int, int]:
		return major, minor


		def try_split(v: Union[str, Sequence[str]], split_regex: str = '[,]') -> List[str]:

		return filemame.lower().endswith('.toml')


		def destructure_overrides(toml_data: "OrderedDict[str, Any]") -> "OrderedDict[str, Any]":

Add support for pulling configuration from pyproject.toml files #10219

Add support for pulling configuration from pyproject.toml files #10219

Conversation

TheCleric commented Mar 17, 2021

Description

Test Plan

TheCleric commented Mar 17, 2021

TheCleric commented Mar 19, 2021

JukkaL commented Mar 19, 2021

TheCleric commented Mar 19, 2021

JukkaL commented Mar 19, 2021

TheCleric commented Mar 19, 2021

TheCleric commented Mar 20, 2021

TheCleric commented Apr 1, 2021

JelleZijlstra commented Apr 1, 2021

TheCleric commented Apr 1, 2021

JukkaL commented Apr 1, 2021

TheCleric commented Apr 1, 2021

TheCleric commented Apr 24, 2021

JukkaL left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TheCleric commented Apr 30, 2021

dciborow commented Apr 30, 2021 • edited Loading

JukkaL left a comment

Choose a reason for hiding this comment

xtofl left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dciborow commented Apr 30, 2021 •

edited

Loading