Provide a toml module in the standard library

[mgorny]

BPO	40059
Nosy	@brettcannon, @tiran, @mcepl, @njsmith, @encukou, @agronholm, @mgorny, @dstufft, @pradyunsg, @eli-schwartz, @miss-islington, @tirkarthi, @skoslowski, @erlend-aasland, @hauntsaninja, @domdfcoding, @hukkin, @YakoYakoYokuYoku
PRs	bpo-40059: tomllib #31498 bpo-40059: Fix installation of tomllib #31784

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-03-25.06:54:21.437>
labels = ['type-feature', 'library']
title = 'Provide a toml module in the standard library'
updated_at = <Date 2022-03-09.13:38:15.300>
user = 'https://github.com/mgorny'

bugs.python.org fields:

activity = <Date 2022-03-09.13:38:15.300>
actor = 'miss-islington'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2020-03-25.06:54:21.437>
creator = 'mgorny'
dependencies = []
files = []
hgrepos = []
issue_num = 40059
keywords = ['patch']
message_count = 27.0
messages = ['364979', '364983', '365000', '373879', '387807', '387892', '401071', '401421', '406458', '406484', '406488', '406515', '406561', '408316', '408320', '408418', '408503', '408505', '408576', '409488', '409489', '409552', '410353', '414731', '414746', '414800', '414801']
nosy_count = 20.0
nosy_names = ['brett.cannon', 'christian.heimes', 'mcepl', 'njs', 'petr.viktorin', 'alex.gronholm', 'mgorny', 'dstufft', 'pradyunsg', 'eschwartz', 'miss-islington', 'VA', 'xtreak', 'skoslowski', 'erlendaasland', 'hauntsaninja', 'domdfcoding', 'dom1310df', 'hukkinj1', 'YakoYakoYokuYoku']
pr_nums = ['31498', '31784']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue40059'
versions = []

[mgorny]

PEP-518 uses the TOML format to specify build system requirements. AFAIU this means that all new build systems will require a TOML parser. Could you consider adding one to the standard library to reduce the number of chicken-egg problems?

The referenced PEP states that 'pytoml TOML parser is ~300 lines of pure Python code', so I don't think integrating it would be a large maintenance cost.

[1] https://www.python.org/dev/peps/pep-0518/

[tirkarthi]

Relevant python-dev discussion : https://mail.python.org/pipermail/python-dev/2019-May/157405.html . The format has still not reached 1.0. Issue to track the 1.0 release candidate : toml-lang/toml#698

[brettcannon]

The plan is to start discussing adding a TOML parser once the spec reaches 1.0 (which we will know about as one of the pip contributors also manages TOML).

[VA]

1.0.0-rc.1 is out by now: https://github.com/toml-lang/toml/blob/master/CHANGELOG.md

[mgorny]

It seems that the spec has reached 1.0.0:

1.0.0 / 2021-01-11

I guess that there are still ~2 months left to do this before 3.10 reaches beta.

[brettcannon]

We will see if anyone has enough time to get this done between now and May for 3.10, but discussions have been started with some folks to propose something for the stdlib.

[mgorny]

Is there any progress happening? FWIU the common "toml" module on pypi has been discontinued now, projects are switching to yet another one and this is exactly the kind of problem a built-in module would have avoided.

[brettcannon]

No progress as I've been swamped with higher-priority things and the bigger discussion about how we want to manage the stdlib going forward has not started yet (once again, not had the time to start that).

[hauntsaninja]

I'd love it if we could make this happen for Python 3.11. What can I do to help, e.g. would it be helpful if I drafted a PEP?

Several projects have switched to using tomli (https://github.com/hukkin/tomli), which is fully compatible with TOML v1.0.0 and is maybe 800 LoC with claimed 100% branch coverage.

There's also tomlkit, which has been around for longer, but is perhaps larger and more feature-ful than we'd want in stdlib, e.g. it allows you to roundtrip dump and load TOML documents preserving comments, whitespace, etc.

[brettcannon]

I'd love it if we could make this happen for Python 3.11. What can I do to help, e.g. would it be helpful if I drafted a PEP?

A PEP isn't necessarily required right now as https://discuss.python.org/t/how-do-we-want-to-manage-additions-removals-to-the-stdlib/10681 needs to be resolved by the steering council first (it's on our agenda).

Several projects have switched to using tomli (https://github.com/hukkin/tomli), which is fully compatible with TOML v1.0.0 and is maybe 800 LoC with claimed 100% branch coverage.

Part of the issue with tomli is how new it is; https://pypi.org/project/tomli/#history shows it is only 6 months old and we typically require a project be at least a year old before we consider pulling it in.

If you want to help you could ask the author of tomli what they think about it being added to the stdlib and whether they support that idea.

[mgorny]

I've already asked that, and the author said it's fine but with the deprecated text file support removed:

hukkin/tomli#141 (comment)

That said, for consistency with json and so on we'd probably want to combine tomli and tomli_w to get dumping support as well.

[hauntsaninja]

Hopefully tomli being less than a year old should not be a blocker, after all, TOML v1.0.0 is itself less than a year old.

Despite tomli being new code, it's pretty widely used. Reverse dependencies include pip, pytest, mypy, black, flit, coverage, setuptools-scm, cibuildwheel. All in all, it should be well proven in practice by the time it comes to make 3.11 stable.

It also helps that it's easy to anticipate the interface: dump, dumps, load, loads. Overall, hopefully risk here should be well within the budget of a provisional module.

That's all to say, if and when the Steering Council decides that there's work to be done here, let me know how I can help :-)

[brettcannon]

Hopefully tomli being less than a year old should not be a blocker, after all, TOML v1.0.0 is itself less than a year old.

It's actually rather important as the semantics of how things work will effectively be locked in once a module is added to the stdlib, so we need to be careful. Plus had you asked this question less than a year go the answer would have been pypi.org/p/toml and now you're suggesting something different, so the year matters. 😄

[YakoYakoYokuYoku]

A new python-ideas mail thread was created for this, you can check it out at https://mail.python.org/archives/list/python-ideas@python.org/thread/IWJ3I32A4TY6CIVQ6ONPEBPWP4TOV2V7/.

[tiran]

I just noticed that tomli has dropped support for Python 3.6. That's a road block for general adoption of the package in the Python ecosystem. Python 3.6 is the default Python interpreter in CentOS 8, C8S, RHEL 8, and Ubuntu 18.04 LTS. hukkin/tomli#134

[hauntsaninja]

Given that this currently seems blocked on the broad question of "how should additions and removals to the stdlib be managed", I'd like to not focus too hard just yet on the specifics of tomli. I assume it's unlikely, but for all we know, the SC could determine that all newly included modules have to be written from scratch, following along the lines of recent additions like zoneinfo, graphlib and importlib.metadata.

There's a lot we could bikeshed or debate... e.g., it's not even clear to me what a toml package in the stdlib would be named, never mind what it means for an unreleased version of a commonly used third party toml package dropping support for an imminently EOL version of Python. I'm possibly totally out of line, but in my head, the process would look something like this:

Step 1: Hear back from the SC about criteria and considerations for new modules in the stdlib
Step 2: Determine whether a TOML library *in the abstract* is something that would meet the outlined criteria (potentially e.g. is this something we even want? is this something we can maintain?)
Step 3: Determine if we have an implementation (written from scratch, copied, or derived from something pre-existing) that would meet the outlined criteria
Step 4: Do all the rest of the work to meet the outlined criteria (potentially e.g. go through PEP process, create proposed impl, write a backport, bikeshed api and name, etc)

I guess I have the following questions:

• Is my understanding correct that this issue is blocked on SC guidance?
• Is there anything we could do in advance of SC guidance that would be productive?
  Brett previously mentioned bringing it up with the author of tomli, as per Please consider pushing tomli into stdlib hukkin/tomli#141 they seem supportive
• Is there a good place to follow along or be notified of SC thoughts?
  I see no mention of stdlib changes in https://github.com/python/steering-council and the discuss thread linked above seems to have petered out.

[brettcannon]

I just noticed that tomli has dropped support for Python 3.6. That's a road block for general adoption of the package in the Python ecosystem.

It's already in pip, so I think it's already generally adopted 😉. https://github.com/pypa/pip/tree/main/src/pip/_vendor/tomli

Is my understanding correct that this issue is blocked on SC guidance?

Not officially, no. But I'm personally not going to bring it forward right now. If someone else wants to formulate a complete proposal for the SC on this then they are definitely welcome to! You will need to address where the code is coming from, why that code should be used, what's the API, etc.

The only reason the SC is mentioned here is there will be a discussion about how to maintain the stdlib, but it simply hasn't happened yet. You don't have to wait for it and asking for a TOML module might actually force the issue.

Is there anything we could do in advance of SC guidance that would be productive?

Nope, someone eventually has to have the time to make the proposal and manage the deluge of comments.

Is there a good place to follow along or be notified of SC thoughts?

https://github.com/python/steering-council as you already pointed out through the issues and monthly summaries. Otherwise you just need to open an issue and ask. 😃

[YakoYakoYokuYoku]

Not officially, no. But I'm personally not going to bring it forward right now. If someone else wants to formulate a complete proposal for the SC on this then they are definitely welcome to! You will need to address where the code is coming from, why that code should be used, what's the API, etc.

The only reason the SC is mentioned here is there will be a discussion about how to maintain the stdlib, but it simply hasn't happened yet. You don't have to wait for it and asking for a TOML module might actually force the issue.

I've been working on a document detailing the API implementation (as a PEP draft) see https://gitlab.com/YakoYakoYokuYoku/pep-toml. Although I've yet to write the code I'll to move forward with it.

[brettcannon]

I opened python/steering-council#92 for the SC to discuss stdlib additions in case I am not re-elected.

[hauntsaninja]

You will need to address where the code is coming from, why that code should be used, what's the API, etc.

Happy new year, potentially-toml-wanting friends!

I wrote up a draft of a proposal here: https://gist.github.com/hauntsaninja/9f136a5a60f63d8ca2cdfadb50edba44

I've passed it along to hukkin (author of tomli) for feedback. Once they reply, I'll send it to python-dev and the deluge of comments can get underway. If in the meantime anyone else has thoughts, I'd love to hear from you.

[hauntsaninja]

pradyunsg kindly pointed me to an ongoing thread in Packaging discuss: https://discuss.python.org/t/adopting-recommending-a-toml-parser/4068/84

So seems like a PEP will be necessary. I'm happy to do legwork for that. (The PEP that I'll write differs from YakoYakoYokuYoku's json-esque draft in that it'll be based on tomli instead of a from-scratch implementation)

[hauntsaninja]

We've started a PEP draft.

https://github.com/hauntsaninja/peps/blob/toml-pep/pep-9999.rst shows a rendered version.
hauntsaninja/peps#1 is a PR from the toml-pep branch to main, to help ease of review / discussion.

If you're following along and need to get your bearings straight, here are the recently active locations of discussion:

• this issue
• Please consider pushing tomli into stdlib hukkin/tomli#141
• https://discuss.python.org/t/adopting-recommending-a-toml-parser/4068/84
• PEP ???: TOML hauntsaninja/peps#1

[hauntsaninja]

This is now PEP-680: https://www.python.org/dev/peps/pep-0680/
Currently being discussed at: https://discuss.python.org/t/pep-680-tomllib-support-for-parsing-toml-in-the-standard-library/13040

[encukou]

New changeset 591f675 by Taneli Hukkinen in branch 'main':
bpo-40059: Add tomllib (PEP-680) (GH-31498)
591f675

[encukou]

The PR is merged and buildbots are green. Thank you to everyone who helped!

Now would be a good time to bikeshed wording in the documentation.

From the PR:

Would it be good to mention in the docs why load() takes only binary files? The encoding requirement probably isn't obvious for first-time users.

[dom1310df]

When building Python from source (as of the latest GitHub commit) the tomllib directory doesn't actually get copied over to the install prefix.

It looks like an entry's needed in Makefile.pre.in under LIBSUBDIRS, along the lines of #13563 and #30311

[miss-islington]

New changeset 23dcea5 by Dominic Davis-Foster in branch 'main':
bpo-40059: Fix installation of tomllib (GH-31784)
23dcea5