Add a new Include/cpython/ subdirectory for the "CPython API" with implementation details

[vstinner]

BPO	35134
Nosy	@brettcannon, @ncoghlan, @vstinner, @jkloth, @ericsnowcurrently, @serhiy-storchaka, @shihai1991, @erlend-aasland, @nw0
PRs	[WIP] bpo-35134: Create Include/pycapi/ subdirectory #10285 bpo-35134: Create Include/cpython/ subdirectory #10624 bpo-35134: Create Include/cpython/object.h #10679 bpo-35134: Create Include/cpython/unicodeobject.h #10680 bpo-35134: Add Include/cpython/pyerrors.h #10727 bpo-35134: Create Include/cpython/abstract.h #10728 bpo-35134: Create Include/cpython/pylifecycle.h #10731 bpo-35134: Create Include/cpython/dictobject.h #10732 bpo-35134: Create Include/cpython/pystate.h #10733 bpo-35134: Update "make tags": add Include/cpython/ #10739 bpo-35134: Don't define types twice in header files #10754 bpo-35134: Create Include/cpython/tupleobject.h #10764 bpo-35134: Add Include/cpython/pymem.h #12840 bpo-35134: Add cpython/pymem.h to build system #12842 bpo-35134: Split traceback.h header #13430 bpo-35134: Register new traceback.h header files #13431 bpo-35134: Add Include/cpython/import.h header file #14213 bpo-35134: Migrate frameobject.h contents to cpython/frameobject.h #18052 bpo-35134: Create Include/cpython/listobject.h #18395 [WIP] bpo-35134: Move header files to Include/cpython/ #18490 bpo-35134: Add Include/cpython/fileutils.h header file #18493 bpo-35134: Add Include/cpython/bytesobject.h file #18494 bpo-40421: Add Include/cpython/code.h header file #19756 bpo-35134: Add Include/cpython/pythonrun.h file #23701 bpo-35134: Add Include/cpython/pytime.h file #23988 bpo-35134: Move Include/{pyarena.h,pyctype.h} to Include/cpython #24550 bpo-35134: Move non-limited C API files to Include/cpython/ #24561 bpo-35134: move asdl.h, pystrhex.h, symtable.h, token.h, tracemalloc.h into Include/cpython #24770 bpo-35134: Move non-limited C API of include/compile.h into include/cpython. #24922 bpo-35134: Add Include/cpython/floatobject.h #28957 bpo-35134: Move Include/funcobject.h to Include/cpython/ #28958 bpo-35134: Move Include/cellobject.h to Include/cpython/ #28964 bpo-35134: Move classobject.h to Include/cpython/ #28968 bpo-35134: Split warnings.h and weakrefobject.h #29042 bpo-35134: Add Include/cpython/longobject.h #29044 bpo-35134: Add Include/cpython/descrobject.h #30923 bpo-35134: Add Include/cpython/complexobject.h header #32383 bpo-35134: Add Include/cpython/setobject.h header #32384 bpo-35134: Remove the Include/code.h header file #32385

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2022-01-30.23:33:02.262>
created_at = <Date 2018-11-01.12:46:49.299>
labels = ['interpreter-core', '3.8']
title = 'Add a new Include/cpython/ subdirectory for the "CPython API" with implementation details'
updated_at = <Date 2022-04-07.00:29:55.966>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2022-04-07.00:29:55.966>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2022-01-30.23:33:02.262>
closer = 'vstinner'
components = ['Interpreter Core']
creation = <Date 2018-11-01.12:46:49.299>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 35134
keywords = ['patch']
message_count = 61.0
messages = ['329060', '329087', '329123', '329124', '329126', '329290', '329292', '330161', '330162', '330165', '330246', '330261', '330276', '330282', '330283', '330285', '330296', '330324', '330331', '330336', '330337', '330446', '330447', '330463', '330467', '330477', '330478', '330480', '330512', '330556', '330557', '330594', '340283', '340285', '342876', '342881', '346014', '360338', '361542', '361922', '361923', '367536', '382771', '383970', '387105', '387179', '387327', '389285', '394491', '403952', '403955', '403957', '403964', '403967', '403983', '404248', '404252', '411772', '416906', '416910', '416912']
nosy_count = 9.0
nosy_names = ['brett.cannon', 'ncoghlan', 'vstinner', 'jkloth', 'eric.snow', 'serhiy.storchaka', 'shihai1991', 'erlendaasland', 'nw0']
pr_nums = ['10285', '10624', '10679', '10680', '10727', '10728', '10731', '10732', '10733', '10739', '10754', '10764', '12840', '12842', '13430', '13431', '14213', '18052', '18395', '18490', '18493', '18494', '19756', '23701', '23988', '24550', '24561', '24770', '24922', '28957', '28958', '28964', '28968', '29042', '29044', '30923', '32383', '32384', '32385']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue35134'
versions = ['Python 3.8']

Linked PRs

• gh-79315: Add Include/cpython/memoryobject.h header #99723

[vstinner]

The PEP-384 "Defining a Stable ABI" introduced Py_LIMITED_API define to exclude functions from the Python C API. The problem is when a new API is introduced, it has to explicitly be excluded using "#ifndef Py_LIMITED_API". If the author forgets it, the function is added to be stable API by mistake.

I propose to move the API which should be excluded from the stable ABI to a new subdirectory: Include/pycapi/.

To not break the backward compatibility, I propose to automatically include new header files from existing header files. For example, Include/pycapi/pyapi_objimpl.h would be automatically included by Include/pycapi/pycapi_objimpl.h.

New header files would have a "pycapi_" prefix to avoid conflict Include/ header files, if Include/pycapi/ directory is in the header search paths.

This change is a follow-up of bpo-35081 which moved Py_BUILD_CORE code to Include/internal/.

It is also part of a larger project to cleanup the C API, see:

• https://pythoncapi.readthedocs.io/split_include.html
• https://pythoncapi.readthedocs.io/

The change is backward compatible: #include <Python.h> will still provide exactly the same API.

[serhiy-storchaka]

There are not just two sides. It is common to wrap new stable C API with something like:

#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x03050000

What will you do with this?

[vstinner]

There are not just two sides. It is common to wrap new stable C API with something like:
#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x03050000
What will you do with this?

objimpl.h always includes pycapi/pycapi_objimpl.h, so I don't think that we need a strong rules. I propose to always add move code using "#if ... Py_LIMITED_API" to the pycapi/ subdirectory, even if it uses "#if !defined(Py_LIMITED_API)".

[serhiy-storchaka]

Do you want to keep only stable ABI v.3.2 and move both newer stable API and non-stable API to the pycapi/ subdirectory? Sorry, I don't found a sense in this.

[vstinner]

Do you want to keep only stable ABI v.3.2 and move both newer stable API and non-stable API to the pycapi/ subdirectory? Sorry, I don't found a sense in this.

The raw definition could be that Include/.h is part of the stable ABI, and Include/pycapi/.h are the definitions using Py_LIMITED_API and so can be stable or not stable depending on Py_LIMITED_API value :-)

To be honest, I'm not sure that I understand how "Py_LIMITED_API+0 >= 0x03050000" works and should be used.

I understand that you would prefer to leave PyObject_Calloc() in Include/objimpl.h. Honestly, I have no strong opinion on that. We can leave it there if you prefer.

--

Maybe the rule "move everything using Py_LIMITED_API to pycapi" is misleading.

My intent is that API in Include/*.h should not leak implementation details. It should be the starting point to design a new C API which does not leak any implementation detail:
http://pythoncapi.readthedocs.io/

It's easier with an example:

#define _PyObject_GC_TRACK(o) do { \
    PyGC_Head *g = _Py_AS_GC(o); \
    if (g->_gc_next != 0) { \
        Py_FatalError("GC object already tracked"); \
    } \
    assert((g->_gc_prev & _PyGC_PREV_MASK_COLLECTING) == 0); \
    ...

This macro is private: it starts with "_Py", so it doesn't belong to Include/*.h. Moreover, it access private fields like PyGC_Head._gc_prev.

From my point of view, the ideal API would not access *any* structure field and PyGC_Header structure must not be used nor part of the C API.

--

After saying that, I looked again at my PR, and I still see private functions in objimpl.h. Example:

PyAPI_FUNC(PyObject *) _PyObject_New(PyTypeObject *);
PyAPI_FUNC(PyVarObject *) _PyObject_NewVar(PyTypeObject *, Py_ssize_t);

#define PyObject_New(type, typeobj) \
                ( (type *) _PyObject_New(typeobj) )
#define PyObject_NewVar(type, typeobj, n) \
                ( (type *) _PyObject_NewVar((typeobj), (n)) )

These functions are not excluded from Py_LIMITED_API. Since they are private, we are free to remove them whenever we want, so IMHO it's fine to exclude from Py_LIMITED_API right now if we want.

Another example:

static inline PyObject*
PyObject_INIT(PyObject *op, PyTypeObject *typeobj)
{
    assert(op != NULL);
    Py_TYPE(op) = typeobj;
    _Py_NewReference(op);
    return op;
}

It's a public function but it calls the private function _Py_NewReference(). So _Py_NewReference() must be part of Py_LIMITED_API somehow...

In release mode (if Py_TRACE_REFS is not defined), _Py_NewReference() is defined like that:

/* Without Py_TRACE_REFS, there's little enough to do that we expand code
   inline. */
static inline void _Py_NewReference(PyObject *op)
{
    if (_Py_tracemalloc_config.tracing) {
        _PyTraceMalloc_NewReference(op);
    }
    _Py_INC_TPALLOCS(op);
    _Py_INC_REFTOTAL;
    Py_REFCNT(op) = 1;
}

It does access to the private _Py_tracemalloc_config variable and private macros/functions _Py_INC_TPALLOCS(op) and _Py_INC_REFTOTAL.

We *can* always define _Py_NewReference() as a function call if Py_LIMITED_API is defined, but it would have an impact on performance.

Right now, I don't want to risk to introduce a performance slowdown.

I have a "Proof-of-concept" implementation of my proposed "new C API":
https://github.com/pythoncapi/cpython/

My implementation currently uses 3 defines:

• Py_NEWCAPI_NO_MACRO: replace macros with function calls PyTuple_GET_SIZE() becomes PyTuple_Size()
• Py_NEWCAPI_NO_STRUCT: must not use PyObject.ob_refcnt or any other field of Python object structures; structures should hide their fields: compilation error.
• Py_NEWCAPI: new C API without borrowed references, without macro, without struct

But this project is highly experimental and I don't want to make it upstream before we measured properly the impact on the performance, the API has been properly reviewed and discussed, and the overall project has been approved by core developers. For example, by writing a PEP :-)

--

In short, I'm not sure of what can or should be done right now for Include/pycapi/ :-)

I wrote the PR to open the discussion :-)

[ncoghlan]

To be honest, I'm not sure that I understand how "Py_LIMITED_API+0 >= 0x03050000" works and should be used.

It's described here: https://docs.python.org/3/c-api/stable.html

If a stable ABI consumer just declares "#define PY_LIMITED_API 1", then they'll get the original stable ABI as defined in Python 3.2.

If they don't care about versions prior to 3.6, they can instead declare "#define PY_LIMITED_API 0x03060000", and get access to the functions added to the stable ABI in 3.3, 3.4, 3.5, and 3.6.

For this PR though, I think it's OK to ignore that detail, as once all the internal APIs are in "Include/internal", and all the APIs that don't offer ABI stability guarantees are in "Include/TBD" (see note below), then the general rule to follow is that everything added to the headers directly in "Include/" needs a Py_LIMITED_API guard that matches the upcoming release.

Note: I wrote "TBD" rather than "pycapi" above, as "pycapi" sounds like the name of a preferred public API to me, rather than "code compiled against this API is not portable to later versions, and may not be portable to other implementations". Given the name of the macro, "Include/unlimited/.h" may make sense, especially if those header files are all written to error out at compile time if PY_LIMITED_API is defined. "Include/unstable_abi/.h" would be another self-describing name.

[ncoghlan]

On actually looking at the initial changes in the PR:

• declarations that aren't part of the stable ABI in any version (i.e. "#ifndef PY_LIMITED_API", "#if !defined(PY_LIMITED_API)") should move to the new directory

• declarations that are part of the stable ABI in *some* version should remain where they are (i.e. in "Include/*.h")

In your initial PR, the only API that subtle distinction affects is PyObject_Calloc (since that's a new addition to the stable ABI in 3.5+), and moving that back to the public header means you can add the desired "Py_LIMITED_API is not defined" check to the header in the new directory.

[vstinner]

I created a new PR 10624:

• move "#ifndef Py_LIMITED_API" code to a new unstable/objimpl.h header file
• Include/unstable/ files (Include/unstable/objimpl.h) are no longer prefixed with "unstable_".

Include/unstable/ directory must not be added to the search paths for headers (gcc -I Include/unstable/). unstable/objimpl.h must not be included directly: it fails with a compiler error.

[vstinner]

I propose the following organization:

• Include/*.h should be the "stable API"
• Include/unstable/.h is the "unstable API" (if Py_LIMITED_API is *not defined at all)
• Include/internal/pycore_*.h is the "internal" API

It should become easier to see what is exposed or not to the stable ABI just by looking at Include.*.h.

It should also become easier to spot in a review when a pull request something to the stable ABI, whereas it should be added to the unstable or internal API.

[vstinner]

Just to avoid the risk of name conflict, would it make sense to rename "unstable" to "pyunstable" or something else with "py" inside? I'm not sure if #include "unstable/objimpl.h" first looks the same directory than the header file that does the include?

Note: I tested "make install" and I get a /opt/py38/include/python3.8dm/unstable/ directory which contains a single file (yet): objimpl.h.

[ncoghlan]

I think the rules for C includes are that "path/header.h" looks next to the current file first, whereas <path/header.h> looks only in include directories.

However, given your technique of mostly hiding the new directory name from API consumers, what do you think of calling the new directory "cpython" rather than "unstable"?

The idea there would be that the "unstable ABI" eventually become known as "the CPython C API" (since it exposes a lot of CPython implementation details", while the limited API could become known as "the portable cross-implementation Python C API".

(I know, I know, you were aiming to avoid further bikeshedding on the name, but "cpython" would namespace things nicely even if a compiler does something weird with header file lookups, and helps make it clearer to CPython contributors that we still need to care about public API stability in that directory, we just don't need to care about cross-implementation portability)

[vstinner]

I think the rules for C includes are that "path/header.h" looks next to the current file first, whereas <path/header.h> looks only in include directories.

Oh ok, thanks.

However, given your technique of mostly hiding the new directory name from API consumers, what do you think of calling the new directory "cpython" rather than "unstable"?

I'm not comfortable with "CPython" name. For me, everything the "CPython C API" is the concatenation of all files in Include/ but also in subdirectories. Right now, it's unclear what is the "Python" API ("portable" API, without implemenetation details) vs the "CPython API" (implementation details).

"unstable" comes from the PEP-384: "Defining a Stable ABI". IMHO what is not in the "Stable ABI" is the "Unstable ABI". By extension, APIs excluded by Py_LIMITED_API make the "unstable API".

From my point of view, "CPython API" would be more internal/ + unstable/ APIs.

The idea there would be that the "unstable ABI" eventually become known as "the CPython C API" (since it exposes a lot of CPython implementation details", while the limited API could become known as "the portable cross-implementation Python C API".

Everybody seems to be confused by what is the "Python C API"... I see even more confusion if we have a "CPython C API". Do you see? "CPython" vs "Python", "Python C" vs "CPython"...

IMHO "unstable" is more explicit :-) It means: "don't touch this" :-D

[brettcannon]

The "unstable" name bugs me as it suggests we might change it without notice which isn't true at all. It's more a limited versus broad API. So maybe rename the directory "broad"?

[vstinner]

Brett:

The "unstable" name bugs me as it suggests we might change it without notice which isn't true at all. It's more a limited versus broad API. So maybe rename the directory "broad"?

Brett: Nick proposed "Include\cpython", do you prefer this name?

[vstinner]

Another proposal: Include\impl\ as in "implementation details".

[vstinner]

I created a poll on discuss.python.org for the name of the new subdirectory :-)
https://discuss.python.org/t/poll-what-is-your-favorite-name-for-the-new-include-subdirectory/477

[jkloth]

As a heavy user of the non-limited Python C API, I would like to offer my suggestions for consideration. (I'm not allowed to post in discourse)

First off, to me, 'unstable' comes off quite negative, i.e. risky or erratic. Brett's suggestion of 'broad' is, well, seemingly too "broad" :)

In no particular order, some ideas:

• 'extended' or 'extra'; as opposed to limited (from the macro)

• 'volatile'; synonym for 'unstable' but with the benefit of being a C concept. I like this one as the APIs covered by this include would have access to the "volatile" object details. Things like borrowed references, direct array access or even the structure fields of the objects themselves.

• 'crunchy'; Monty Python reference, "If we took the bones out, it wouldn't be crunchy, would it?"
  https://en.wikipedia.org/wiki/Crunchy_Frog

[vstinner]

Jeremy Kloth:

First off, to me, 'unstable' comes off quite negative, i.e. risky or erratic.

Ok, the 3rd people who dislike my "unstable" name, so it sounds really bad :-)

Jeremy Kloth:

'volatile'; synonym for 'unstable' but with the benefit of being a C concept.

I don't think that it's true that the "#ifndef Py_LIMITED_API" is unstable or volatile. Most of this API didn't change much in the last 10 years. So sorry, "unstable" was really a bad name.

Brett Cannon:

It's more a limited versus broad API. So maybe rename the directory "broad"?

Jeremy Kloth:

• 'extended' or 'extra'; as opposed to limited (from the macro)

The name by itself doesn't explain why an API should be in Include/ or Include/<name>/. What is extra or not?

Jeremy Kloth:

• 'crunchy'; Monty Python reference, "If we took the bones out, it wouldn't be crunchy, would it?" https://en.wikipedia.org/wiki/Crunchy_Frog

Sorry, I dislike humor in an API. An API has to make sense :-(

--

Ok, after I read all proposition, I now prefer "cpython".

Extract of my updated PR which gives the rationale:

Include/.h should be the "portable Python API", whereas
Include/cpython/.h should be the "CPython API": CPython
implementation details.

It now makes sense to me what should go to Include/ and what should go to Include/cpython/.

Obviously, Include/cpython/ is incomplete. It's only the public flavor of the "CPython API". There is also the private CPython internal API in Include/internal/.

[vstinner]

Nick Coghlan, Steve Dower and Paul Moore and me prefer "cpython" name, so let's go with that one!

[vstinner]

New changeset e421106 by Victor Stinner in branch 'master':
bpo-35134: Create Include/cpython/ subdirectory (GH-10624)
e421106

[vstinner]

Number lines containing Py_LIMITED_API in Include/ dir:

13:pyerrors.h
12:abstract.h
11:pylifecycle.h
11:dictobject.h
10:pystate.h
8:longobject.h
7:modsupport.h
7:ceval.h
7:bytesobject.h
6:pythonrun.h
5:warnings.h
5:tupleobject.h
5:methodobject.h
5:complexobject.h
...

[vstinner]

New changeset 6eb9966 by Victor Stinner in branch 'master':
bpo-35134: Create Include/cpython/object.h (GH-10679)
6eb9966

[vstinner]

New changeset 75e4699 by Victor Stinner in branch 'master':
bpo-35134: Create Include/cpython/unicodeobject.h (GH-10680)
75e4699

[vstinner]

New changeset 5a8c240 by Victor Stinner in branch 'master':
bpo-35134: Add Include/cpython/pyerrors.h (GH-10727)
5a8c240

[vstinner]

New changeset 4060283 by Victor Stinner in branch 'master':
bpo-35134: Create Include/cpython/abstract.h (GH-10728)
4060283

[vstinner]

New changeset 17dbd40 by Nicholas Sim in branch 'master':
bpo-35134, Include: Move pytime.h to cpython/pytime.h (GH-23988)
17dbd40

[vstinner]

New changeset 366dc3a by Nicholas Sim in branch 'master':
bpo-35134: Move Include/{pyarena.h,pyctype.h} to Include/cpython/ (GH-24550)
366dc3a

[vstinner]

New changeset 4a6bf27 by Nicholas Sim in branch 'master':
bpo-35134: Move non-limited C API files to Include/cpython/ (GH-24561)
4a6bf27

[vstinner]

New changeset 56f031e by Hai Shi in branch 'master':
bpo-35134: Add include/cpython/compile.h (GH-24922)
56f031e

[vstinner]

Include/README.rst and https://devguide.python.org/c-api/ now define guideliens for header files and the 3 APIs. I consider that this issue is now fixed. Even if there are still non-limited API declared in Include/*.h, changing that can be done in follow-up issues.

[ericsnowcurrently]

Thanks, Victor!

[vstinner]

New changeset 0a883a7 by Victor Stinner in branch 'main':
bpo-35134: Add Include/cpython/floatobject.h (GH-28957)
0a883a7

[vstinner]

I reopen the issue since there is new activity on it :-)

[vstinner]

commit 37b1d60 (upstream/main, main)
Author: Victor Stinner <vstinner@python.org>
Date: Fri Oct 15 01:50:28 2021 +0200

po-35134: Move [Include/funcobject.h](https://github.com/python/cpython/blob/main/Include/funcobject.h) to [Include/cpython/](https://github.com/python/cpython/blob/main/Include/cpython) (GH-28958)

Remove redundant "#ifndef Py_LIMITED_API" in funcobject.h.

[vstinner]

New changeset 77b24ba by Victor Stinner in branch 'main':
bpo-35134: Move Include/cellobject.h to Include/cpython/ (GH-28964)
77b24ba

[vstinner]

New changeset 8e5de40 by Victor Stinner in branch 'main':
bpo-35134: Move classobject.h to Include/cpython/ (GH-28968)
8e5de40

[vstinner]

New changeset aad88d3 by Victor Stinner in branch 'main':
bpo-35134: Split warnings.h and weakrefobject.h (GH-29042)
aad88d3

[vstinner]

New changeset 5f09bb0 by Victor Stinner in branch 'main':
bpo-35134: Add Include/cpython/longobject.h (GH-29044)
5f09bb0

[vstinner]

New changeset d4a85f1 by Victor Stinner in branch 'main':
bpo-35134: Add Include/cpython/descrobject.h (GH-30923)
d4a85f1

[vstinner]

New changeset ca219f6 by Victor Stinner in branch 'main':
bpo-35134: Add Include/cpython/complexobject.h header (GH-32383)
ca219f6

[vstinner]

New changeset 5c4d1f6 by Victor Stinner in branch 'main':
bpo-35134: Add Include/cpython/setobject.h header (GH-32384)
5c4d1f6

[vstinner]

New changeset 85addfb by Victor Stinner in branch 'main':
bpo-35134: Remove the Include/code.h header file (GH-32385)
85addfb