PEP-662: Move to a wheel based approach

Makes it easier for frontends to reuse existing implementations
of projects metadata. Also keeps the existing practice of the
build backend and frontend communicating via an artifact object.

Signed-off-by: Bernát Gábor <bgabor8@bloomberg.net>

.github/CODEOWNERS

@@ -517,6 +517,7 @@ pep-0659.rst  @markshannon
 pep-0660.rst  @pfmoore
 pep-0661.rst  @taleinat
 pep-0662.rst  @brettcannon
+pep-0662-editables.json  @brettcannon
 # ...
 # pep-0666.txt
 # ...

pep-0662-editables.json

@@ -0,0 +1,38 @@
+{
+  "$schema": "http://json-schema.org/draft-07/schema",
+  "$id": "http://pypa.io/editables.json",
+  "type": "object",
+  "title": "Virtual wheel editable schema.",
+  "required": ["version", "scheme"],
+  "properties": {
+    "version": {
+      "$id": "#/properties/version",
+      "type": "integer",
+      "minimum": 1,
+      "maximum": 1,
+      "title": "The version of the schema."
+    },
+    "scheme": {
+      "$id": "#/properties/scheme",
+      "type": "object",
+      "title": "Files to expose.",
+      "required": ["purelib", "platlib", "data", "headers", "scripts"],
+      "properties": {
+        "purelib": { "$ref": "#/$defs/mapping" },
+        "platlib": { "$ref": "#/$defs/mapping" },
+        "data": { "$ref": "#/$defs/mapping" },
+        "headers": { "$ref": "#/$defs/mapping" },
+        "scripts": { "$ref": "#/$defs/mapping" }
+      },
+      "additionalProperties": true
+    }
+  },
+  "additionalProperties": true,
+  "$defs": {
+    "mapping": {
+      "type": "object",
+      "description": "A mapping of source to target paths. The source is absolute path, the destination is relative path.",
+      "additionalProperties": true
+    }
+  }
+}

pep-0662.rst

@@ -26,8 +26,7 @@ This mode is usually called "development mode" or "editable installs".
 Currently, there is no standardized way to accomplish this, as it was explicitly
 left out of :pep:`517` due to the complexity of the actual observed behaviors.
 
-At the moment, users can achieve this in a few ways, neither of them being a
-standard:
+At the moment, users to get this behaviour perform one of the following:
 
 -  For just Python code by adding the relevant source directories to
    ``sys.path`` (configurable from the command line interface via the
@@ -58,9 +57,10 @@ installation is:
 The author of this PEP believes there's no one size fits all solution here,
 each method of achieving editable effect has its pros and cons. Therefore
 this PEP rejects option three as it's unlikely for the community to agree on a
-single solution. Therefore, question remains as to whether the frontend or the
+single solution. Furthermore, question remains as to whether the frontend or the
 build backend should own this responsibility. :pep:`660` proposes the build
-backend to own this, while the current PEP proposes the frontend.
+backend to own this, while the current PEP proposes primarily the frontend, 
+but still allows the backend to take take control if it wants to do so.
 
 Rationale
 =========
@@ -83,18 +83,21 @@ their users. In this proposal, the backend's role is to prepare the project for
 an editable installation, and then provide enough information to the frontend
 so that the frontend can manifest and enforce the editable installation.
 
-The information the backend provides to the frontend is:
+The information the backend provides to the frontend is a wheel that follows
+the existing specification within :pep:`427`. The wheel metadata about the
+archive itself (``{distribution}-{version}.dist-info/WHEEL``) must also contain
+the key ``Editable`` with value of ``true``.
 
-- the project metadata (as defined by :pep:`427` under ``.dist-info``),
-- the files to expose (formulated as a mapping of absolute source tree
-  paths to relative target interpreter destination paths).
+However, instead of providing the project files within the wheel, it must
+provide an ``editable.json`` file (at the root level of the wheel) that defines
+the files to be exposed by the frontend. The content of this file is formulated
+as a mapping of absolute source tree paths to relative target interpreter
+destination paths within a scheme mapping.
 
-We refer to this set of information as the virtual wheel. This virtual wheel
-should contain all information a wheel contains, however it's not zipped and
-its installation will not be done by copying the files. The frontend's role is
-to take the virtual wheel and install the project in editable mode. The way it
-achieves this is entirely up to the frontend and is considered implementation
-detail.
+A wheel that satisfies teh previous two paragraphs is a virtual wheel. The
+frontend's role is to take the virtual wheel and install the project in
+editable mode. The way it achieves this is entirely up to the frontend and is
+considered implementation detail.
 
 The editable installation mode implies that the source code of the project
 being installed is available in a local directory. Once the project is
@@ -116,9 +119,10 @@ installations, this may not always be possible and may be in tension with other
 user expectations. Depending on how a frontend implements the editable mode,
 some differences may be visible, such as the presence of additional files
 (compared to a typical installation), either in the source tree or the
-interpreter's installation path. Frontends should seek to minimize differences
-between the behavior of editable and standard installations and document known
-differences.
+interpreter's installation path.
+
+Frontends should seek to minimize differences between the behavior of editable
+and standard installations and document known differences.
 
 For reference, a non-editable installation works as follows:
 
@@ -164,39 +168,52 @@ If not defined, the default implementation is equivalent to returning ``[]``.
 
 .. code::
 
-   def build_editable(config_settings=None):
+   def build_editable(self, wheel_directory, config_settings=None,
+                       metadata_directory=None):
        ...
 
-The function returns an object of type ``EditableInfo`` as defined below:
-
-.. code::
+Must build a .whl file, and place it in the specified ``wheel_directory``. It
+must return the basename (not the full path) of the ``.whl`` file it creates,
+as a unicode string. The wheel file must be of type virtual wheel as defined
+under the terminology section.
 
-   from typing import Mapping, TypedDict
+If the build frontend has previously called ``prepare_metadata_for_build_wheel``
+and depends on the wheel resulting from this call to have metadata
+matching this earlier call, then it should provide the path to the created
+``.dist-info`` directory as the ``metadata_directory`` argument. If this
+argument is provided, then ``build_editable`` MUST produce a wheel with identical
+metadata. The directory passed in by the build frontend MUST be
+identical to the directory created by ``prepare_metadata_for_build_wheel``,
+including any unrecognized files it created.
 
-   class SchemePaths(TypedDict, total=False):
-       """
-       Files and folders that should be mapped:
-       - key is the absolute source path
-       - value is the relative path within the target interpreters prefix
-       """
+Backends which do not provide the ``prepare_metadata_for_build_wheel`` hook may
+either silently ignore the ``metadata_directory`` parameter to ``build_editable``,
+or else raise an exception when it is set to anything other than ``None``.
 
-       purelib: Mapping[str, str]
-       platlib: Mapping[str, str]
-       headers: Mapping[str, str]
-       scripts: Mapping[str, str]
-       data: Mapping[str, str]
+The source directory may be read-only, in such cases the backend may raise an
+error that the frontend can display to the user. The backend may store intermediate
+artifacts in cache locations or temporary directories. The presence or absence of
+any caches should not make a material difference to the final result of the build.
 
+The content of the ``editables.json`` MUST pass against the following JSON schema:
 
-   class EditableInfo(TypedDict, total=True):
-       version: int
-       """protocol version of the editable metadata, this PEP defines version 1"""
+.. include:: pep-0662-editables.json
+   :code:
 
-       metadata_for_build_editable: str
-       """distribution information of the package as defined by PEP-491"""
+For example:
 
-       paths: SchemePaths
-       """files to expose into the target interpreter"""
+.. code::
 
+   {
+      "version": 1,
+      "scheme": {
+         "purelib": {"/src/tree/a.py": "tree/a.py"},
+         "platlib": {},
+         "data": {"/src/tree/py.typed": "tree/py.typed"},
+         "headers": {},
+         "scripts": {}
+      }
+   }
 
 The scheme paths map from project source absolute paths to target directory
 relative paths. We allow backends to change the project layout from the project
@@ -210,42 +227,26 @@ Build frontend requirements
 ---------------------------
 
 The build frontend is responsible for setting up the environment for the build
-backend to generate the necessary information for an editable build. It's also
-responsible for communicating with the backend and receiving the
-``EditableInfo`` object. All recommendations from :pep:`517` for the build wheel
-hook applies here too.
+backend to generate the virtual wheel. All recommendations from :pep:`517` for
+the build wheel hook applies here too.
 
 Frontend requirements
 ---------------------
 
-The frontend is responsible for ensuring the ``.dist-info`` folder is available
-at runtime within the target interpreter for the ``importlib.metadata`` and
-``importlib.resources`` modules.
-
-The frontend must ensure that all installation requirements specified in the
-distribution information files are installed as part of the editable
-installation into the target interpreter. Additionally, the user might also
-select additional ``extras`` groups that also should be installed as part of the
-editable installation.
-
-The frontend also must generate entrypoints, which may be for the console or the
-GUI. Those entrypoints are defined by the distribution information files, which
-are generated during the editable installation process.
-
-The frontend is responsible for generating the ``RECORD`` file based on the
-object the build backend returns and their chosen editable implementation. For
-this reason, the uninstallation of editables should not require any special
-treatment.
+The frontend must install the virtual wheel exactly as defined within
+:pep:`427`. Furthermore is responsible for also installing the files defined
+witin the ``editables.json`` file. The manner in which it does is left up to
+the frontend, and is encouraged for the frontend to communicate with the user
+exactly the method choosen, and what limitations that solution will have.
 
 The frontend must create a ``direct_url.json`` file in the ``.dist-info``
 directory of the installed distribution, in compliance with PEP 610. The ``url``
 value must be a ``file://`` URL pointing to the project directory (i.e., the
 directory containing ``pyproject.toml``), and the ``dir_info`` value must be
 ``{'editable': true}``.
 
-The frontend must not rely on the ``prepare_metadata_for_build_wheel`` hook when
-installing in editable mode. It must instead invoke ``build_editable`` and use
-the ``.dist-info`` folder returned by that.
+The frontend can rely on the ``prepare_metadata_for_build_wheel`` hook when
+installing in editable mode.
 
 If the frontend concludes it cannot achieve an editable installation with the
 information provided by the build backend it should fail and raise an error to
@@ -255,25 +256,27 @@ The frontend might implement one or more editable installation mechanisms and
 can leave it up to the user the choose one that its optimal to the use case
 of the user. For example, pip could add an editable mode flag, and allow the
 user to choose between ``pth`` files or symlinks (
-``pip install -e . --editable=pth`` vs ``pip install -e . --editable=symlink``).
+``pip install -e . --editable-mode=pth`` vs
+``pip install -e . --editable-mode=symlink``).
 
 Example editable implementations
 --------------------------------
 
 To show how this PEP might be used, we'll now present a few case studies. Note
-the offered solutions are purely for illustrating purpose.
+the offered solutions are purely for illustration purpose and are not normative
+for the frontend/backend.
 
 Add the source tree as is to the interpreter
 ''''''''''''''''''''''''''''''''''''''''''''
 
 This is one of the simplest implementations, it will add the source tree as is
-into the interpreters scheme paths, the virtual wheel might look like:
+into the interpreters scheme paths, the ``editable.json`` within the virtual wheel
+might look like:
 
 .. code::
 
    {
-     "metadata_for_build_editable": "<dir to dist-info>",
-     {"scheme": "purelib": {"<project dir>": "<project dir>"}}
+     {"version": 1, "scheme": {"purelib": {"<project dir>": "<project dir>"}}}
    }
 
 The frontend then could either:
@@ -305,9 +308,8 @@ interpreter startup by adding a ``pth`` file.
 .. code::
 
    {
-     "metadata_for_build_editable": "<dir to dist-info>",
-     {
-       "scheme": {
+      "version": 1,
+      "scheme": {
          "purelib": {
            "<project dir>/.editable/_register_importer.pth": "<project dir>/_register_importer.pth".
            "<project dir>/.editable/_editable_importer.py": "<project dir>/_editable_importer.py"