Deoptimise pathlib ABCs #113528

barneygale · 2023-12-28T00:21:56Z

Feature or enhancement

pathlib._abc.PurePathBase and PathBase have a few slots and methods that facilitate fast path object generation:

_drv, _root, _tail_cached: roughly, the result of os.path.splitroot() on the path
_str: the normalized path string
_make_child_relpath(): used when walking directories
_from_parsed_parts(): used in parent, parents, with_name(), relative_to()

These carefully-tuned features are useful in pathlib, where speed is important, but much less so in pathlib._abc, where the clarity of the interfaces and interactions should win out.

It should be possible to move them to PurePath, leaving PurePathBase with only _raw_paths and _resolving slots.

This should not affect the behaviour or performance of the public classes in pathlib proper.

Linked PRs

The text was updated successfully, but these errors were encountered:

Replace use of `_from_parsed_parts()` with `with_segments()` in `PurePathBase.relative_to()`, and move the assignment of `_drv`, `_root`, `_tail_cached` and `_str` slots into `PurePath.relative_to()`. Also shuffle a deprecation warning into `PurePath.relative_to()` to ensure it's raised at the right stack level, and do the same in `is_relative_to()` for code consistency.

Replace use of `_from_parsed_parts()` with `with_segments()`, and move assignments to `_drv`, `_root`, _tail_cached` and `_str` slots into `PurePath`.

Replace usage of `_from_parsed_parts()` with `with_segments()` in `with_name()`, and take a similar approach in `name` for consistency's sake.

…h()` Call straight through to `joinpath()` in `PathBase._make_child_relpath()`. Move optimised/caching code to `pathlib.Path._make_child_relpath()`

Run expensive tests for walking and globbing from `test_pathlib` but not `test_pathlib_abc`. The ABCs are not as tightly optimised as the classes in top-level `pathlib`, and so these tests are taking rather a long time on some buildbots. Coverage of the main `pathlib` classes should suffice.

Move `_str` cache slot from `PurePathBase` to `PurePath`. As a result, pathlib ABCs no longer cache their string representations.

Split test cases for invalid names into dedicated test methods. This will make it easier to refactor tests for invalid name handling in ABCs later. No change of coverage, just a change of test suite organisation.

The `DummyPurePath` and `DummyPath` test classes are simple subclasses of `PurePathBase` and `Pathbase`. This commit adds `__repr__()` methods to the dummy classes, which makes debugging test failures less painful.

… (again) (python#113882) Restore full battle-tested implementations of `PurePath.[is_]relative_to()`. These were recently split up in 3375dfe and a15a773. In `PurePathBase`, add entirely new implementations based on `_stack`, which itself calls `pathmod.split()` repeatedly to disassemble a path. These new implementations preserve features like trailing slashes where possible, while still observing that a `..` segment cannot be added to traverse an empty or `.` segment in *walk_up* mode. They do not rely on `parents` nor `__eq__()`, nor do they spin up temporary path objects. Unfortunately calling `pathmod.relpath()` isn't an option, as it calls `abspath()` and in turn `os.getcwd()`, which is impure.

Apply pathlib's normalization and performance tuning in `pathlib.PurePath`, but not `pathlib._abc.PurePathBase`. With this change, the pathlib ABCs do not normalize away alternate path separators, empty segments, or dot segments. A single string given to the initialiser will round-trip by default, i.e. `str(PurePathBase(my_string)) == my_string`. Implementors can set their own path domain-specific normalization scheme by overriding `__str__()` Eliminating path normalization makes maintaining and caching the path's parts and string representation both optional and not very useful, so this commit moves the `_drv`, `_root`, `_tail_cached` and `_str` slots from `PurePathBase` to `PurePath`. Only `_raw_paths` and `_resolving` slots remain in `PurePathBase`. This frees the ABCs from the burden of some of pathlib's hardest-to-understand code.

…on#113534) Run expensive tests for walking and globbing from `test_pathlib` but not `test_pathlib_abc`. The ABCs are not as tightly optimised as the classes in top-level `pathlib`, and so these tests are taking rather a long time on some buildbots. Coverage of the main `pathlib` classes should suffice.

…ython#113777) The `DummyPurePath` and `DummyPath` test classes are simple subclasses of `PurePathBase` and `PathBase`. This commit adds `__repr__()` methods to the dummy classes, which makes debugging test failures less painful.

…n#113776) Split test cases for invalid names into dedicated test methods. This will make it easier to refactor tests for invalid name handling in ABCs later. No change of coverage, just a change of test suite organisation.

…link loop handling (python#113763) Slightly improve `pathlib.Path.glob()` tests for symlink loop handling When filtering results, ignore paths with more than one `linkD/` segment, rather than all paths below the first `linkD/` segment. This allows us to test that other paths under `linkD/` are correctly returned.

…113531) Replace usage of `_from_parsed_parts()` with `with_segments()` in `with_name()`, and take a similar approach in `name` for consistency's sake.

…n#113530) Replace use of `_from_parsed_parts()` with `with_segments()`, and move assignments to `_drv`, `_root`, _tail_cached` and `_str` slots into `PurePath`.

…python#113529) Replace use of `_from_parsed_parts()` with `with_segments()` in `PurePathBase.relative_to()`, and move the assignment of `_drv`, `_root` and `_tail_cached` slots into `PurePath.relative_to()`.

`PurePathBase` does not define `__eq__()`, and so we have no business checking path equality in `test_eq_common` and `test_equivalences`. The tests only pass at the moment because we define the test class's `__eq__()` for use elsewhere. Also move `test_parse_path_common` into the main pathlib test suite. It exercises a private `_parse_path()` method that will be moved to `PurePath` soon. Lastly move a couple more tests concerned with optimisations and path normalisation.

- Add `__slots__` to dummy path classes. - Return namedtuple rather than `os.stat_result` from `DummyPath.stat()`. - Reduce maximum symlink count in `DummyPathWithSymlinks.resolve()`.

…h()` (python#113532) Call straight through to `joinpath()` in `PathBase._make_child_relpath()`. Move optimised/caching code to `pathlib.Path._make_child_relpath()`

…#113782) Replace use of `_from_parsed_parts()` with `with_segments()` in `resolve()`. No effect on `Path.resolve()`, which uses `os.path.realpath()`.

…#113883) Implement `parts` using `_stack`, which itself calls `pathmod.split()` repeatedly. This avoids use of `_tail`, which will be moved to `PurePath` shortly.