add initial support for Git protocol v2 #1244

stspdotname · 2023-12-12T15:47:36Z

Implement Git protocol version negotiation and use Git protocol v2 for fetches if supported. For now, the observable behaviour of Dulwich is equivalent regardless of protocol version, except that two new features may be used if the server supports Git protocol v2.

The first feature is a reference prefix list which may be used to filter refs server-side. This can be used to reduce the size of the initial reference listing announced by the server. Reducing the size of this announcement was a major design goal for protocol v2 to avoid sending of very large announcements when a repository contains a lot of references. This feature is intended as an optimization which servers are free to ignore depending on available server-side resources. Therefore, users of Dulwich should still be prepared to filter redundant refs manually (this limitation also applies to Git itself).
A new --refspec porcelain option is provided in order to test this feature on the command line.

The second feature is an object filter specification, which corresponds to the --filter option of 'git clone'. This can be used to omit objects while cloning repositories. For instance, the following command will clone a given repsitory without fetching any blob objects:

dulwich clone --filter blob:none --bare REPO_URL

(In this example the --bare option is used because creation of a work tree would fail without any blobs present.)

The test suite now enables protocol v2 and keeps passing for me.

stspdotname · 2023-12-12T16:03:36Z

Not sure what the "unexpected success" failures on Windows are about. Are there more detailed test logs available? I don't have a Windows machine to run tests on myself.

dulwich/porcelain.py

jelmer

thanks! LGTM overall; I left some nits inline, but nothing major.

It would be good to have at least some basic tests; maybe there's a way of forcing the git server to force a particular protocol version and rely on the compat tests?

dulwich/porcelain.py

dulwich/cli.py

jelmer · 2023-12-13T23:41:24Z

dulwich/cli.py

-        porcelain.pull(".", from_location)
+        parser = argparse.ArgumentParser()
+        parser.add_argument('--from_location', type=str)
+        parser.add_argument('--refspec', type=str, nargs='*')


Suggested change

parser.add_argument('--refspec', type=str, nargs='*')

parser.add_argument('refspec', type=str, nargs='*')

for consistency with C Git

dulwich/client.py

jelmer · 2023-12-14T00:23:24Z

Not sure what the "unexpected success" failures on Windows are about. Are there more detailed test logs available? I don't have a Windows machine to run tests on myself.

These are spurious - I've been meaning to look into them and possibly mark them as flapping.

stspdotname · 2023-12-14T19:11:06Z

It would be good to have at least some basic tests; maybe there's a way of forcing the git server to force a particular protocol version and rely on the compat tests?

Reading Git's server code suggests that the server always uses the highest mutually supported version . See determine_protocol_version_server() in Git's protocol.c.

It doesn't make much sense to run dulwich compat tests against ancient versions of Git so in practice the protocol version used during dulwich tests depends entirely on what the client is requesting.

Therefore we should add a configuration knob to dulwich that controls the protocol version the client will request. Git clients already support such an option: protocol.version in git-config.

I will try to add such an option to dulwich as a command-line parameter, and/or environment variable, and/or via the existing git config option -- any preference?

Once this parameter exists, ideally the entire compat test suite would always run twice, once with the old protocol (v0/v1) and again with protocol v2. Any tests which only make sense in one case could be skipped in the other.

stspdotname · 2023-12-19T11:32:10Z

I am not quite sure yet how to adjust the test suite to run compat tests across both protocol versions. I will look into this next. Meanwhile, feel free to merge this PR if you are happy with the changes so far. I can always open a new PR for the test suite changes later.

jelmer · 2024-06-05T21:22:11Z

tests/__init__.py

@@ -213,11 +214,21 @@ def nocompat_test_suite():
    return result


+@patch("dulwich.protocol.DEFAULT_GIT_PROTOCOL_VERSION_FETCH", new=0)
+def compat_test_suite_git_protocol_v0():


Is this running all compat tests twice, not just the client ones?

This should really just be specific to the client code.

jelmer · 2024-06-06T13:10:36Z

dulwich/protocol.py

@@ -30,6 +30,10 @@

 TCP_GIT_PORT = 9418

+GIT_PROTOCOL_VERSIONS = [0, 1, 2]


Is there actually a version 1? Can you add a comment about these versions.

Yes, I am adding comments in the next version.

They will appear in an earlier commit which adds this list.
I have rewritten history a bit, also to get rid of the 'ruff format' commit in the middle.

jelmer · 2024-06-06T13:12:59Z

tests/compat/test_client.py

    ):
        cmd, path = command.split(" ")
        cmd = cmd.split("-", 1)
        path = path.replace("'", "")
+        env = copy.deepcopy(os.environ)
+        if protocol_version is None or protocol_version == 2:


Can we do the inverse here, i.e.

If protocol version is not set, set it to 2.

If protocol version != 0 then set this header.

That prepares us a bit better to future versions.

Yes, we can do that. I don't anticipate a new protocol version in the foreseeable future, but it doesn't hurt to handle things that way.

tests/compat/test_client.py

tests/__init__.py

jelmer

Please take a look at the test failures.

Implement Git protocol version negotiation and use Git protocol v2 for fetches if supported. For now, the observable behaviour of Dulwich is equivalent regardless of protocol version, except that two new features may be used if the server supports Git protocol v2. The first feature is a reference prefix list which may be used to filter refs server-side. This can be used to reduce the size of the initial reference listing announced by the server. Reducing the size of this announcement was a major design goal for protocol v2 to avoid sending of very large announcements when a repository contains a lot of references. This feature is intended as an optimization which servers are free to ignore depending on available server-side resources. Therefore, users of Dulwich should still be prepared to filter redundant refs manually (this limitation also applies to Git itself). A new --refspec porcelain option is provided in order to test this feature on the command line. The second feature is an object filter specification, which corresponds to the --filter option of 'git clone'. This can be used to omit objects while cloning repositories. For instance, the following command will clone a given repsitory without fetching any blob objects: dulwich clone --filter blob:none --bare REPO_URL (In this example the --bare option is used because creation of a work tree would fail without any blobs present.) The test suite now enables protocol v2 and keeps passing for me.

…elmer

suggested by Jelmer

Git prints a warning and proceeds with an unfiltered clone/fetch operation. Make dulwich behave the same way, for now.

…e-proof Based on a suggestion by Jelmer

Git protocol v2 implies the side-band-64k capability so this test is meaningless in that case.

stspdotname requested a review from jelmer as a code owner December 12, 2023 15:47

jelmer reviewed Dec 13, 2023

View reviewed changes

dulwich/porcelain.py Outdated Show resolved Hide resolved

jelmer requested changes Dec 14, 2023

View reviewed changes

stspdotname force-pushed the protov2 branch 2 times, most recently from a615d01 to bddc28e Compare December 19, 2023 11:15

jelmer force-pushed the master branch from 7271a76 to 98f303b Compare January 27, 2024 23:10

stspdotname force-pushed the protov2 branch 4 times, most recently from b572358 to dc11b97 Compare June 5, 2024 09:47

jelmer requested changes Jun 5, 2024

View reviewed changes

jelmer requested changes Jun 6, 2024

View reviewed changes

stspdotname force-pushed the protov2 branch 4 times, most recently from 0dfa109 to 83dde9f Compare June 11, 2024 14:48

stspdotname requested a review from jelmer June 11, 2024 14:48

jelmer requested changes Jun 20, 2024

View reviewed changes

stspdotname added 8 commits June 25, 2024 08:54

keep dulwich pull command line similar to the Git CLI; suggested by J…

e543e98

…elmer

use None to represent unspecified optional parameters; suggested by J…

26344f8

…elmer

make the Git protocol version configurable from the command line

325de7d

make Git protocol v2 work over SSH connections

2400933

use dict() instead of deepcopy()

2647c58

suggested by Jelmer

emulate Git's behaviour when a server does not support object filtering

65220bb

Git prints a warning and proceeds with an unfiltered clone/fetch operation. Make dulwich behave the same way, for now.

set the default git protocol version via global constants

343ed68

stspdotname added 4 commits June 25, 2024 08:54

don't hard-code protocol v2 and tweak version comparisons to be futur…

5c1a5ec

…e-proof Based on a suggestion by Jelmer

skip test_fetch_pack_no_side_band_64k if git protocol v2 is used

5b2f74f

Git protocol v2 implies the side-band-64k capability so this test is meaningless in that case.

run client compat tests with git protocol version 0, too

0e18ede

blind attempt at fixing the unexpected-success "failures" on windows CI

f270f0c

stspdotname force-pushed the protov2 branch from 83dde9f to f270f0c Compare June 25, 2024 08:57

jelmer approved these changes Jun 28, 2024

View reviewed changes

jelmer merged commit 101a80c into jelmer:master Jun 28, 2024
25 checks passed

dimbleby mentioned this pull request Nov 17, 2024

vcs: use peeled ref when retrieving tag revision python-poetry/poetry#9849

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add initial support for Git protocol v2 #1244

add initial support for Git protocol v2 #1244

stspdotname commented Dec 12, 2023

stspdotname commented Dec 12, 2023 •

edited

Loading

jelmer left a comment

jelmer Dec 13, 2023

jelmer commented Dec 14, 2023

stspdotname commented Dec 14, 2023

stspdotname commented Dec 19, 2023

jelmer Jun 5, 2024

jelmer Jun 6, 2024

stspdotname Jun 11, 2024

jelmer Jun 6, 2024

stspdotname Jun 11, 2024

jelmer left a comment

	parser.add_argument('--refspec', type=str, nargs='*')
	parser.add_argument('refspec', type=str, nargs='*')

		@@ -30,6 +30,10 @@

		TCP_GIT_PORT = 9418

		GIT_PROTOCOL_VERSIONS = [0, 1, 2]

add initial support for Git protocol v2 #1244

add initial support for Git protocol v2 #1244

Conversation

stspdotname commented Dec 12, 2023

stspdotname commented Dec 12, 2023 • edited Loading

jelmer left a comment

Choose a reason for hiding this comment

jelmer Dec 13, 2023

Choose a reason for hiding this comment

jelmer commented Dec 14, 2023

stspdotname commented Dec 14, 2023

stspdotname commented Dec 19, 2023

jelmer Jun 5, 2024

Choose a reason for hiding this comment

jelmer Jun 6, 2024

Choose a reason for hiding this comment

stspdotname Jun 11, 2024

Choose a reason for hiding this comment

jelmer Jun 6, 2024

Choose a reason for hiding this comment

stspdotname Jun 11, 2024

Choose a reason for hiding this comment

jelmer left a comment

Choose a reason for hiding this comment

stspdotname commented Dec 12, 2023 •

edited

Loading