-
-
Notifications
You must be signed in to change notification settings - Fork 385
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[WIP] OdbBackend: Python subclassing #948
Conversation
5f7476e
to
f4f4cd7
Compare
All of the basics are here, would like some feedback on the approach. If this looks good, the remainder of the work is basically to repeat 52d7669 for each member of the git_odb_backend struct from libgit2. I have a small script I've been using to "test" this: from pygit2 import Odb, OdbBackend, OdbBackendPack
pack = OdbBackendPack(".git/objects")
class FooBackend(OdbBackend):
def __init__(self):
super().__init__(self)
def read(self, oid):
print("read called")
return pack.read(oid)
def read_prefix(self, oid):
print("read_prefix called")
# TODO: do this properly
typ, data = pack.read(oid)
return oid, typ, data
def read_header(self, oid):
print("read_header called")
# TODO: do this properly
typ, data = pack.read(oid)
return typ, len(data)
fbe = FooBackend()
odb = Odb()
odb.add_backend(fbe, 1)
print(odb.read("209afad181e3f5f7b9f00088f2e9b6c63329a878")) (This object ID can be found in pygit2). Proper unit tests will come prior to merge. |
It's a good start, but there're memory issues. Maybe something like this? struct pygit2_odb_backend
{
git_odb_backend backend;
void *payload;
};
typedef struct {
PyObject_HEAD
pygit2_odb_backend *odb_backend;
PyObject *read_callable,
*read_prefix_callable;
} OdbBackendUD; (UD for User Defined) So the C callbacks need a reference to the Python functions. Maybe libgit2's Note that we must |
I'm not sure I understand the advantage (or difference) to the approach you've described here. Would the current approach work if I made sure to incref the callables? |
52d7669
to
6f38154
Compare
Most important is to handle memory correctly. Otherwise the difference is small, it's about readability and maintenance:
So I'm not asking to change the approach, only about improving readability. A comment describing how Finally, the way I've written We need a pointer back to Python to find our callbacks written in Python, this should be Or maybe I've misunderstood something, or there's another way. Do other bindings support |
👋 The custom backend types don’t add payloads since they’re extendable by the implementer. We’d encourage you to do what it looks like you recommended - with the I’m not sure which bindings implement custom backends but the .NET bindings do. Others may? Not something that I’ve surveyed. |
6f38154
to
ffe93aa
Compare
Regarding the use of an opaque pointer (e.g. Latest patchset includes the following changes:
I think that this settles the design constraints, if we like it then I think the approach can be generalized to the rest of the |
Cheers! I should have this finished up tomorrow. |
ffe93aa
to
72605b7
Compare
Actually, something else came up of a higher priority, I'm going to return to this in a few weeks at the latest. |
Apologies for the wait - I'll be picking this up next week. |
The basic approach here is to have backends created from C (loose, pack) circumvent the OdbBackend_init function. However, backends created from Python will not. We then use this backend to pull out the functions implemented in Python and rig them up through shim functions to the libgit2 git_odb_backend.
72605b7
to
ded2b4e
Compare
Okay, this should be ready for review. I've trimmed down the scope to just supporting read-only backends, added tests, and cleaned things up. |
Next set of patches will add similar riggings for the refdb, and expanding both for read/write access separately. |
Looks good, but tests are failing in AppVeyor. |
Looks like some undecipherable Windows nonsense. I don't use or have any interest in supporting Windows, can someone who does take a look? |
The error happens in the call to /* XXX: This assumes the default libgit2 allocator is in use and will
* probably segfault and/or destroy the universe otherwise */
free(data); See https://ci.appveyor.com/project/jdavid/pygit2/builds/29550154 |
That assumption is not guaranteed for a number of reasons: allocators come in many flavors on Windows (debug builds get instrumented malloc/free, release builds do not, and these allocators use different pools and are not interoperable; this is true across library/consumer boundaries) and libgit2 has a pluggable allocator.
libgit2/libgit2#5005 should have introduced a function to help with this problem.
… On Dec 15, 2019, at 23:10, J. David Ibáñez ***@***.***> wrote:
The error happens in the call to free:
/* XXX: This assumes the default libgit2 allocator is in use and will
* probably segfault and/or destroy the universe otherwise */
free(data);
See https://ci.appveyor.com/project/jdavid/pygit2/builds/29550154
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
Updated to use the new libgit2 functions, will rebase & revisit when the next libgit2 release lands. |
Thanks! Happy holidays :) |
you too! |
1.1.0 (2020-03-01) ------------------------- - Upgrade to libgit2 0.99 `#959 <https://github.com/libgit2/pygit2/pull/959>`_ - Continued work on custom odb backends `#948 <https://github.com/libgit2/pygit2/pull/948>`_ - New ``Diff.patchid`` getter `#960 <https://github.com/libgit2/pygit2/pull/960>`_ `#877 <https://github.com/libgit2/pygit2/issues/877>`_ - New ``settings.disable_pack_keep_file_checks(...)`` `#908 <https://github.com/libgit2/pygit2/pull/908>`_ - New ``GIT_DIFF_`` and ``GIT_DELTA_`` constants `#738 <https://github.com/libgit2/pygit2/issues/738>`_ - Fix crash in iteration of config entries `#970 <https://github.com/libgit2/pygit2/issues/970>`_ - Travis: fix printing features when building Linux wheels `#977 <https://github.com/libgit2/pygit2/pull/977>`_ - Move ``_pygit2`` to ``pygit2._pygit2`` `#978 <https://github.com/libgit2/pygit2/pull/978>`_ Requirements changes: - Now libgit2 0.99 is required - New requirement: cached-property Breaking changes: - In the rare case you're directly importing the low level ``_pygit2``, the import has changed:: # Before import _pygit2 # Now from pygit2 import _pygit2
The basic approach here is to have backends created from C (loose, pack) circumvent the OdbBackend_init function. However, backends created from Python will not. We then use this backend to pull out the functions implemented in Python and rig them up through shim functions to the libgit2 git_odb_backend.
TODO:
Separate patch?