GTK+ 3 (GObject introspection)

[spillz]

It appears the jedi can't handle the GObject introspection module (using jedi 0.8.0-final0 on Ubuntu 14.04):

s3 = jedi.Script('from gi.repository import Gtk\nGtk.')
s3.completions()

s2 = jedi.Script('import gtk\ngtk.')
s2.completions()

s2 completions show as expected, s3 is an empty list. Note that if I manually import Gtk3 and type dir(Gtk) I get a complete list of module symbols.

Are there any workarounds for this? (Don't get me started on what a disaster GI is for long suffering GTK+ app devs.)

[ghost]

Do you know fakegir ?
https://github.com/20115681/fakegir

[spillz]

No, but that looks very good and is code that even I can understand. Docstrings?

@davidhalter should persistent pre-caching (which could be generated from a separately spawned process and kept in sync with timestamps) be used for more modules? Would solve many of the problems with builtins and also issues with being able to manage memory. Although maybe that's what you are already doing?

[davidhalter]

@spillz I'm not sure what you are talking about. There's some caching for Python modules. But none for builtins (However, they are usually really fast).

[spillz]

Maybe i am using wrong terminology. By builtin I mean c libs and other
modules that can't be parsed and have to be imported. Are they cached? (And
by cached I mean to disk/storage not just memory)
On Jan 22, 2015 9:42 AM, "Dave Halter" notifications@github.com wrote:

@spillz https://github.com/spillz I'm not sure what you are talking
about. There's some caching for Python modules. But none for builtins.

—
Reply to this email directly or view it on GitHub
#531 (comment).

[davidhalter]

No they are not cached, because importing them is very fast. Caching them would probably be slower.

[spillz]

Probably not much slower, but more important is the possibility of
unloading cached stuff from memory. (And the slight improvement in safety
from running the import in a separate process.)
On Jan 22, 2015 10:01 AM, "Dave Halter" notifications@github.com wrote:

No they are not cached, because importing them is very fast. Caching them
would probably be slower.

—
Reply to this email directly or view it on GitHub
#531 (comment).

[davidhalter]

Possible. But not a high priority for me. Such a solution would be nice, but I just know that it's a lot of work and the current solution works very well.

[elad661]

Hi!

I'm working on a patch for gnome-builder (the new and awesome GNOME IDE) to provide auto-completion support for GObject Introspection libraries. For Python code, Builder already uses Jedi and it works great. Since it's a GNOME IDE, we really want it to be able to be able to have auto-completion for GObject Introspection.

I can read and parse the GIR files (they are basically XML) or the typelib files (unsure if I can do this with python, it's a binary mmap-able database that has the same content of the XML files) of GObject Introspecton, but I'm unsure how to integrate that with Jedi...

Can you give me some pointers? Would you accept a patch that will give Jedi the ability to parse these files? Alternatively, if that's out of the scope of Jedi itself, can I somehow monkey-patch this from within Builder when I call Jedi? I need Jedi in there (instead of just writing a whole new thing) because it already knows about types within the python code and works very well.

Another option to consider is maybe (that's a big maybe, I'm not sure how simple it is to do) we can change pygobject to somehow be more Jedi friendly, so no change to Jedi itself will be needed?

[davidhalter]

What does the DynamicImporter exactly do? How does it work? Or even better: Why do you need it?

The problem seems to be that in gi/repository/__init__.py you have a line that uses sys.meta_path, which is not supported by Jedi. This linje is the problem: sys.meta_path.append(DynamicImporter('gi.repository')).

[elad661]

The Dynamic Importer (implementation: https://git.gnome.org/browse/pygobject/tree/gi/importer.py#n126 ) loads GObject-Intropsection enabled libraries. It implements the PEP302 Importer Protocol.

It uses the IntrospectionModule class from https://git.gnome.org/browse/pygobject/tree/gi/module.py#n111 which is a wrapper around the introspection typelib. The actual wrapping is done by __getattr__(). This class implements a __dir__() method which basically shows how to get a list of properties / methods of GObjects in pygobject.

The actual loading is done in C, by this file https://git.gnome.org/browse/pygobject/tree/gi/pygi-repository.c (methods accessible by Python are defined in line 304). It's done by using the GObject Introspection library to load info about the libraries from the .typelib files which describe all the interfaces, properties, functions and such that the libraries provide.

Here's some example code to get a list of properties of a GObject-Introspection library (module):

from gi import Repository
r = Repository.get_default()
r.require('Gtk') # See note about versions
print(r.get_infos('Gtk'))

For each object in this result set you can use .get_fields(), .get_properties() and similar methods to get their "children" and .get_name() to get the actual object name that can be used for completion.

Note about versions:
by default the .require() method (and the actual import in user code) selects the newest version available of the library. However, different major versions might be installed and might have slightly different APIs, so PyGObject has gi.require_version() and it asks users to do it before the import so they get the major version they want.
The second parameter for .require() in your code would be the version string if gi.require_version was called. This is an example call:
gi.require_version('Ide', '1.0')

The first parameter is the "namespace" - the name of the library/module you want to import, and the second parameter is the version.

Is this clear enough? I could probably do more research and dive further into specifics.

Regarding the problem you mentioned, I can't think of a way to implement PyGObject without using the meta_path feature - that's how from gi.repository import Gtk works. (I'm not sure I could convince the PyGObject people to change this even if there is a way to do this which is supported by Jedi).

If you feel special-casing PyGObject is out of scope for Jedi, can you suggest a way to monkey-patch Jedi when using it in Builder to add this kind of support?

[davidhalter]

So why are you not just importing Gtk by default?

[elad661]

Because not every script / app that uses GObject has to use Gtk, and because that would only fix the case for one GObject-Intropsection library - Gtk.

There are many more of these and we want to support them all for completion, and it doesn't make sense for pygobject to import all libraries just so completion would work.

On my Fedora install I have 192 different library that can be used with GObject-Introspection. Gtk is just one of them.

[davidhalter]

Ok. That's what I wanted to know.

If you just want to monkey-patch Jedi, I guess it's best to replace the Importer class. You could just inherit from it and overwrite follow or _do_import. I think that would work.

[elad661]

I've written a monkey patch to make jedi support GObject Introspection modules. You can re-use this code in Jedi if you want.

from gi.importer import DynamicImporter
from gi.module import IntrospectionModule
import jedi
from jedi.evaluate.compiled import CompiledObject
from jedi.evaluate.imports import Importer
gi_importer = DynamicImporter('gi.repository')

class PatchedJediCompiledObject(CompiledObject):
    "A modified version of Jedi CompiledObject to work with GObject Introspection modules"
    def _cls(self):
        if self.obj.__class__ == IntrospectionModule:
            return self
        else:
            return super()._cls()

class PatchedJediImporter(Importer):
    "A modified version of Jedi Importer to work with GObject Introspection modules"
    def follow(self):
        module_list = super().follow()
        if module_list == []:
            import_path = '.'.join([str(i) for i in self.import_path])
            if import_path.startswith('gi.repository'):
                try:
                    module = gi_importer.load_module(import_path)
                    module_list = [PatchedJediCompiledObject(module)]
                except ImportError:
                    pass
        return module_list

jedi.evaluate.imports.Importer = PatchedJediImporter

Basically, in follow(), if Jedi couldn't find the module in the "normal" way, I check if the import was done from gi.repository. If it is, I use pygobject's dynamic importer to load the module, convert it into a Jedi CompiledObject, and return that.

I patched CompiledObject as well because many GObject Introspection modules are instances of IntrospectionModule, which is basically empty. Without this condition there completion would only work for modules which have GObject Overrides.

I can re-format this as a patch for Jedi (and wrap it all in conditions, of course, so Jedi won't require pygojbect) if you want, but I am quite sure that my naive way of implementing this is not good enough to be upstreamed :)

Another issue with this monkey patch is that iterating over (for example) GLib to get the completions will result in a lot of PyGIDeprecationWarnings being spewed to the terminal when Jedi iterates over deprecated properties or functions, but these can be easily filtered.

[davidhalter]

Well, it's actually a pretty good solution. Though, I don't know, why you need the PatchedJediCompiledObject. That being said, I don't like to push these kinds of things into Jedi, because it just clutters Jedi's import logic.

However, I just remembered that we have settings.auto_import_modules. Could you try if that works for gi.repository.Gtk? If it does, we could create a star logic like:

auto_import_modules = [
     'hashlib',  # setattr    
     'gi.repository.Gtk',                                          
]

[elad661]

I need PatchedJediCompiledObject because GObject Introspection that don't have overrides (eg. GtkSource) are basically class instances of IntrospectionModule, and IntrospectionModule is basically empty, not patching it there would mean Jedi would only be able to provide completions for GObject Introspection modules that have overrides (_names_dict_ensure_one_dict calls LazyNamesDict with self._cls(), which would be IntrospectionModule, which is empty).

Overrides in GObject Introspection are basically python files that wrap around the module to make some APIs nicer, they are not used in many libraries because usually the automatic bindings just do the right thing.

Regarding auto_import_modules, as I mentioned before (when you suggested to just always import Gtk), it's unfortunately not good enough, because Gtk is not the only library available via pygobject... there are many many others. In a common Gtk app you would likely use more than just Gtk itself, you could use GLib, gstreamer, GIO, GnomeBluetooth, GUdev... and many many others.

GObject Introspection is not just for Gtk, it's a generic system that allows you to get automatic language bindings for C libraries that use GObject, so the platform libraries are written in C and then via GObject Introspection you can use them from Python, JavaScript, Vala, and so on, without the developer having to generate bindings for each language manually.

[davidhalter]

it's unfortunately not good enough, because Gtk is not the only library available via pygobject... there are many many others.

I know, but is it working with just Gtk? Because I'm thinking about a solution, how it could work for all the modules.

[elad661]

Doesn't seem to be working:

In [3]: jedi.settings.auto_import_modules.append('gi.repository.Gtk')

In [4]: s = jedi.Script('from gi.repository import Gtk\nGtk.')

In [5]: s.completions()
Out[5]: []

[davidhalter]

Hmm ok. I (or you) should look into that sometime in the future. It's something that I would expect to work. Once it does we can add this star logic (which needs to be implemented first):

auto_import_modules = [
     'hashlib',  # setattr    
     'gi.repository.*',                                          
]

[elad661]

I'm not sure it's a good solution because loading all the GObject Introspection libraries available on the user's machine will waste a lot of memory - I'm pretty sure it loads the .so file after it parses the typelib.

But it'd still be interesting to see why it didn't work.

[davidhalter]

I'm not sure it's a good solution because loading all the GObject Introspection libraries available on the user's machine will waste a lot of memory

The star import wouldn't imply loading every single module. It would imply that Jedi would load each and every module requested from gi.repository by native imports.

[elad661]

I'll debug why the auto import didn't work later this week. Meanwhile, I'm trying to make jedi understand pygobject style docstrings so it will know the return types of functions, to make completion work in more cases.

These annotations either look like this:
get_info_window(self) -> GtkSource.CompletionInfo

To test my implementation I'm using this line of code:
jedi.Script('from gi.repository import GtkSource\nf=GtkSource.Completion().get_info_window()\nf.').completions()

If I try to run this without any modifications (apart from my patch to make jedi know about pygobject), it just outputs a very long traceback that ends in:

  File "/home/elad/Sources/jedi/jedi/evaluate/compiled/fake.py", line 70, in get_module
    return __import__(imp_plz)
ImportError: No module named 'gi._gobject._gobject'

To work around that now I added a simple condition to change gi._gobject._gobject to gi._gobject.

After working around this, I get no completions, so I tried adding a simple regex to catch the return type annotation in DOCSTRING_RETURN_PATTERNS in docstrings.py, but it didn't do anything. I added a debug print to search_return_in_docstr() and found out it is not called at all when executing my code, and I'm not sure why is that.

I kept looking and found _parse_function_doc() which seems to be a different implementation of a docstring parser. This one is actually called, however it seems to be called with a different docstring than the docstring of the actual function (the docstring that get passed to this function is Initialize self. See help(type(self)) for accurate signature.).

Enabling jedi debugging makes it seems it goes too deep in the tree and finds documentation of a more basic type for some reason? I'm not sure what's going on here:

<object object at 0x7f5f729d60d0>   dbg: finder.filter_name "<Name: Completion@2,12>" in (<CompiledObject: <IntrospectionModule 'GtkSource' from '/usr/lib64/girepository-1.0/GtkSource-3.0.typelib'>>): '[<CompiledName: (gi.repository.GtkSource).Completion>]'@None
<object object at 0x7f5f729d60d0>   dbg: finder._names_to_types: [<CompiledName: (gi.repository.GtkSource).Completion>] -> [<CompiledObject: <class 'gi.repository.GtkSource.Completion'>>]
<object object at 0x7f5f729d60d0>  dbg: eval_trailer: Node(trailer, [<Operator: (>, <Operator: )>]) in scope <CompiledObject: <class 'gi.repository.GtkSource.Completion'>>
<object object at 0x7f5f729d60d0>   dbg: execute: <CompiledObject: <class 'gi.repository.GtkSource.Completion'>> <Arguments: ()>
<object object at 0x7f5f729d60d0>    dbg: execute: <InstanceElement of <CompiledObject: <slot wrapper '__init__' of 'gi._gobject.GObject' objects>>> <Arguments: ()>

Clearly the relevant information would be in gi.repository.GtkSource.Completion and not in whatever it is it dives into after that.

Can you give me any pointers as to how to debug this issue so I can get jedi to understand these annotations?

Thanks.

[davidhalter]

Hmm. To me it looks more like a bug. It actually reminds me a lot of a recent issue. Pretty sure this one would fix it: #615 (see last comment).

[elad661]

It took me a while to find the time for this, but, I looked at the auto_import_modules issue today, and it seems that it's only used once in jedi's code, in _load_module, which is only called from _do_import if a module file was found.

Running with debug enabled, I can see this:

dbg: Parsed None, with 1 parsers in 1 splits.
dbg: start: 'Gtk' in <fast.FastModule: @1-2>
dbg: eval_element <Name: Gtk@2,4>@(2, 4)
dbg: finder.filter_name "<Name: Gtk@2,4>" in (<ModuleWrapper: <fast.FastModule: @1-2>>): '[<Name: Gtk@1,26>]'@(2, 4)
dbg: search_module 'gi' in None
dbg: pickle loaded: '/usr/lib64/python3.4/site-packages/gi/__init__.py'
dbg: search_module 'gi.repository' in paths ['/usr/lib64/python3.4/site-packages/gi']
dbg: pickle loaded: '/usr/lib64/python3.4/site-packages/gi/repository/__init__.py'
 dbg: finder.filter_name "'Gtk'" in (<ModuleWrapper: <fast.FastModule: repository@21-31>>): '[]'@None
 dbg: finder._names_to_types: [] -> []
dbg: search_module 'gi.repository.Gtk' in paths ['/usr/lib64/python3.4/site-packages/gi/repository']
warning: None:1:26: E3 Potential ImportError.

It looks for the file, and since it doesn't find it, it raises an exception and _load_module is never called, which means the auto_import_modules setting is never taken into consideration in this case.

The ImportError is added in the # The module is not a package. case.

[davidhalter]

Hmm ok. We might actually need to modify the architecture then. Thanks for the research.

[aswinmohanme]

What is the status of this issue ?

[davidhalter]

Obivously no one has worked on it.

[albfan]

@elad661 can you add a wip branch on your fork for this?

[albfan]

Someone points out fakegir (but the link is broken)

$ git clone https://github.com/strycore/fakegir
$ cd fakegir
$ WITHDOCS=1 ./fakegir.py

Seems to workaround introspection autocompletion

[elad661]

@albfan As far as I understand, and correct me if I'm wrong, @davidhalter will not accept GObject-Introspection-specific workarounds into Jedi. That's why we've been monkey-patching jedi in gnome-builder to add this functionality.

[albfan]

Sure. is a good decision. Just adding some related info for people looking into this issue (because I see a broken link).

[albfan]

After reread all this:

This would be the way (accepted by jedi maintainer) to recognize GObject introspection

auto_import_modules = [
     'hashlib',  # setattr    
     'gi.repository.*',                                          
]

but it is not working #531 (comment)

Meanwhile it is monkey-patched on Builder https://gitlab.gnome.org/GNOME/gnome-builder/blob/master/src/plugins/jedi/jedi_plugin.py#L100

[albfan]

I tried initial example on this issue using ipython:

I get autocompletions like import:

In [2]: import jedi

In [3]: s3 = jedi.Script('''import gi
   ...: gi.require_version('Gtk', '3.0')
   ...: from gi.repository im''')

In [4]: s3.completions()
Out[4]: [<Completion: import>]

but no for things like Gtk.:

In [5]: s3 = jedi.Script('''import gi
   ...: gi.require_version('Gtk', '3.0')
   ...: from gi.repository import Gtk
   ...: Gtk.''')

In [6]: s3.completions()
Out[6]: []

As ipython comes with jedi for autocompletion I tried this directly and it offers autocompletion correctly:

In [7]: import gi

In [8]: gi.require_version('Gtk', '3.0')

In [9]: from gi.repository import Gtk

In [10]: Gtk.                  
              Gtk.AboutDialog                    Gtk.AccelGroup                     Gtk.AccelKey                       Gtk.AccelMap                        
              Gtk.AboutDialogClass               Gtk.AccelGroupClass                Gtk.AccelLabel                     Gtk.AccelMapClass                   
              Gtk.AboutDialogPrivate             Gtk.AccelGroupEntry                Gtk.AccelLabelClass                Gtk.Accessible                     >
              Gtk.AccelFlags                     Gtk.AccelGroupPrivate              Gtk.AccelLabelPrivate              Gtk.AccessibleClass                 

So I think the example just miss some configuration: what is missing from jedi.Script() to ipython REPL?

That could clarify why autoimport_modules could work to offer autocompletions

Here is the complete screencast:

[davidhalter]

autoimport_modules does a different thing than IPython. In IPython you have live objects, because of how imports work, but maybe we should just change that and use normal Python imports in those cases. I'll think about it.

[davidhalter]

@albfan and all the others that wanted this. IMO the current jedi master branch is working well with the issues you've had. Sorry it took so long, but I think I never really figured out what the optimal solution was. IMO the current solution is not perfect, but it should be good enough for gi.

[elad661]

Thanks @davidhalter!!

[albfan]

Work with upstream directly feels great!

[sherifrad]

Can anyone plz help me with fixing this issue ?!

[albfan]

@ShevoPramide this is fixed, open another issue if you have any problem

[karbiv]

github.com/strycore/fakegir is a logical workaround for GObject dynamic import. It works.
For fakegir to work, this line in jedi/setting.py must be removed(commented out):

'gi', # This third-party repository (GTK stuff) doesn't really work with jedi

Fakegir-like solution should be a standard for native modules, like a minimal documentation.
https://www.python.org/dev/peps/pep-0484/

Everything else in comments is hard/impossible to follow.

[davidhalter]

I think once stubs exist for gi I'm happy to remove the jedi setting @karbiv was mentioning.

[karbiv]

@davidhalter Probably a new open issue should be created, after further investigation.
I started debugging Jedi when Emacs showed message "file 'nil' does not exist", because in Jedi module_path was empty. gi was detected by Jedi as a "compiled context" and such native modules/packages in Jedi are considered as not worthy to have a file path.

import gi

But there exists /usr/lib/python3.8/site-packages/gi path with __init__.py in it.
Jedi should return at least a path to __init__.py;
Instead, Jedi just ignores modules that are included in jedi.settings.auto_import_modules, currently only gi is there.

jedi.settings.auto_import_modules should be a separate python file that resides in a user's home directory.
That file should map module or package names to stub paths.

[davidhalter]

I really don't understand what you want :)

[karbiv]

@davidhalter

1. To not pollute Jedi's sys_path with stub files paths.
2. To make stub files paths configuration explicit in Jedi.
3. Search in sys_path first, if not found then search in stub files paths.

Why?

import gi

Jedi can find it as /usr/lib/python3.8/site-packages/gi/__init__.py
If I put generated stub files in sys_path, that file will be there, and Jedi yields that stub file.

But I still want from Jedi a sys_path file: /usr/lib/python3.8/site-packages/gi/__init__.py that Jedi could find before polluting sys_path with a stub files path. That file in stubs can be empty, or not complete.

from gi.repository import Gtk, GLib

Without stubs Jedi can find gi.repository as:
/usr/lib/python3.8/site-packages/gi/repository/__init__.py

I as a user don't want that file from stubs path if Jedi can find it in sys_path. Again, what if that file in stubs is empty, incomplete, or even missing? But Jedi can find it in sys_path without stubs.

Currently by putting gi into jedi.settings.auto_import_modules list makes Jedi to ignore gi completely. Nothing found by "go to definition" in Emacs because Jedi returns empty path.
So a user has to open /usr/lib/python3.8/site-packages/jedi/setting.py, residing in installation directory and comment out this:

auto_import_modules = [
    #'gi',  # This third-party repository (GTK stuff) doesn't really work with jedi
]

Wrong assumption in the comment?
Jedi works with gi, OR could find "auto imports" in the library using stub files paths configuration.

[davidhalter]

Do stubs exist for gi? If they do I'm happy to remove it from auto_import_modules. If they don't I won't, because there's really no solution for now.

Also about the sys path configuration, this is not really an issue and with the projects branch, one can configure quite a lot about sys paths.

[karbiv]

@davidhalter
Yes, there are all stubs for gi.
The GObject system from its conception describes all stubs or bindings in "*.gir" XML files.
gi can be removed from auto_import_modules

https://github.com/strycore/fakegir generates stubs for all found Gobject libs automatically and puts them into ~/.cache/fakegir

And you are right, sys_path is easy, it's better to patch stub generating code if something is missing. fakegir is just in one python file.

[davidhalter]

I'm not removing it for now. This is not a stub repository and as long as I don't see that I'm not removing it. (except if most other people using gi also prefer that fakegir).

[karbiv]

Doesn't matter after all.(low priority)
gi in a list in jedi.settings.py can be commented out and Jedi stops ignoring gi.

After looking at how pyi(PEP 484) is used, it seems to me that Gobject Introspection was ahead of its time in 2008.
mypy project with stubgen tool still can't generate draft *.pyi files for Qt's pyside2 native modules using runtime introspection. It exits with error, because it expects mro() methods in classes to have no arguments. Kind of fragile still.

[major-mayer]

Can somebody tell me how to remove gi from auto_import_modules using the JediLSP language server in Visual Studio Code?
I don't think editing Jedi's settings.py inside some internal folder of Visual Studio Code will last long. Is there some other possibility?

[unavaliabl3]

Can't get completion from objects created by classes under Gtk:

In

code = '''
import gi
gi.require_version('Gtk', '3.0')
from gi.repository import Gtk

btn = Gtk.Button(label='Button')
btn.'''
jedi.Script(code).complete()

Out

[]