cEP-0028.md

Optimizing coala for language server

Metadata
cEP	0028
Version	0.1
Title	Optimizing coala for language server
Authors	Kilari Teja mailto:ksdme9@gmail.com
Status	Proposed
Type	Process

Abstract

This cEP describes the optimizations that need to be carried out to improve the performance of the coala based language server.

Introduction

Language servers are the tools used to serve code editors with functionalities such as linting, code fixing, analysis etc in a standalone and independent manner. coala as a code style checker, linter and code fixer could be a powerful tool to be integrated into the code editor.

Code Editing is a continuous process and the editors are expected to provide feedback on the fly, coala on the other hand was primarily designed as an analysis tool to be used after an incremental update to the codebase. Thus, the current implementation of a coala language server is severely limited by coala's performance.

The performance of the server can be improved by primarily optimizing the coala and language server interface and improving the bear configuration options to avoid long running bears.

Proposal

The optimization process focuses to maintain minimal changes to coala core and rather optimize the coala and language server interface. The following are the proposed ideas.

• Section Tagging

  Some sections are not often required to be executed on every cycle of coala analysis especially during use cases such as this. The parameter set can thus be extended to support a new parameter named tags. Since these parameters are not associated with any particular bear but rather control the effect of various sections during executing coala, no support extension for bears would be necessary. These parameters enable support for hooking a section for a particular tag type. For example:

  [longrunning]
  tags = save, change
  enabled = True
  bears = SomeLongRunningBear
  

  This kind of a configuration will enable this section during analysis cycles of coala via a language server when a save or change request is raised while not impacting any untagged regular invocations.

  [longrunning]
  tags = save
  files = *.py
  bears = SomeLongRunningBear
  
  [longrunning.some]
  tags += change
  bears = SomeLongRunningBear
  

  This example shows a case where the sections inherit the tags and thereby longrunning.some runs on both save and change tagged execution cycles.

  To enable tagged executional support, coala-core needs to be extended and the existing --filter-by cli option will need to support runtime section filtering. A new filter will have to be introduced named section_tags. The execution of sections will then be based upon tags passed to --filter-by section_tags. This extension of support for tags would require at least the following changes:

  • coalib.parsing.filters.SectionTagsFilter

    Adds a new section filter to the to check if the section/bear should be enabled for a particular tag based invocation.

    def section_tags_filter(section_or_bear, args):
      """
      Filters the bears or sections by ``tags``.
      :param section_or_bear: A section or bear instance on which filtering
                              needs to be carried out.
      :param args:            Set of tags on which it needs to be filtered.
      :return:                ``True`` if this instance matches the criteria
                              inside args, ``False`` otherwise.
      """
      enabled_tags = list(map(str.lower, args))
      if len(enabled_tags) == 0:
        return True
    
      section = section_or_bear
      if hasattr(section_or_bear, 'section'):
        # If it is a bear or bear like object it has
        # to have an associated section which contains
        # all settings, including tags.
        section = section_or_bear.section
    
      section_tags = section.get('tags', False)
      if str(section_tags) == 'False':
        return False
    
      section_tags = map(str.lower, section_tags)
      return bool(set(section_tags) & set(enabled_tags))

  • coalib.coala_main.run_coala

    Enhance run_coala() to check if the corresponding tags of the sections are enabled.

    # Collect all the filters and try to filter sections
    filters = collect_filters(args, arg_list, arg_parser)
    if len(filters) > 0:
      all_sections = list(sections.values())
      try:
        filtered = apply_filters(filters, sections=all_sections)
        sections = OrderedDict(
            (sect.name.lower(), sect) for sect in filtered)
      except (InvalidFilterException, NotImplementedError) as ex:
        console_printer.print(ex)

  Supported Tags

  The following are the tags that shall be used by coala-ls to trigger sections that need to be run on corresponding request types, these tagged sections can also be used by any other editor plugin to serve similar functionality.

  • open: Tag group to execute during a open event on a file.

  • change: Tag group to execute during a change event on the project files in an editor.

  • save: Tag group to execute on a save event on the project files.

  • format: Tag group to execute when a formatting event is raised on a file.

• Custom entry point into coala core

  Since coala core was not designed to be a long running process coalib does not provide a flexible entry point into it. To maximize the efficiency of coala language server, it will require an entry point that supports:

  • FileProxies: Not all changes on the editor are flushed to the disk, coala core will have to support processing file contents directly as opposed to files. This can be achieved using FileProxies which provide a gateway to files and their in-memory counterparts.

  Support for FileProxies can be introduced by extending FileCaches to additionally provide file content generating/reading methods. The following functions are core to the implementation and API of coala and need to be patched or extended to bring support for coala-ls.

  • coalib.io.FileProxy

    Addition of a FileProxy class.

    @generate_eq('filename')
    class FileProxy:
      """
      ``FileProxy`` is responsible for providing access to
      contents of files and also provides methods to update
      the in memory content register of the said file.
      """
    
      def __init__(self, filename, workspace=None, contents=''):
        """
        Initialize the FileProxy instance with the passed
        parameters. A FileProxy instance always starts at
        a fresh state with a negative version indicating
        that no updating operation has been performed on it.
    
        :param filename:
          The name of the file to create a FileProxy of.
          The filename is internally normcased.
        :param workspace:
          The workspace/project this file belongs to.
          Can be None.
        :param contents:
          The contents of the file to initialize the
          instance with. Integrity of the content or the
          sync state is never checked during initialization.
        """
        logging.debug('File proxy for {} created'.format(filename))
    
        # The file may not exist yet, hence there is no
        # reliable way of knowing if it is a file on the
        # disk or a directory.
        if not path.isabs(filename) or filename.endswith(path.sep):
          raise ValueError('expecting absolute filename')
    
        self._version = -1
        self._contents = contents
        self._filename = path.normcase(filename)
        self._workspace = workspace and path.normcase(workspace)
    
      def __str__(self):
        """
        :return:
          Return a string representation of a file proxy
          with information about its version and filename.
        """
        return '<FileProxy {}, {}>'.format(
          self._filename, self._version)
    
      def __hash__(self):
        """
        :return:
          Returns hash of the instance.
        """
        return hash(self.filename)
    
      def replace(self, contents, version):
        """
        The method replaces the content of the proxy
        entirely and does not push the change to the
        history. It is similar to updating the proxy
        with the range spanning to the entire content.
        :param contents:
          The new contents of the proxy.
        :param version:
          The version number proxy upgrades to after
          the update. This needs to be greater than
          the current version number.
        :return:
          Returns a boolean indicating the status of
          the update.
        """
        if version > self._version:
          self._contents = contents
          self._version = version
    
          logging.debug('File proxy for {} updated to version {}.'
                        .format(self.filename, self.version))
    
          return True
    
        logging.debug('Updating file proxy for {} failed.'
                      .format(self.filename))
    
        return False
    
      def get_disk_contents(self):
        """
        :return:
          Returns the contents of a copy of the file
          on the disk. It might not be in sync with
          the editor version of the file.
        """
        with open(self.filename, 'r',
                  encoding=detect_encoding(self.filename)) as disk:
          return disk.read()
    
      def contents(self):
        """
        :return:
          Returns the current contents of the proxy.
        """
        return self._contents
    
      def lines(self):
        """
        :return:
          Returns the tuple of lines from the contents
          of current proxy instance.
        """
        # If the file is binary, splitlines returns
        # an empty tuple.
        return tuple(self.contents().splitlines(True))
    
      def clear(self):
        """
        Clearing a proxy essentially means emptying the
        contents of the proxy instance.
        """
        self._contents = ''
        self._version = -1
    
      @property
      def filename(self):
        """
        :return:
          Returns the complete normcased file name.
        """
        return self._filename
    
      @property
      def workspace(self):
        """
        :return:
          Returns the normcased workspace of the file.
        """
        return self._workspace
    
      @property
      def version(self):
        """
        :return:
          Returns the current edit version of the file.
        """
        return self._version
    
      @classmethod
      def from_file(cls, filename, workspace, binary=False):
        """
        Construct a FileProxy instance from an existing
        file on the drive.
        :param filename:
          The name of the file to be represented by
          the proxy instance.
        :param workspace:
          The workspace the file belongs to. This can
          be none representing that the the directory
          server is currently serving from is the workspace.
        :return:
          Returns a FileProxy instance of the file with
          the content synced from a disk copy.
        """
        if not binary:
          with open(filename, 'r',
                    encoding=detect_encoding(filename)) as reader:
            return cls(filename, workspace, reader.read())
        else:
          with open(filename, 'rb') as reader:
            return cls(filename, workspace, reader.read())

  • coalib.io.FileProxyMap

    Addition of a FileProxyMap.

    class FileProxyMap:
    """
    FileProxyMap handles a collection of proxies
    and provides a mechanism to reliably resolve
    missing proxies.
    """
    
    def __init__(self, file_proxies=[]):
      """
      :param file_proxies:
        A list of FileProxy instances to initialize
        the ProxyMap with.
      """
      self._map = {proxy.filename: proxy for proxy in file_proxies}
    
    @enforce_signature
    def add(self, proxy: FileProxy, replace=False):
      """
      Add a proxy instance to the map or replaces
      optionally if it already exists.
      :param proxy:
        The proxy instance to register in the map.
      :param replace:
        A boolean flag indicating if the proxy should
        replace an existing proxy of the same file.
      :return:
        Boolean true if registering of the proxy was
        successful else false.
      """
      if self._map.get(proxy.filename) is not None:
        if replace:
          self._map[proxy.filename] = proxy
          return True
        return False
    
      self._map[proxy.filename] = proxy
      return True
    
    def remove(self, filename):
      """
      Remove the proxy associated with a file from the
      proxy map.
      :param filename:
        The name of the file to remove the proxy
        associated with.
      """
      filename = path.normcase(filename)
      if self.get(filename):
        del self._map[filename]
    
    def get(self, filename):
      """
      :param filename:
        The name of file to get the associated proxy instance.
      :return:
        A file proxy instance or None if not available.
      """
      filename = path.normcase(filename)
      return self._map.get(filename)
    
    def resolve(self, filename, workspace=None, hard_sync=True, binary=False):
      """
      Resolve tries to find an available proxy or creates one
      if there is no available proxy for the said file.
      :param filename:
        The filename to search for in the map or to create
        a proxy instance using.
      :param workspace:
        Used in case the lookup fails and a new instance is
        being initialized.
      :hard_sync:
        Boolean flag indicating if the file should be initialized
        from the file on disk or fail otherwise.
      :return:
        Returns a proxy instance or raises associated exceptions.
      """
      filename = path.normcase(filename)
    
      proxy = self.get(filename)
      if proxy is not None:
        return proxy
    
      try:
        proxy = FileProxy.from_file(filename, workspace, binary=binary)
      except (OSError, ValueError) as ex:
        if hard_sync:
          raise ex
    
        # Could raise a ValueError
        proxy = FileProxy(filename, workspace)
        self.add(proxy)
    
      return proxy

  • coalib.misc.Caching

    Support for FileProxyMaps is currently being extended via subclasses of FileCache, this also makes FileCache instances responsible for providing access to both the caches and the contents of associated/other files. Note that these are primarily caches with extended support for additional features.

    • FileDictFileCache

      class FileDictFileCache(FileCache):
        """
        FileDictFileCache extends a traditional FileCache
        to support generation of complete file dict, this
        lets FileDictFileCache provide access to both file
        cache and contents of files from disk.
        """
      
        def __init__(self, *args, **kargs):
          super().__init__(*args, **kargs)
      
        def get_file_dict(self, filename_list, allow_raw_files=False):
          """
          Returns a file dictionary mapping from filename => lines of
          file. Uses coalib.processes.Processing.get_file_dict().
          """
          return get_file_dict(filename_list,
                               allow_raw_files=allow_raw_files)

    • ProxyMapFileCache

      class ProxyMapFileCache(FileCache):
        """
        ProxyMapFileCache is a FileCache that also provides
        methods to produce a file dict from a FileProxyMap.
        This enables analysis of in-memory files by coala.
        """
      
        def __init__(self, *args, **kargs):
          super().__init__(*args, **kargs)
          self.__proxymap = None
      
        def set_proxymap(self, fileproxy_map):
          self.__proxymap = fileproxy_map
      
        def get_file_dict(self, filename_list, allow_raw_files=False):
          return self.get_file_dict_from_fileproxy_map(
                    filename_list, self.__proxymap, allow_raw_files)

    • coalaLsFileCache

      coalals.utils.files.coalaLsFileCache will be a direct subclass of ProxyMapFileCache and is intended to stay as part of coala-ls. Instances of coalaLsFileCache will be passed down to run_coala(), this class will additionally support collection of information from coala-core.

      class coalaLsFileCache(ProxyMapFileCache):
        """
        Subclass of ProxyMapFileCache with support
        for information logging.
        """
      
        def __init__(self, *args, **kargs):
          super().__init__(*args, **kargs)
          self._ls_logger = None
      
        def ls_init(self, fileproxy_map=None, ls_logger=None):
          if fileproxy_map is not None:
            super().set_proxymap(fileproxy_map)
      
          if ls_logger is not None:
            self._ls_logger = ls_logger
      
        def track_files(self, files):
          self._ls_logger('Tracking files: '+str(files))
          return super().track_files(files)

  • coalib.misc.Caching.ProxyMapFileCache.get_file_dict_from_fileproxy_map()

    ProxyMapFileCache.get_file_dict() should generate the file dict from file proxy map.

    def get_file_dict_from_fileproxy_map(self,
                                         filename_list,
                                         fileproxy_map,
                                         allow_raw_files=False):
      """
      Reads all files into a dictionary.
    
      :param filename_list:   List of files to get the contents of.
      :param allow_raw_files: Allow the usage of raw files (non text files),
                              disabled by default
      :param fileproxy_map:   The FileProxyMap instance to use while building
                              a file dict when available.
      :return:                Reads the content of each file into dictionary
                              with filenames as keys.
      """
      file_dict = {}
      for filename in filename_list:
        try:
          proxy = fileproxy_map.resolve(filename,
                                        hard_sync=True)
    
          file_contents = proxy.contents()
          file_lines = file_contents.splitlines(keepends=True)
          file_dict[filename] = tuple(file_lines)
        except (OSError, ValueError) as exception:
          # Raw files are not currently tested.
          if fileproxy_map is not None and allow_raw_files:
            file_dict[filename] = None
            continue
    
          log_exception("Failed to read file '{}' because of an unknown "
                        'error. Leaving it out.'.format(filename),
                        exception,
                        log_level=LOG_LEVEL.WARNING)
    
      logging.debug('Files that will be checked:\n' +
                    '\n'.join(file_dict.keys()))
    
      return file_dict

  • coalib.processes.Processing.instantiate_processes()

    Enhance instantiate_processes() to conditionally make use of cache instance to generate file dict or use default.

    # generate file dict from a cache or use the default
    # generator in case caching is disabled/unavailable.
    file_dict_gen = get_file_dict if cache is None else cache.get_file_dict
    complete_file_dict = file_dict_gen(complete_filename_list,
                                       allow_raw_files=use_raw_files)

  • coalib.coala_main.run_coala()

    Requires changes to function signatures and addition of cache parameter with a default value of None to be passed down upwards.

Known Issues

• Currently, because the support for in-memory analysis has to come as part of FileCaches, FileProxies cannot be used when caching is disabled.