extend plugin design guidelines

docs/source/concepts/calculations.rst

@@ -147,7 +147,7 @@ When a calculation job is launched, the engine will take it roughly through the
  * **Upload**: the calculation job implementation is used to transform the input nodes into the required input files, which are uploaded to a 'working' directory on the target machine
  * **Submit**: to execute the calculation, a job is submitted to the scheduler of the computer on which the input `code` is configured.
  * **Update**: the engine will query the scheduler to check for the status of the calculation job
- * **Retrieve**: once the job has finished, the engine will retrieve the output files, specified by the plugin and store them in a node attached as an output node to the calculation
+ * **Retrieve**: once the job has finished, the engine will retrieve the output files, specified by the calculation plugin and store them in a node attached as an output node to the calculation
 
 All of these tasks require the engine to interact with the computer, or machine, that will actually run the external code.
 Since the :py:class:`~aiida.orm.nodes.data.code.Code` that is used as an input for the calculation job, which is configured for a specific :py:class:`~aiida.orm.computers.Computer`, the engine knows exactly how to execute all these tasks.

docs/source/developer_guide/core/caching.rst

@@ -2,7 +2,7 @@ Caching: implementation details
 +++++++++++++++++++++++++++++++
 
 This section covers some details of the caching mechanism which are not discussed in the :ref:`user guide <caching>`.
-If you are developing a plugin and want to modify the caching behavior of your classes, we recommend you read :ref:`this section <caching_matches>` first.
+If you are developing plugins and want to modify the caching behavior of your classes, we recommend you read :ref:`this section <caching_matches>` first.
 
 .. _devel_controlling_hashing:
 

docs/source/developer_guide/design/changes.rst

@@ -95,4 +95,33 @@ In particular we will strive to:
  - if we are forced to change it anyway, deprecate a signifcant amount of time in advance
  - for backwards incompatible changes, increase the major version
 
-For better clarity, we are :ref:`curating a list of classes and functions<python_api_public_list>` (exposed at the second level) that are intended to be public and for which the above policy will be enforced
+For better clarity, we are :ref:`curating a list of classes and functions<python_api_public_list>` (exposed at the second level) that are intended to be public and for which the above policy will be enforced
+
+Version 0.9.0
++++++++++++++
+
+The plugin system
+-----------------
+
+The plugin system was designed with the following goals in mind.
+
+* **Sharing of calculations, workflows and data types**: plugins are bundled in a python package, distributed as a zip source archive, python ``egg`` or PyPI package. There is extensive documentation available for how to distribute python packages `here <https://packaging.python.org/>`_.
+
+* **Ease of use**: plugins are listed on the `AiiDA plugin registry <registry>`_ and can be installed with one simple command. This process is familiar to every regular python user.
+
+* **Decouple development and update cycles of AiiDA and plugins**: since plugins are separate python packages, they can be developed in a separate code repository and updated when the developer sees fit without a need to update AiiDA. Similarly, if AiiDA is updated, plugins may not need to release a new version.
+
+* **Promote modular design in AiiDA development**: separating plugins into their own python packages ensures that plugins can not (easily) access parts of the AiiDA code which are not part of the public API, enabling AiiDA development to stay agile. The same applies to plugins relying on other plugins.
+
+* **Low overhead for developers**: plugin developers can write their extensions the same way they would write any python code meant for distribution.
+
+* **Automatic AiiDA setup and testing of plugins**: installation of complete python environments consisting of many packages can be automated, provided all packages use ``setuptools`` as a distribution tool. This enables use of AiiDA in a service-based way using, e.g., docker images. At the same it becomes possible to create automated tests for any combination of plugins, as long as the plugins provide test entry points.
+
+
+The chosen approach to plugins has some limitations:
+
+* the interface for entry point objects is enforced implicitly by the way the object is used. It is the responsibility of the plugin developer to test for compliance, especially if the object is not derived from the recommended base classes provided by AiiDA. This is to be clearly communicated in the documentation for plugin developers;
+* The freedom of the plugin developer to name and rename classes ends where the information in question is stored in the database as, e.g., node attributes.
+* The system is designed with the possibility of plugin versioning in mind, however this is not implemented yet.
+* In principle, two different plugins can give the same name to an entry point, creating ambiguity when trying to load the associated objects. Plugin development guidelines in the documentation will advise on how to avoid this problem, and this is addressed via the use of a centralized registry of known AiiDA plugins.
+* Plugins can potentially contain malicious or otherwise dangerous code. In the registry of AiiDA plugins, we try to flag plugins that we know are safe to be used.

docs/source/developer_guide/plugins.rst

@@ -12,4 +12,3 @@ Plugin development
    plugins/documenting
    plugins/plugin_tests
    plugins/publish
-   plugins/update_plugin

docs/source/developer_guide/plugins/basics.rst

@@ -4,72 +4,28 @@ Basics
 ======
 
 
-What a plugin Is
-----------------
+Nomenclature
+------------
 
-An AiiDA plugin is a `python package <packages>`_ that provides a set of extensions to AiiDA.
+An AiiDA plugin is an extension of AiiDA, announcing itself to ``aiida-core`` by means of a new :ref:`entry point <plugins.entry_points>`.
 
-AiiDA plugins can use :ref:`entry points <plugins.entry_points>` in order to make the ``aiida-core`` package aware of the extensions.
+AiiDA plugins can be bundled and distributed in a `python package <packages>`_ that provides a set of extensions to AiiDA.
 
 .. note::
 
-  In the python community, the term 'package' is used rather loosely.
+  The python community uses the term 'package' rather loosely.
   Depending on context, it can refer to a collection of python modules or it may, in addition, include the files necessary for building and installing the package.
 
 .. _packages: https://docs.python.org/2/tutorial/modules.html?highlight=package#packages
 
 
-Goals
------
-
-The plugin system was designed with the following goals in mind.
-
-* **Sharing of workflows and extensions**: a workflow or extension is written as a python package, distributed as a zip source archive, python ``egg`` or PyPI package. There is extensive documentation available for how to distribute python packages `here <https://packaging.python.org/>`_.
-
-* **Ease of use**: plugins can be found in an online curated list of plugins and installed with one simple command. This process is familiar to every regular python user.
-
-* **Decouple development and update cycles of AiiDA and plugins**: since plugins are separate python packages, they can be developed in a separate code repository and updated when the developer sees fit without a need to update AiiDA. Similarly, if AiiDA is updated, plugins may not need to release a new version.
-
-* **Promote modular design in AiiDA development**: separating plugins into their own python packages ensures that plugins can not (easily) access parts of the AiiDA code which are not part of the public API, enabling AiiDA development to stay agile. The same applies to plugins relying on other plugins.
-
-* **Low overhead for developers**: plugin developers can write their extensions the same way they would write any python code meant for distribution.
-
-* **Automatic AiiDA setup and testing of plugins**: installation of complete python environments consisting of many packages can be automated, provided all packages use ``setuptools`` as a distribution tool. This enables use of AiiDA in a service-based way using, e.g., docker images. At the same it becomes possible to create automated tests for any combination of plugins, as long as the plugins provide test entry points.
-
-
-Design guidelines
-------------------
-
-* **Start simple.**: make use of existing classes like :py:class:`~aiida.orm.nodes.process.calculation.calcjob.CalcJobNode`, :py:class:`~aiida.orm.nodes.data.dict.Dict`, :py:class:`~aiida.orm.nodes.data.singlefile.SinglefileData`, ... Write only what is necessary to pass information from and to AiiDA.
-
-* **Don't break data provenance.**: store *at least* what is needed for full reproducibility.
-
-* **Parse what you want to query for.**: make a list of which information to:
-
-  #. parse into the database for querying (:py:class:`~aiida.orm.nodes.data.dict.Dict`, ...)
-  #. store in files for safe-keeping (:py:class:`~aiida.orm.nodes.data.singlefile.SinglefileData`, ...)
-  #. leave on the remote computer (:py:class:`~aiida.orm.nodes.data.remote.RemoteData`, ...)
-
-* **Expose the full functionality.**: standardization is good but don't artificially limit the power of a code you are wrapping - or your users will get frustrated. If the code can do it, there should be *some* way to do it with your plugin.
-
-
 What a plugin can do
 --------------------
 
-* Add new classes to AiiDA's unified interface, including:
-
-  - calculations
-  - parsers
-  - data types
-  - schedulers
-  - transports
-  - db importers
-  - db exporters
-  - subcommands to some ``verdi`` commands
-
+* Add a new class to AiiDA's :ref:`entry point groups <plugins.aiida_entry_points>`, including:: calculations, parsers, workflows, data types, verdi commands, schedulers, transports and importers/exporters from external databases.
   This typically involves subclassing the respective base class AiiDA provides for that purpose.
-* Install separate commandline and/or GUI executables
-* Depend on any number of other plugins (the required versions must not clash with AiiDA's requirements)
+* Install new commandline and/or GUI executables
+* Depend on, and build on top of any number of other plugins (as long as their requirements do not clash)
 
 
 .. _plugins.maynot:
@@ -93,13 +49,70 @@ We will advise on how to proceed.
 .. _registry: https://github.com/aiidateam/aiida-registry
 
 
-Limitations
------------
-
-The chosen approach to plugins has some limitations:
+Design guidelines
+------------------
 
-* In the current version the interface for entry point objects is enforced implicitly by the way the object is used. It is the responsibility of the plugin developer to test for compliance, especially if the object is not derived from the recommended base classes provided by AiiDA. This is to be clearly communicated in the documentation for plugin developers;
-* The freedom of the plugin developer to name and rename classes ends where the information in question is stored in the database as, e.g., node attributes.
-* The system is designed with the possibility of plugin versioning in mind, however this is not implemented yet.
-* In principle, two different plugins can give the same name to an entry point, creating ambiguity when trying to load the associated objects. Plugin development guidelines in the documentation will advise on how to avoid this problem, and this is addressed via the use of a centralized registry of known AiiDA plugins.
-* Plugins can potentially contain malicious or otherwise dangerous code. In the registry of AiiDA plugins, we try to flag plugins that we know are safe to be used.
+Wrapping an external code
+.........................
+
+In order to wrap an external simulation code for use in AiiDA, you will need to write a calculation input plugin (subclassing the :py:class:`~aiida.engine.CalcJob` class) and an output parser plugin (subclassing the :py:class:`~aiida.parsers.Parser` class):
+
+ * | **Start simple.**
+   | Make use of existing classes like :py:class:`~aiida.orm.nodes.data.dict.Dict`, :py:class:`~aiida.orm.nodes.data.singlefile.SinglefileData`, ...
+   | Write only what is necessary to pass information from and to AiiDA.
+ * | **Don't break data provenance.**
+   | Store *at least* what is needed for full reproducibility.
+ * | **Parse what you want to query for.**
+   | Make a list of which information to:
+
+     #. parse into the database for querying (:py:class:`~aiida.orm.nodes.data.dict.Dict`, ...)
+     #. store in files for safe-keeping (:py:class:`~aiida.orm.nodes.data.singlefile.SinglefileData`, ...)
+     #. leave on the remote computer (:py:class:`~aiida.orm.nodes.data.remote.RemoteData`, ...)
+
+ * | **Expose the full functionality.**
+   | Standardization is good but don't artificially limit the power of a code you are wrapping - or your users will get frustrated.
+   | If the code can do it, there should be *some* way to do it with your plugin.
+
+ * | **Don't rely on AiiDA internals.**
+   | AiiDA's :ref:`public python API<python_api_public_list>` includes anything that you can import via  ``from aiida.module import thing``.
+   | Functionality at deeper nesting levels is not considered part of the public API and may change between minor AiiDA releases, forcing you to update your plugin.
+
+Folder structure
+................
+
+While it is up to you to decide the folder structure for your plugin, here is how a typical AiiDA plugin package may look like (see also the `aiida-diff`_ demo plugin)::
+
+   aiida-mycode/           - distribution folder
+      aiida_mycode/        - toplevel package (from aiida_code import ..)
+         __init__.py
+         calcs/
+            __init__.py
+            mycode.py      - contains MycodeCalculation
+         parsers/
+            __init__.py
+            mycode.py      - contains MycodeParser
+         data/
+            __init__.py
+            mydat.py       - contains MyData (supports code specific format)
+         commands/
+            __init__.py
+            mydat.py       - contains visualization subcommand for MyData
+         workflows/
+            __init__.py
+            mywf.py        - contains a basic workflow using mycode
+         ...
+      setup.py             - install script
+      setup.json           - install configuration
+      ...
+
+A minimal plugin package instead might look like::
+
+   aiida-minimal/
+      aiida_minimal/
+         __init__.py
+         simpledata.py
+      setup.py
+      setup.json
+
+
+.. _aiida-diff: https://github.com/aiidateam/aiida-diff

docs/source/developer_guide/plugins/documenting.rst

@@ -1,8 +1,8 @@
-====================
-Documenting a plugin
-====================
+===========================
+Documenting plugin packages
+===========================
 
-If you used the `AiiDA plugin cutter`_,  your plugin already comes with a basic
+If you used the `AiiDA plugin cutter`_,  your plugin package already comes with a basic
 documentation that just needs to be adjusted to your needs.
 
  #. Install the ``docs`` extra::
@@ -29,7 +29,7 @@ documentation that just needs to be adjusted to your needs.
     requirements file ``docs/requirements_for_rtd.txt`` and the Python
     configuration file ``docs/source/conf.py`` in Admin => Advanced settings.
 
-Note: When updating the plugin to a new version, remember to update the
+Note: When updating the plugin package to a new version, remember to update the
 version number both in ``setup.json`` and ``aiida_mycode/__init__.py``.
 
 .. _aiida plugin cutter: https://github.com/aiidateam/aiida-plugin-cutter