Add automatically generated API doc

[ltalirz]

While working on #1228 I noticed that basic classes like Group or User were not part of the module guide.

I suggest we use sphinx-apidoc to produce an automatically generated & complete API documentation.
Probably, quite a few of the docs/source/.../dev.rst files can be removed (even though in some cases there may be some value in selecting subset of the actual documentation).

[ltalirz]

build succeeded, 1162 warnings.

I'll see whether I can get rid of some of the most prominent problems... then at some point other people will need to help.

[sphuber]

Didn't know that existed, but sounds like a great plan. Just tag me if you open PR or a branch and need help with fixing docstrings

[ltalirz]

I'll be working on this here: https://github.com/ltalirz/aiida_core/tree/issue_1242_sphinx-apidoc
Down to 80 warnings now that actually need looking into.

If it's like usual, then most of these will be simple formatting problems.
I wonder whether there are any tools for helping with formatting the documentation according to sphinx's standards...

[ltalirz]

@sphuber I'm now down to 42 warnings
Many of those related to imports of things that either don't exist or don't want to be imported...
Happy for your assistance, when you find the time (you already have edit rights on my fork)

For reference, here are the warnings

/aiida_core/aiida/backends/djsite/queries.py:docstring of aiida.backends.djsite.queries.get_closest_parents:10: ERROR: Unexpected indentation.
/aiida_core/docs/source/apidoc/aiida.backends.djsite.db.rst:43: WARNING: autodoc: failed to import module u'aiida.backends.djsite.db.tests'; the following exception was raised:
Traceback (most recent call last):
  File "/Users/leopold/Applications/miniconda3/envs/aiida_nanoporous/lib/python2.7/site-packages/sphinx/ext/autodoc.py", line 551, in import_object
    __import__(self.modname)
  File "/aiida_core/aiida/backends/djsite/db/tests.py", line 34, in <module>
    raise InternalError("aiida_test_list is not set, but you are trying to run the tests...")
InternalError: aiida_test_list is not set, but you are trying to run the tests...
/aiida_core/docs/source/apidoc/aiida.backends.djsite.settings.rst:26: WARNING: autodoc: failed to import module u'aiida.backends.djsite.settings.urls'; the following exception was raised:
Traceback (most recent call last):
  File "/Users/leopold/Applications/miniconda3/envs/aiida_nanoporous/lib/python2.7/site-packages/sphinx/ext/autodoc.py", line 551, in import_object
    __import__(self.modname)
  File "/aiida_core/aiida/backends/djsite/settings/urls.py", line 10, in <module>
    from django.conf.urls.defaults import *
ImportError: No module named defaults
/aiida_core/aiida/backends/sqlalchemy/cmdline.py:docstring of aiida.backends.sqlalchemy.cmdline.get_workflow_list:6: ERROR: Unexpected indentation.
/aiida_core/docs/source/apidoc/aiida.backends.sqlalchemy.migrations.rst:17: WARNING: autodoc: failed to import module u'aiida.backends.sqlalchemy.migrations.env'; the following exception was raised:
Traceback (most recent call last):
  File "/Users/leopold/Applications/miniconda3/envs/aiida_nanoporous/lib/python2.7/site-packages/sphinx/ext/autodoc.py", line 551, in import_object
    __import__(self.modname)
  File "/aiida_core/aiida/backends/sqlalchemy/migrations/env.py", line 15, in <module>
    config = context.config
AttributeError: 'module' object has no attribute 'config'
/aiida_core/aiida/backends/sqlalchemy/tests/migrations.py:docstring of aiida.backends.sqlalchemy.tests.migrations.TestMigrationApplicationSQLA.test_migrations_forward_backward:4: ERROR: Unexpected indentation.
/aiida_core/aiida/backends/sqlalchemy/tests/migrations.py:docstring of aiida.backends.sqlalchemy.tests.migrations.TestMigrationApplicationSQLA.test_migrations_forward_backward:5: WARNING: Block quote ends without a blank line; unexpected unindent.
/aiida_core/aiida/backends/tests/restapi.py:docstring of aiida.backends.tests.restapi.RESTApiTestSuite.test_computers_list_page_default:4: WARNING: Inline strong start-string without end-string.
/aiida_core/aiida/backends/tests/restapi.py:docstring of aiida.backends.tests.restapi.RESTApiTestSuite.test_computers_list_page_perpage:1: WARNING: Inline strong start-string without end-string.
/aiida_core/aiida/backends/tests/restapi.py:docstring of aiida.backends.tests.restapi.RESTApiTestSuite.test_computers_list_page_perpage_exceed:1: WARNING: Inline strong start-string without end-string.
/aiida_core/aiida/backends/tests/restapi.py:docstring of aiida.backends.tests.restapi.RESTApiTestSuite.test_computers_unknown_param:5: ERROR: Unexpected indentation.
/aiida_core/aiida/cmdline/commands/graph.py:docstring of aiida.cmdline.commands.graph.Graph:3: ERROR: Unexpected indentation.
/aiida_core/aiida/common/setup.py:docstring of aiida.common.setup.create_htaccess_file:5: WARNING: Explicit markup ends without a blank line; unexpected unindent.
/aiida_core/aiida/common/additions/backup_script/backup_setup.py:docstring of aiida.common.additions.backup_script.backup_setup.BackupSetup:3: ERROR: Unexpected indentation.
/aiida_core/aiida/common/additions/backup_script/backup_setup.py:docstring of aiida.common.additions.backup_script.backup_setup.BackupSetup:4: WARNING: Block quote ends without a blank line; unexpected unindent.
/aiida_core/aiida/orm/autogroup.py:docstring of aiida.orm.autogroup.Autogroup.is_to_be_grouped:2: WARNING: Field list ends without a blank line; unexpected unindent.
/aiida_core/docs/source/apidoc/aiida.orm.rst:107: WARNING: autodoc: failed to import module u'aiida.orm.querytool'; the following exception was raised:
Traceback (most recent call last):
  File "/Users/leopold/Applications/miniconda3/envs/aiida_nanoporous/lib/python2.7/site-packages/sphinx/ext/autodoc.py", line 551, in import_object
    __import__(self.modname)
  File "/aiida_core/aiida/orm/querytool.py", line 10, in <module>
    from aiida.orm.implementation import QueryTool
ImportError: cannot import name QueryTool
/aiida_core/docs/source/apidoc/aiida.orm.rst:140: WARNING: __all__ should be a list of strings, not (['JobCalculation', 'WorkCalculation', 'Code', 'Computer', 'CalculationFactory', 'DataFactory', 'WorkflowFactory', 'QueryBuilder', 'Workflow', 'User', 'Group', 'delete_computer', 'Calculation', 'JobCalculation', 'InlineCalculation', 'make_inline'], ['CalculationFactory', 'DataFactory', 'WorkflowFactory', 'load_node', 'load_workflow']) (in module aiida.orm) -- ignoring __all__
/aiida_core/aiida/orm/implementation/general/group.py:docstring of aiida.orm.implementation.general.group.AbstractGroup.query:3: ERROR: Unexpected indentation.
/aiida_core/aiida/orm/implementation/general/group.py:docstring of aiida.orm.implementation.general.group.AbstractGroup.query:31: WARNING: Inline literal start-string without end-string.
/aiida_core/aiida/restapi/run_api.py:docstring of aiida.restapi.run_api.run_api:7: WARNING: Inline emphasis start-string without end-string.
/aiida_core/aiida/restapi/run_api.py:docstring of aiida.restapi.run_api.run_api:9: WARNING: Inline emphasis start-string without end-string.
/aiida_core/aiida/restapi/__init__.py:docstring of aiida.restapi:14: ERROR: Unexpected indentation.
/aiida_core/docs/source/apidoc/aiida.restapi.common.rst:10: WARNING: autodoc: failed to import module u'aiida.restapi.common.caching'; the following exception was raised:
Traceback (most recent call last):
  File "/Users/leopold/Applications/miniconda3/envs/aiida_nanoporous/lib/python2.7/site-packages/sphinx/ext/autodoc.py", line 551, in import_object
    __import__(self.modname)
  File "/aiida_core/aiida/restapi/common/caching.py", line 11, in <module>
    from aiida.restapi.api import app
ImportError: cannot import name app
/aiida_core/aiida/restapi/common/exceptions.py:docstring of aiida.restapi.common.exceptions:6: ERROR: Unexpected indentation.
/aiida_core/aiida/restapi/common/utils.py:docstring of aiida.restapi.common.utils.Utils.build_headers:3: ERROR: Unexpected indentation.
/aiida_core/aiida/restapi/common/utils.py:docstring of aiida.restapi.common.utils.Utils.build_headers:4: WARNING: Block quote ends without a blank line; unexpected unindent.
/aiida_core/aiida/restapi/common/utils.py:docstring of aiida.restapi.common.utils.Utils.build_headers:5: WARNING: Field list ends without a blank line; unexpected unindent.
/aiida_core/aiida/restapi/common/utils.py:docstring of aiida.restapi.common.utils.Utils.build_response:3: WARNING: Field list ends without a blank line; unexpected unindent.
/aiida_core/aiida/restapi/common/utils.py:docstring of aiida.restapi.common.utils.Utils.build_translator_parameters:5: WARNING: Field list ends without a blank line; unexpected unindent.
/aiida_core/aiida/restapi/common/utils.py:docstring of aiida.restapi.common.utils.Utils.parse_path:8: WARNING: Field list ends without a blank line; unexpected unindent.
/aiida_core/aiida/restapi/common/utils.py:docstring of aiida.restapi.common.utils.Utils.parse_path:10: ERROR: Unexpected indentation.
/aiida_core/aiida/restapi/common/utils.py:docstring of aiida.restapi.common.utils.Utils.strip_prefix:4: ERROR: Unexpected indentation.
/aiida_core/aiida/restapi/translator/base.py:docstring of aiida.restapi.translator.base.BaseTranslator.set_filters:8: ERROR: Unexpected indentation.
/aiida_core/aiida/restapi/translator/base.py:docstring of aiida.restapi.translator.base.BaseTranslator.set_filters:10: WARNING: Block quote ends without a blank line; unexpected unindent.
/aiida_core/aiida/restapi/translator/node.py:docstring of aiida.restapi.translator.node.NodeTranslator.get_results:4: WARNING: Field list ends without a blank line; unexpected unindent.
/aiida_core/aiida/restapi/translator/node.py:docstring of aiida.restapi.translator.node.NodeTranslator.set_query:7: WARNING: Field list ends without a blank line; unexpected unindent.
/aiida_core/docs/source/apidoc/aiida.tools.dbimporters.plugins.rst:66: WARNING: autodoc: failed to import module u'aiida.tools.dbimporters.plugins.test_icsd'; the following exception was raised:
Traceback (most recent call last):
  File "/Users/leopold/Applications/miniconda3/envs/aiida_nanoporous/lib/python2.7/site-packages/sphinx/ext/autodoc.py", line 551, in import_object
    __import__(self.modname)
  File "/aiida_core/aiida/tools/dbimporters/plugins/test_icsd.py", line 14, in <module>
    from aiida.djsite.db.testbase import AiidaTestCase
ImportError: No module named djsite.db.testbase
/aiida_core/docs/source/apidoc/aiida.utils.rst:58: WARNING: autodoc: failed to import module u'aiida.utils.queries'; the following exception was raised:
Traceback (most recent call last):
  File "/Users/leopold/Applications/miniconda3/envs/aiida_nanoporous/lib/python2.7/site-packages/sphinx/ext/autodoc.py", line 551, in import_object
    __import__(self.modname)
  File "/aiida_core/aiida/utils/queries.py", line 10, in <module>
    class jsonb_array_length(FunctionElement):
NameError: name 'FunctionElement' is not defined
/aiida_core/aiida/work/workchain.py:docstring of aiida.work.workchain.Stepper.step:3: ERROR: Unexpected indentation.
/aiida_core/aiida/work/workchain.py:docstring of aiida.work.workchain.Stepper.step:5: WARNING: Block quote ends without a blank line; unexpected unindent.

[sphuber]

Excellent, I will see if I can take a look soon, thanks

[sphuber]

I have had a look and fixed a bunch of errors. We are now down to 3, that I don't really now how to fix.
These are the warnings:

/home/seb/code/aiida/env/dev/aiida-core/docs/source/apidoc/aiida.backends.djsite.db.rst:43: WARNING: autodoc: failed to import module u'aiida.backends.djsite.db.tests'; the following exception was raised:
Traceback (most recent call last):
  File "/home/seb/.virtualenvs/aiida_dev/local/lib/python2.7/site-packages/sphinx/ext/autodoc.py", line 551, in import_object
    __import__(self.modname)
  File "/home/seb/code/aiida/env/dev/aiida-core/aiida/backends/djsite/db/tests.py", line 34, in <module>
    raise InternalError("aiida_test_list is not set, but you are trying to run the tests...")
InternalError: aiida_test_list is not set, but you are trying to run the tests...
/home/seb/code/aiida/env/dev/aiida-core/docs/source/apidoc/aiida.backends.sqlalchemy.migrations.rst:17: WARNING: autodoc: failed to import module u'aiida.backends.sqlalchemy.migrations.env'; the following exception was raised:
Traceback (most recent call last):
  File "/home/seb/.virtualenvs/aiida_dev/local/lib/python2.7/site-packages/sphinx/ext/autodoc.py", line 551, in import_object
    __import__(self.modname)
  File "/home/seb/code/aiida/env/dev/aiida-core/aiida/backends/sqlalchemy/migrations/env.py", line 13, in <module>
    from alembic.context import config
ImportError: cannot import name config
/home/seb/code/aiida/env/dev/aiida-core/docs/source/apidoc/aiida.restapi.common.rst:10: WARNING: autodoc: failed to import module u'aiida.restapi.common.caching'; the following exception was raised:
Traceback (most recent call last):
  File "/home/seb/.virtualenvs/aiida_dev/local/lib/python2.7/site-packages/sphinx/ext/autodoc.py", line 551, in import_object
    __import__(self.modname)
  File "/home/seb/code/aiida/env/dev/aiida-core/aiida/restapi/common/caching.py", line 11, in <module>
    from aiida.restapi.api import app
ImportError: cannot import name app

The second and third are import errors. They come from the restapi and alembic (SqlAlchemy migrations). Maybe you know about the first and we can ask Spyros about the latter.

The first warning is due to an internal error that we raise related to testing in aiida/backends/djsite/db/tests.py. Looking at the code, I think this is an old file that is no longer used, after the test system was adapted to be able to run both backends. I think we could remove the file, but am not sure. Let's find out tomorrow

[ltalirz]

@sphuber Thanks a bunch, this is moving forward much faster than I expected!

The restapi error looks like a typo (should be App), but correcting the typo yields another error

 File "/Users/leopold/Applications/miniconda3/envs/aiida_nanoporous/lib/python2.7/site-packages/flask_cache/__init__.py", line 121, in __init__
    self.init_app(app, config)
  File "/Users/leopold/Applications/miniconda3/envs/aiida_nanoporous/lib/python2.7/site-packages/flask_cache/__init__.py", line 134, in init_app
    base_config = app.config.copy()
AttributeError: type object 'App' has no attribute 'config'

@waychal aiida/restapi/common/caching.py was written by you, could you please fix this? I have given you write access to my fork as well.
https://github.com/ltalirz/aiida_core/tree/issue_1242_sphinx-apidoc
I guess this broken piece of code anyhow never gets called, so perhaps it should best be deleted(?)

[ltalirz]

This work continues in PR #1330

[ltalirz]

#1330 merged