Feature/test coverage improval

.github/workflows/run-tests.yaml

@@ -22,7 +22,8 @@ jobs:
       fail-fast: false
       matrix:
         python-version: ["3.9", "3.10"]
-        os: [ubuntu-18.04, macOS-latest, windows-latest]
+        poetry-version: [1.4.2]
+        os: [ubuntu-latest, macOS-latest, windows-latest]
 
     steps:
     - uses: actions/checkout@v3
@@ -33,21 +34,23 @@ jobs:
     - name: Run poetry image
       uses: abatilo/actions-poetry@v2.0.0
       with:
-          poetry-version: 1.1.13
+          poetry-version: ${{ matrix.poetry-version }}
     - name: Install dependencies
       run: |
         poetry install
     - name: Run tests
       run: |
         poetry run pytest --cov=pystatis tests
   code-quality:
+    runs-on: ${{ matrix.os }}
+    timeout-minutes: 10
     strategy:
       fail-fast: false
       matrix:
-        python-version: [3.10.2]
-        poetry-version: [1.1.13]
+        python-version: ["3.10"]
+        poetry-version: [1.4.2]
         os: [ubuntu-latest]
-    runs-on: ${{ matrix.os }}
+
     steps:
       - uses: actions/checkout@v3
       - uses: actions/setup-python@v2
@@ -73,3 +76,36 @@ jobs:
         run: poetry run safety check
       - name: Run mypy
         run: poetry run mypy src
+  sphinx-documentation:
+    runs-on: ${{ matrix.os }}
+    timeout-minutes: 10
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.10"]
+        poetry-version: [1.4.2]
+        os: [ubuntu-latest]
+
+    steps:
+      - uses: actions/checkout@v3
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v3
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Run poetry image
+        uses: abatilo/actions-poetry@v2.0.0
+        with:
+            poetry-version: ${{ matrix.poetry-version }}
+      - name: Install dependencies
+        run: |
+          poetry install
+      - name: Build documentation
+        run: |
+          cd docs
+          poetry run make clean
+          poetry run make html
+      - name: Upload documentation as artifacts
+        uses: actions/upload-artifact@v3
+        with:
+          name: docs
+          path: docs/build/html/*

.pre-commit-config.yaml

@@ -12,7 +12,7 @@ repos:
       - id: check-added-large-files
       - id: check-merge-conflict
   - repo: https://github.com/pycqa/isort
-    rev: 5.10.1
+    rev: 5.12.0
     hooks:
       - id: isort
         name: isort (python)
@@ -29,3 +29,10 @@ repos:
         files: ^\.github/workflows/
         types: [yaml]
         args: ["--schemafile", "https://json.schemastore.org/github-workflow"]
+  - repo: https://github.com/thclark/pre-commit-sphinx
+    rev: 0.0.1
+    hooks:
+      - id: build-docs
+        name: "Check if documentation compiles"
+        args: ['--cache-dir', 'docs/build/doctrees', '--html-dir', 'docs/build/html', '--source-dir', 'docs/source']
+        language_version: python3

README.md

@@ -129,6 +129,16 @@ clear_cache("21311-0001")  # only deletes the data for the object with the speci
 clear_cache()  # deletes the complete cache
 ```
 
+### Full documentation
+
+The full documentation currently is not hosted online and has to be build locally. This can is achieved by running
+
+```bash
+cd docs && make html
+```
+
+from the project root directory. Besides providing parsed docstrings of the individual package modules, the full documentation currently mirrors most of the readme, like installation and usage. More information on how to use sphinx is provided [here](https://docs.readthedocs.io/en/stable/intro/getting-started-with-sphinx.html).
+
 ## License
 
 Distributed under the MIT License. See `LICENSE.txt` for more information.
@@ -137,8 +147,8 @@ Distributed under the MIT License. See `LICENSE.txt` for more information.
 
 A few ideas we should implement in the maybe-near future:
 
+- Host the documentation online, e.g. on [readthedocs](https://readthedocs.org/)
 - Improve Table parsing. Right now, the parsing is really simple and we should align the cube and table format so that the data frame for tables is more convenient to use.
-- Create a source code documentation with Sphinx or similar tools.
 - Mechanism to download data that is newer than the cached version. Right now, once data is cached, it is always retrieved from cache no matter if there is a newer version online. However, this could be quite challenging as the GENESIS API is really bad in providing a good and consistent field for the last update datetime.
 - Improve Table and Cube metadata so the user can look up the variables contained in the dataset and for each variable the values that this variable can have.
 - Understand and support time series.
@@ -157,7 +167,7 @@ Contributions to this project are highly appreciated! You can either contact the
 
 To contribute to this project, please follow these steps:
 
-1. Install [poetry](https://python-poetry.org/docs/). We recommend installing `poetry` via [pipx](https://pypa.github.io/pipx/) which gives you a global `poetry` command in an isolated virtual environment.
+1. Install [poetry](https://python-poetry.org/docs/).
 2. Clone the repository via git.
 3. Change into the project root directory.
 4. Run `poetry install` to create the virtual environment within `poetry`'s cache folder (run `poetry env info` to see the details of this new virtual environment). `poetry` has installed all dependencies for you, as well as the package itself.

docs/Makefile

@@ -0,0 +1,20 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?=
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = source
+BUILDDIR      = build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

docs/make.bat

@@ -0,0 +1,35 @@
+@ECHO OFF
+
+pushd %~dp0
+
+REM Command file for Sphinx documentation
+
+if "%SPHINXBUILD%" == "" (
+	set SPHINXBUILD=sphinx-build
+)
+set SOURCEDIR=source
+set BUILDDIR=build
+
+%SPHINXBUILD% >NUL 2>NUL
+if errorlevel 9009 (
+	echo.
+	echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
+	echo.installed, then set the SPHINXBUILD environment variable to point
+	echo.to the full path of the 'sphinx-build' executable. Alternatively you
+	echo.may add the Sphinx directory to PATH.
+	echo.
+	echo.If you don't have Sphinx installed, grab it from
+	echo.https://www.sphinx-doc.org/
+	exit /b 1
+)
+
+if "%1" == "" goto help
+
+%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+goto end
+
+:help
+%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+
+:end
+popd

docs/source/conf.py

@@ -0,0 +1,46 @@
+# Configuration file for the Sphinx documentation builder.
+#
+# For the full list of built-in configuration values, see the documentation:
+# https://www.sphinx-doc.org/en/master/usage/configuration.html
+
+# -- Project information -----------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
+
+project = "pystatis"
+copyright = "2022, Michael Aydinbas"
+authors = [
+    "Michael Aydinbas <michael.aydinbas@gmail.com>",
+    "Ariz Weber <ariz.weber@protonmail.com>",
+    "CorrelAid <info@correlaid.org>",
+    "Daniel Pleus <danielpleus@gmail.com>",
+    "Felix Schmitz <felix.schmitz@philosophy-economics.de>",
+    "Frederik Hering <jobs.fhering@gmail.com>",
+    "Marco Hübner <marco_huebner1@gmx.de>",
+]
+maintainers = ["Michael Aydinbas <michael.aydinbas@gmail.com>"]
+release = "0.1.4"  # hard-coded, since pre-commit hook fails to import pystatis
+version = release
+
+
+# -- General configuration ---------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration
+
+extensions = [
+    "sphinx.ext.autodoc",
+    "sphinx.ext.autosummary",  # used to generate overview tables
+]
+
+templates_path = ["_templates"]
+exclude_patterns = []
+
+
+# -- Options for HTML output -------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output
+
+html_theme = "sphinx_rtd_theme"
+html_static_path = ["_static"]
+
+html_title = "pystatis"
+html_short_title = "pystatis documentation"
+html_logo = "_static/pystatis_logo.png"
+html_favicon = "_static/pystatis_logo.ico"

docs/source/contribute.rst

@@ -0,0 +1,33 @@
+Developer Information
+=====================
+
+How to contribute?
+------------------
+
+Contributions to this project are highly appreciated! You can either contact the maintainers 
+or directly create a pull request for your proposed changes:
+
+1. Fork the Project
+2. Create your Feature Branch (``git checkout -b feature/<descriptive-name>``)
+3. Commit your changes (``git commit -m 'Added NewFeature'``)
+4. Push to remote (``git push origin feature/<descriptive-name>``)
+5. Open a Pull Request to be merged with dev
+
+Developer information
+---------------------
+
+To contribute to this project, please follow these steps:
+
+1. Install `poetry <https://python-poetry.org/docs/>`_. We recommend installing ``poetry`` via `pipx <https://pypa.github.io/pipx/>`_ which gives you a global ``poetry`` command in an isolated virtual environment.
+2. Clone the repository via git.
+3. Change into the project root directory.
+4. Run ``poetry install`` to create the virtual environment within ``poetry``'s cache folder (run ``poetry env info`` to see the details of this new virtual environment). ``poetry`` has installed all dependencies for you, as well as the package itself.
+5. Install pre-commit: ``poetry run pre-commit install``. This will activate the pre-commit hooks that will run prior every commit to ensure code quality.
+6. Do your changes.
+7. Run ``poetry run pytest`` to see if all existing tests still run through. It is important to use ``poetry run`` to call ``pytest`` so that ``poetry`` uses the created virtual environment and not the system's default Python interpreter. Alternatively, you can run ``poetry shell`` to let ``poetry`` activate the virtual environment for the current session. Afterwards, you can run ``pytest`` as usual without any prefix. You can leave the poetry shell with the ``exit`` command.
+8. Add new tests depending on your changes.
+9. Run ``poetry run pytest`` again to make sure your tests are also passed.
+10. Commit and push your changes.
+11. Create a PR.
+
+To learn more about ``poetry``, see `Dependency Management With Python Poetry <https://realpython.com/dependency-management-python-poetry/#command-reference>`_ by realpython.com.

docs/source/index.rst

@@ -0,0 +1,43 @@
+.. pystatis documentation master file, created by
+   sphinx-quickstart on Sun May  7 19:10:45 2023.
+   You can adapt this file completely to your liking, but it should at least
+   contain the root `toctree` directive.
+
+pystatis
+====================================
+
+```pystatis``` is a Python wrapper for the GENESIS web service interface (API). It simplifies accessing the data from the German statistical federal office.
+
+The main features are:
+
+- **Simplified access** to the API. No more need to write cumbersome API calls.
+- **Credential management** removes need to manually add credentials.
+- **Integrated workflow** enables an end-to-end process from finding the relevant data to download it.
+- **Pandas support** instead of manually parsing results.
+- **Caching** to enable productive work despite strict query limits.
+- **Starting and handling background jobs** for datasets that are to big to be downloaded directly from GENESIS.
+
+To learn more about GENESIS refer to the official documentation `here <https://www.destatis.de/EN/Service/OpenData/api-webservice.html>`_.
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Contents
+
+   install
+   start
+   use
+   roadmap
+   contribute
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Modules
+
+   pystatis
+
+Indices and tables
+------------------
+
+* :ref:`genindex`
+* :ref:`modindex`
+* :ref:`search`

docs/source/install.rst

@@ -0,0 +1,16 @@
+Installation
+============
+
+You can install the package via
+
+.. code-block:: bash
+
+    $ pip install pystatis
+
+If everything worked out correctly, you should be able to import ``pystatis`` like this
+
+.. code-block:: python
+
+    import pystatis as pystat
+
+    print("Version:", pystat.__version__)