[IO-557][internal] Move to use Poetry for packaging and publishing (#512

)

# Move to use [Poetry](https://python-poetry.org/) for packaging and publishing

[Poetry](https://python-poetry.org/) is a dependency manager tool, that creates dev environments and installs requisite dependencies.   It's also [PEP-518](https://peps.python.org/pep-0518/) compliant, which is no longer optional after the current default version of `pip`, so moving to [PEP-518](https://peps.python.org/pep-0518/) compliance was not optional if we wish to carry on publishing on [pypi.org].  Further details in #512.

# Changelog message

 Changed our build system from old style `setup.py` to `PEP-518` style, based on `pyproject.toml`.  Incorporated use of Poetry as a virtual environment and dependency manager, as this is already meeting these requirements.

.github/ISSUE_TEMPLATE

@@ -0,0 +1,40 @@
+name: Bug Report
+description: File a bug report
+title: "[Bug]: "
+labels: ["bug", "triage"]
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Thanks for taking the time to fill out this bug report!
+  - type: input
+    id: contact
+    attributes:
+      label: Contact Details
+      description: How can we get in touch with you if we need more info?
+      placeholder: ex. email@example.com
+    validations:
+      required: false
+  - type: textarea
+    id: what-happened
+    attributes:
+      label: What happened?
+      description: Also tell us, what did you expect to happen?
+      placeholder: Tell us what you see!
+    validations:
+      required: true
+  - type: textbox
+    id: version
+    attributes:
+      label: What version of darwin-py were you using?
+      description: Run darwin-py --version at command line and paste the output here
+    validations:
+      required: true
+  - type: checkboxes
+    id: terms
+    attributes:
+      label: Code of Conduct
+      description: By submitting this issue, you agree to follow our [Code of Conduct](https://example.com)
+      options:
+        - label: I agree to follow this project's Code of Conduct
+          required: true

.github/workflows/document.yml

@@ -2,7 +2,7 @@ name: Documentation
 
 on:
   push:
-    branches: ['master']
+    branches: ["master"]
   workflow_dispatch:
 
 jobs:
@@ -20,8 +20,11 @@ jobs:
       - name: Install dependencies
         run: |
           python -m pip install --upgrade pip
-          pip install setuptools wheel
-          pip install --editable ".[test,medical]"
+          pip install poetry
+          poetry install --all-extras --no-interaction --no-root
+          pip install wheel
+          pip install --upgrade setuptools
+          pip install --editable ".[test,ml,medical,dev]"
           pip install torch torchvision
           pip install -U sphinx
           # Locking mistune version so m2r works. More info on issue:
@@ -35,7 +38,7 @@ jobs:
           m2r README.md
           mv README.rst source/
       - name: Generate new docs
-        env: 
+        env:
           PYTHONPATH: "."
         run: |
           rm -rf docs/* 

.github/workflows/pythonpublish_sdist.yml

@@ -14,14 +14,18 @@ jobs:
         uses: actions/setup-python@v2
         with:
           python-version: "3.x"
+      - run: pip install pip --upgrade
+      - name: Setup Poetry
+        uses: abatilo/actions-poetry@v2
+        with:
+          poetry-version: "1.3.1"
       - name: Install dependencies
         run: |
-          python -m pip install --upgrade pip
-          pip install setuptools wheel twine
-          python setup.py sdist
+          poetry install --no-interaction --no-root --all-extras -vvv
+          poetry build
       - name: Publish
         env:
-          TWINE_USERNAME: ${{ secrets.PYPI_USERNAME }}
-          TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }}
+          POETRY_HTTP_BASIC_PYPI_USERNAME: ${{ secrets.PYPI_USERNAME }}
+          POETRY_HTTP_BASIC_PYPI_PASSWORD: ${{ secrets.PYPI_PASSWORD }}
         run: |
-          twine upload dist/*
+          poetry publish

.github/workflows/tests.yml

@@ -1,49 +1,40 @@
 name: Testing
 
-on: [push]
+on:
+  push:
+    branches:
+      - master
+  pull_request:
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
+  cancel-in-progress: true
 
 jobs:
-  ubuntu:
-    runs-on: ubuntu-latest
-    timeout-minutes: 30
+  ci:
     strategy:
+      fail-fast: false
       matrix:
-        python-version: [3.7, 3.8, 3.9, "3.10"]
-
+        python-version: ["3.8", "3.9", "3.10"]
+        poetry-version: ["1.3.1"]
+        os: [ubuntu-latest, windows-latest]
+    runs-on: ${{ matrix.os }}
     steps:
       - uses: actions/checkout@v2
-      - name: Set up Python ${{ matrix.python-version }}
-        uses: actions/setup-python@v2
+      - uses: actions/setup-python@v2
         with:
           python-version: ${{ matrix.python-version }}
-      - name: Install dependencies
-        run: |
-          python -m pip install --upgrade pip
-          pip install wheel
-          pip install --upgrade setuptools
-          pip install --editable ".[test,ml,medical]"
-
-      - name: Run tests
-        run: pytest
-  windows:
-    runs-on: windows-latest
-    timeout-minutes: 30
-    strategy:
-      matrix:
-        python-version: [3.6, 3.7, 3.8, 3.9, "3.10"]
-
-    steps:
-      - uses: actions/checkout@v2
-      - name: Set up Python ${{ matrix.python-version }}
-        uses: actions/setup-python@v2
+      - run: pip install pip --upgrade
+      - name: Setup Poetry
+        uses: abatilo/actions-poetry@v2
         with:
-          python-version: ${{ matrix.python-version }}
+          poetry-version: ${{ matrix.poetry-version }}
       - name: Install dependencies
         run: |
-          python -m pip install --upgrade pip
+          poetry install --no-interaction --no-root --all-extras -vvv
           pip install wheel
           pip install --upgrade setuptools
-          pip install --editable ".[test,ml,medical]"
-
-      - name: Run tests
+          pip install --editable ".[test,ml,medical,dev]"
+          pip install pytest pytest-describe
+      - name: Run Tests
         run: pytest

.python-version

@@ -0,0 +1 @@
+3.10

docs/README.md

@@ -0,0 +1,73 @@
+# Developing for `darwin-py`
+
+`darwin-py` uses a `pyproject.toml` file, and manages it with `poetry`
+
+- [Developing for `darwin-py`](#developing-for-darwin-py)
+  - [Development environment](#development-environment)
+  - [Basic poetry commands](#basic-poetry-commands)
+    - [Add a package to general dependencies:](#add-a-package-to-general-dependencies)
+    - [Add a package to one of the extras groups](#add-a-package-to-one-of-the-extras-groups)
+    - [Add a package to the poetry dev dependencies](#add-a-package-to-the-poetry-dev-dependencies)
+  - [The `pyproject.toml` file](#the-pyprojecttoml-file)
+
+
+## Development environment
+
+You can either:
+* Install a `poetry` environment, and work with that, or
+* Install `darwin-py` using `pip` from Pypi.org, and make your own arrangements for debugging.
+
+The recommended setup is to install poetry, and use this to setup a dev environment:
+
+```sh
+$ python --version  # must be 3.9 or greater for development tools
+$ pip install poetry
+$ poetry install --extras "test ml medical dev" -G dev
+```
+
+This creates a `virtualenv` for the project.  If you prefer, you can use `venv` by running this command before `poetry install`:
+
+```sh
+$ python -m venv .venv
+```
+
+You can give it a different name if you wish, but the name `.venv` is pre-ignored in the `.gitignore` file.
+
+
+## Basic poetry commands
+
+### Add a package to general dependencies:
+```sh
+$ poetry add [PACKAGENAME]
+```
+
+
+### Add a package to one of the extras groups
+
+(these are those that are used in `pip install darwinpy[option]` type commands - ours are `test`, `ml`, `medical`, and `dev` - `dev` is for developers who don't want to use poetry - so they can install using `pip`)
+
+```sh
+$ poetry add [PACKAGENAME] --extras="extra group names"
+```
+
+
+Packages added this way will be installable with `pip` once a release has been made to PyPi.
+
+
+### Add a package to the poetry dev dependencies
+
+```sh
+$ poetry add [PACKAGENAME] -G groupname
+```
+
+Currently the only group is `dev` and contains dev tools for developers using `poetry`.  `poetry` groups have no effect on packaging, and things installed here will not be packaged.
+
+## The `pyproject.toml` file
+
+This was a necessary change for PEP-517, but also allows us a central place to add configuration.
+
+Items in the `pyproject.toml` file that are grouped as `[tool.X]` are configuration settings for specific tools on the system.
+
+For example, settings for `mypy` static type analyser, are in the `[tool.mypy]` group.
+
+A plugin is installed that allows `flake8` to take its configuration from this file, even though usually it only works with `setup.cfg` and `.flake8`.