Merge pull request #421 from ImperialCollegeLondon/documentation

Upgrade of the documentation

CONTRIBUTING.md

@@ -21,13 +21,6 @@ The following is a set of guidelines for contributing to Paricia, a Python-based
 - [Git Commit Messages](#git-commit-messages)
 - [Documentation Styleguide](#documentation-styleguide)
 
-[Developer's setup](#developers-setup)
-
-- [Installing Paricia](#installing-paricia)
-- [Tests](#tests)
-- [Synthetic data](#synthetic-data)
-- [Continuous integration](#continuous-integration)
-
 ## Code of Conduct
 
 This project and everyone participating in it is governed by the [Paricia Code of Conduct](CODE_OF_CONDUCT.md). By participating, you are expected to uphold this code. Please report unacceptable behavior to [the repository Administrator](https://www.imperial.ac.uk/people/w.buytaert).
@@ -71,7 +64,7 @@ Include details about your configuration and environment:
 
 ### Suggesting Enhancements
 
-This section guides you through submitting an enhancement suggestion for Paricia, including completely new features and minor improvements to existing functionality. Following these guidelines helps maintainers and the community understand your suggestion :pencil: and find related suggestions :mag_right:.
+This section guides you through submitting an enhancement suggestion for Paricia, including completely new features and minor improvements to existing functionality. Following these guidelines helps maintainers and the community understand your suggestion and find related ones.
 
 Before creating enhancement suggestions, please check [this list](https://github.com/ImperialCollegeLondon/paricia/issues) (including closed issues) as you might find out that you don't need to create one. When you are creating an enhancement suggestion, please [include as many details as possible](#how-do-i-submit-a-good-enhancement-suggestion).
 
@@ -145,53 +138,3 @@ While the prerequisites above must be satisfied prior to having your pull reques
   - Reference classes with `{ClassName}`
   - Reference instance methods with `{ClassName::methodName}`
   - Reference class methods with `{ClassName.methodName}`
-
-## Developer's setup
-
-### Installing Paricia
-
-If you want to install Paricia from scratch, follow the steps below:
-
-- Run `docker-compose up --build` (requires [Docker](https://www.docker.com/) to be running)
-- After downloading and building the images, Paricia should now be available via a web browser in `http://localhost:8000/`.
-- Create **admin** user running `python manage.py createsuperuser`.
-- If you want to load initial data (variables, units, stations...):
-  - In a separate terminal run `docker exec -it <name_of_docker_container> bash` e.g. `docker exec -it paricia-web-1 bash` to start a bash session in the container. You can find the name of the container in the Docker Desktop GUI, or by running `docker container ls`.
-  - Run `python manage.py shell < utilities/load_initial_data.py`.
-
-## Project structure
-
-In addition to the applications containing the actual functionality, and described in the documentation, the project file structure has other directories that are of interest only for developers.
-
-- The top-level directory contains various config files and directories for git, github, docker and pip.
-- Each django app is in a subdirectory and `djangomain` contains the main django settings, views and urls.
-- The `static` directory contains the static files for the project.
-- The `templates` directory contains the templates for the project.
-- The `utilities` directory contains helper functions for the project.
-- The `tests` directory contains all unit tests for the project.
-
-### Tests
-
-The tests are run with `python manage.py test` from inside the docker container.
-
-For that to work, development-related dependencies needs to be installed. To do that, get into the container (see instructions at the top) and run:
-
-```bash
-python -m pip install -r requirements-dev.txt
-```
-
-### Synthetic data
-
-Synthetic data can be added to the database for benchmarking purposes using one of the scenarios in `utilities/benchmarking` or creating one of your own. To do so:
-
-- Populate the database with some initial data for the `Station`, `Variable` and all the required models (see the *Getting Started* section).
-- Install the development dependencies (read the *Tests* section)
-- Run your desired synthetic data scenario.
-
-If you run one of the built in ones, you should see a progressbar for the process and, if you log in into the Django Admin of Paricia (`http://localhost:8000/admin`), then you will see the records for the `Measurements` model increasing.
-
-### Continuous integration
-
-Pre-commit hooks are set up to run code quality checks (isort and black) before committing. To run these locally, you will need to `pip install pre-commit` then `pre-commit install`. Now, quality assurance tools will be run automatically with every commit.
-
-Github workflows are set up to run the pre-commit actions and the tests automatically on every push action.

docs/Applications/index.md

@@ -2,11 +2,23 @@
 
 All functionality in Parcia is contained in several Django applications, each of them, in turn, including one or more database models that define that functionality.
 
-These pages describe in more detail all these models, what they are for and how they relate to each other. **All registered users can create new models for these applications via the Admin pages**.
-
 - [**Formatting**](formatting.md): Definitions of the different file formats that can be imported, including specifics around delimiters, headers etc.
 - [**Variable:**](variable.md) Information about measured variables including units, max/min allowed values etc.
 - [**Sensor:**](sensor.md) Information on physical sensors including brand and type.
 - [**Station:**](station.md) Everything to do with physical stations including their location, region, ecosystem etc.
 - [**Importing:**](importing.md) Entries are created in this app when datasets are imported, storing information on the the raw data file itself, the user, time of import etc.
 - [**Measurement:**](measurement.md) The actual time-series data is stored here when raw data files are imported.
+
+The objects for all of these apps, except for **Measurements**, can be managed by registered users via the corresponding forms in the front end, and for superusers also via the Admin pages.
+
+## Other utilities
+
+In addition to the applications containing the actual functionality, the project file structure has other directories that are of interest only for developers.
+
+- The top-level directory contains various config files and directories for git, github, docker and pip.
+- Each django app is in a subdirectory and `djangomain` contains the main django settings, views and urls.
+- The **Management** app contains the custom user model, base permission classes and base views, used by all other apps to save boilerplate code.
+- The `static` directory contains the static files for the project.
+- The `templates` directory contains the templates for the project, used to render the different views in the browser.
+- The `utilities` directory contains helper functions for the project.
+- The `tests` directory contains all unit tests for the project.

docs/adding_elements.md

@@ -0,0 +1,32 @@
+# Adding elements
+
+Registered users can either use existing elements - formats, variables, etc. - as long as these are public (see the [Permissions](./permissions.md) section), but they can also create their own to suit their specific needs.
+
+All elements that can be created in Paricia - except for the data import, which is discussed in [its own section](./importing_data.md) - follow a similar workflow:
+
+- Choose the element of interest from the submenu in the top bar, e.g. `Variable` within the `Variables` menu.
+
+![Selecting a element from the top menu](images/selecting_component.png)
+
+- The page now displays the list of existing elements of that type that the user can view - i.e., those that are public or that are their own. You can sort the entries clicking on the column names or filter them to select just some entries.
+
+![List of variables a user can view](images/variables_list.png)
+
+- Clicking on an existing element ID allows to view the details of that element and to edit it, if the user has permission to do so.
+- Clicking on the `New` button at the top allows to create a new element of that type.
+
+A new form will open with the fields that need to be completed for that element.
+
+Some elements are very simple and have just one or two fields to complete. Others are more complicated and link, in turn, to other elements. Not all fields are mandatory, in general. If a mandatory field is not filled, it will be flagged when trying to save the element.
+
+Let's take the variables creation form as an example.
+
+![Form used to create a new variable](images/variable_creation.png)
+
+As we can see in this form, there is a field called `Visibility`. All elements have this field and it defines who else can see the details and use the element to define their own elements.
+
+Other fields, like `Unit`, are foreign keys to other elements and, in this case, are just informative - metadata to better understand the variable.
+
+Finally, some fields are used during the validation or import process. That is the case of the maximum, minimum or difference error, in this case, which helps Paricia to identify and flag suspicious entries.
+
+When creating a new element, is important that the meaning and purpose of the fields are properly understood. They all should have a description underneath explaining what they are for, but if that information is not complete or clear, please report it following the instructions in the [Contributing guidelines](./contributing.md).

docs/admin.md

@@ -8,7 +8,7 @@ There are two ways of becoming an Admin user
 
 ![Checking the third box grants the user all Paricia permissions](images/superuser.png)
 
-2. Via the command line. This is a more advanced method and typically required only when setting up Paricia for the first time, either locally for development or in a new server. We will assume that that Paricia has been launched using `docker compose`, as instructed in the [contributing guidelines](./contributing.md#installing-paricia). The steps in this case are:
+2. Via the command line. This is a more advanced method and typically required only when setting up Paricia for the first time, either locally for development or in a new server. We will assume that that Paricia has been launched using `docker compose`, as instructed in the [installation instructions](./installation.md#docker-deployment). The steps in this case are:
 
     1. Open an terminal and access the server, if not for local development, via SSH or other method.
     2. Find the name of the container running the `paricia` image executing `docker ps`. It should be something like `paricia-web-1` or `paricia-app-1`.

docs/importing_data.md

@@ -1,5 +1,7 @@
 # Importing Data
 
+The whole reason for Paricia to exist is to store and facilitate access to hydrology data. Therefore, one of its main components, that depends on everything else, is the data ingestion process.
+
 ## Submit the data import
 
 Data import is done via Paricia import listing, clicking in the `New` button at the top of the page.
@@ -24,7 +26,7 @@ Clicking in each classification `id` will show you more information about that p
 
 ### Station
 
-For the Station, the user will only be able to choose those for which they have `change` permission. For the Format, the will be able to choose their own formats and those labelled as `public`.
+For the Station, the user will only be able to choose those for which they have `change` permission. For the Format, they will be able to choose their own formats and those labelled as `public`.
 
 The station needs to be complete, i.e. it needs to have all the required fields filled, something that might not be the case if the station was imported into Paricia. A usual field missing is the `timezone` if that were the case, you will be notified when trying to save the data import. To fix it, just go to the station page - `Station -> Station` in the top menu - and update the fields that are missing.
 
@@ -40,7 +42,7 @@ Once the form is complete, click `Save` at the top of the page and the import pr
 
 ![Data ingestion queued](images/importing_queued.png)
 
-- **Completed**: The data ingestion has completed successfully. Information on the start and end dates of the data, as well as the number of records, will appear updated
+- **Completed**: The data ingestion has completed successfully. Information on the start and end dates of the data **in the local timezone of the user**, as well as the number of records, will appear updated
 
 ![Data ingestion completed](images/importing_completed.png)
 

docs/installation.md

@@ -0,0 +1,89 @@
+# Installation
+
+There are two basic setups that can be put in place in order to develop and test Paricia locally:
+
+- Virtual environment, used for daily development of the code and the documentation. See section on the [virtual environment setup](#virtual-environment).
+- Docker, used to run the tool locally, accessible in the browser as well as to run tests. See section on [docker](#docker-deployment).
+
+## Virtual environment
+
+The normal software development should be done within a virtual environment, where the tools and all the dependencies that Paricia requires are installed. This enables to run the appropriate linters, code formatters, and autocompletion features of the code editor specifically for Paricia. Additionally, it also let you create and develop the documentation.
+
+To setup a virtual environment with all the requirements, navigate to Paricia's root directory in a terminal and run (this should work in all platforms):
+
+```bash
+python -m venv .venv
+```
+
+This will create an isolated Python environment within a directory called `.venv`. Notice that Paricia requires Python 3.11 or higher to work. Once the environment has been created, you can activate it with:
+
+```ps
+.venv\Scripts\activate
+```
+
+in Powershell, or
+
+```bash
+. .venv/Scripts/activate
+```
+
+in Bash. Note the extra `.` and space before `.venv` in this case.
+
+Once in the virtual environment, dependencies for development (linters, formatter, etc.) and documentation, respectively, can be installed with:
+
+```
+python -m pip install -r requirements-dev.txt
+python -m pip install -r requirements-doc.txt
+```
+
+That should be it. The virtual environment should be ready for the development of Paricia and its documentation. Just indicate your code editor which environment you are using in case it does not pick it automatically.
+
+!!! warning "Running Paricia and tests"
+
+    You will not be able to run Paricia itself or the tests from the virtual environment as a TimescaleDB is required for that, which we have not installed. See the [docker deployment](#docker-deployment) section to learn how to do that.
+
+## Docker deployment
+
+Paricia developer's setup requires the use of `docker` to easily manage the different services it is made of, namely the web application itself and the database and make the tool accessible from the web browser. It is also necessary to run the tests.
+
+The steps to setup your system in this case are:
+
+- Install [Docker](https://www.docker.com/)
+- In a terminal, run `docker-compose up --build`. This will pull the docker images from the internet, build the local ones and launch the services. Depending on your internet connection, it might take a few minutes to complete.
+- After downloading and building the images, Paricia should now be available via a web browser in `http://localhost:8000/`.
+- Create **admin** user following the command line instructions described in the [Paricia administrator section](./admin.md#paricia-administrator).
+
+If you want to load initial data (variables, units, stations...):
+
+- In a separate terminal run `docker exec -it <name_of_docker_container> bash` e.g. `docker exec -it paricia-web-1 bash` to start a bash session in the container. You can find the name of the container in the Docker Desktop GUI, or by running `docker ps`.
+- Run `python manage.py shell < utilities/load_initial_data.py`.
+
+### Running Paricia after the initial installation
+
+Once the initial setup is done, you can:
+
+- Stop the containers with `docker compose down`
+- Re-launch the containers with `docker compose up`. No need to run with the `--build` flag unless some dependency has changed.
+- If you want to use the same terminal, you can run the services in detached mode with `docker compose up -d`.
+
+Unless you destroy the docker volume containing the database or manually flush it, the database will persist between subsequent calls to docker compose.
+
+### Building the documentation
+
+The documentation uses [`mkdocs`](https://www.mkdocs.org/). This should have been installed alongside all the other doc-related dependencies if you run `python -m pip install -r requirements-doc.txt`, as described above. There's no need to use `docker` to build the documentation locally.
+
+To test the documentation live and have it rebuilt automatically while you edit the documentation files, run:
+
+```
+mkdocs serve -a localhost:8001
+```
+
+The reason for explicitly using `localhost:8001` is because port `8000`, the default, will likely be already in use by Paricia web application.
+
+To build the documentation as standalone html files and related resources, run instead:
+
+```
+mkdocs build
+```
+
+A new directory in the root of the project called `site` would have been created with all the files instead. Open `index.html` in the browser to check this documentation.

docs/introduction.md

@@ -0,0 +1,7 @@
+# Introduction
+
+This guide will help users of Paricia to get up to speed with its functionality and start making use of its features.
+
+There are two types of Paricia users, those who are interested in exploring the data available - either registered users or not - and those who manage stations and want to create new entries in the database. The first group can just check the [Reporting](./reports.md) documentation, as it contains all the information they will need. The others should become familiar with all the pages of this user guide.
+
+It should not be necessary to check the Developers documentation, but it might be helpful, occasionally, to have a look at the [Contributing guidelines](./contributing.md) if you spot something that does not work as expected and you want to report it, and at the [Applications page](./Applications/index.md) where there are mode detailed descriptions of the models that make up Paricia.

docs/permissions.md

@@ -7,7 +7,7 @@ The following principles apply:
 - **Anonymous users** (non-registered users) can view data in the reporting tool of stations that are labelled as **public**. These stations also appear in the map of the front page.
 - **Registered users** can:
     - View data of stations owned by other users and labelled as **public** or **internal**, as well as their own data.
-    - Create new elements, like formats, sensors, stations, etc. via the Admin interface. These elements can depend on other public objects or private objects owned by the user.
+    - Create new elements, like formats, sensors, stations, etc. These elements can depend on other public objects or private objects owned by the user.
     - Upload new data to stations for which they have `change` permission (this includes stations they own).
     - Validate data associated to stations for which they have `change` permission (this includes stations they own).
 - [**Admin users**](./admin.md) can:
@@ -28,3 +28,7 @@ The **visibility** attribute of all objects in the database controls if the obje
    - They have another visibility level, `internal` which allows for the station data to be **visible to registered users only**.
 
 The visibility of new objects always defaults to **private**.
+
+!!! warning "Stations `change` permission"
+
+    Only Admin users can give `change` permission for a station to another user. This is done via de Admin page for that station.