Skip to content

Commit

Permalink
refactor: migrate from Sqlite to MongoDB (#85)
Browse files Browse the repository at this point in the history 
* feat: migrate data from sqlite to mongo
* feat: migrate cogs
* fix: move add_roll_result_thumb to misc
* refactor: reorganize modules
* test: improve coverage
* fix: pass linters
* ci: add mongo to github ci
* build(deps): bump dependencies
* test: mock call to fakeuser.me
* test: use polyfactory factories for db models
* ci: fix poetry install
* ci: remove python 3.12 and refactor CI workflows
* refactor: move init database to utility function
* fix: prepare for deployment
  • Loading branch information
@natelandau
natelandau authored Nov 3, 2023
1 parent 1349817 commit 7994d1d
Show file tree
Hide file tree
Showing 87 changed files with 7,756 additions and 12,143 deletions.
14 changes: 4 additions & 10 deletions .env
View file
Original file line number Diff line number Diff line change
@@ -1,12 +1,6 @@
# Default configuration values. These are overwritten by docker-compose.yml environment variables
VALENTINA_BACKUP_PATH = "/valentina/backup"
VALENTINA_DAILY_RETENTION = "7" # Number of daily backups to keep
VALENTINA_DB_PATH = "/valentina/valentina.sqlite"
VALENTINA_LOG_FILE = "/valentina/valentina.log"
VALENTINA_LOG_LEVEL_AWS = "INFO" # Log level for AWS S3 client
VALENTINA_LOG_LEVEL_DB = "INFO" # Log level for database SQL queries
VALENTINA_LOG_LEVEL_HTTP = "INFO" # Log level for discord HTTP, gateway, webhook, client events
VALENTINA_LOG_LEVEL = "INFO" # Overall log level for the bot
VALENTINA_MONTHLY_RETENTION = "12" # Number of monthly backups to keep
VALENTINA_WEEKLY_RETENTION = "4" # Number of weekly backups to keep
VALENTINA_YEARLY_RETENTION = "2" # Number of yearly backups to keep
VALENTINA_LOG_LEVEL_AWS = "INFO" # Log level for AWS S3 client
VALENTINA_LOG_LEVEL_DB = "INFO" # Log level for database SQL queries
VALENTINA_LOG_LEVEL_HTTP = "INFO" # Log level for discord HTTP, gateway, webhook, client events
VALENTINA_LOG_LEVEL = "INFO" # Overall log level for the bot
60 changes: 25 additions & 35 deletions .github/actions/setup-poetry/action.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -23,46 +23,36 @@ outputs:
runs:
using: composite
steps:
- name: Get pipx env vars
id: pipx-env-vars
shell: bash
run: |
echo "pipx-home=${PIPX_HOME}" >> $GITHUB_OUTPUT
echo "pipx-bin-dir=${PIPX_BIN_DIR}" >> $GITHUB_OUTPUT
# ----------------------------------------------
# Install python and checkout repository
# ----------------------------------------------

- name: Load pipx cache
# If env vars are not defined do not load cache
if: >
steps.pipx-env-vars.outputs.pipx-home != ''
&& steps.pipx-env-vars.outputs.pipx-bin-dir != ''
id: pipx-cache
uses: actions/cache@v3
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
path: |
${{ steps.pipx-env-vars.outputs.pipx-home }}/venvs/poetry
${{ steps.pipx-env-vars.outputs.pipx-bin-dir }}/poetry
key: ${{ runner.os }}-${{ inputs.python-version }}-pipx-${{ hashFiles('**/poetry.lock') }}
python-version: ${{ matrix.python-version }}

- name: Install poetry
# If env vars are not defined or we missed pipx cache, install poetry
if: >
(
steps.pipx-env-vars.outputs.pipx-home == ''
&& steps.pipx-env-vars.outputs.pipx-bin-dir == ''
)
|| steps.pipx-cache.outputs.cache-hit != 'true'
shell: bash
run: pipx install poetry
# ----------------------------------------------
# Install poetry and project dependencies
# ----------------------------------------------
# poetry config installer.max-workers 1
- name: Install Poetry
uses: snok/install-poetry@v1
with:
virtualenvs-create: true
virtualenvs-in-project: true
installer-parallel: true

- name: Load poetry cache
uses: actions/setup-python@v4
id: setup-python
# Load cached venv if cache exists
- name: Load cached venv
id: cached-poetry-dependencies
uses: actions/cache@v3
with:
python-version: ${{ inputs.python-version }}
cache: poetry
path: .venv
key: venv-${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-${{ hashFiles('**/poetry.lock') }}

- name: Install poetry dependencies
# If we missed poetry cache install dependencies
if: steps.setup-python.outputs.cache-hit != 'true'
# Install dependencies if cache does not exist
- name: Install dependencies
if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'
shell: bash
run: poetry install --all-extras
31 changes: 27 additions & 4 deletions .github/workflows/automated-tests.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
name: "Automated Tests"

on:
workflow_call:
workflow_dispatch:
push:
paths:
Expand All @@ -26,7 +27,9 @@ on:

env:
VALENTINA_LOG_FILE: "/tmp/valentina.log"
VALENTINA_DB_PATH: "/tmp/valentina.sqlite"
VALENTINA_TEST_MONGO_URI: "mongodb://localhost:27017"
VALENTINA_TEST_MONGO_DATABASE_NAME: "valentina-test"
VALENTINA_DB_PATH: "valentina.sqlite" # TODO: Depreciated

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
Expand All @@ -38,7 +41,7 @@ jobs:
strategy:
fail-fast: true
matrix:
python-version: ["3.11", "3.12"]
python-version: ["3.11"]
steps:
- name: Harden Security Runner
uses: step-security/harden-runner@v2
Expand All @@ -48,21 +51,41 @@ jobs:
allowed-endpoints: >
api.github.com:443
api.snapcraft.io:443
auth.docker.io:443
codecov.io:443
files.pythonhosted.org:443
github.com:443
install.python-poetry.org:443
production.cloudflare.docker.com:443
pypi.org:443
python-poetry.org:443
randomuser.me:443
registry-1.docker.io:443
storage.googleapis.com:443
uploader.codecov.io:443
# ----------------------------------------------
# Install python and checkout repository
# ----------------------------------------------

- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}

- name: Checkout repository
uses: actions/checkout@v4

# ----------------------------------------------
# install dependencies
# install Mongo
# ----------------------------------------------

- name: Start MongoDB
uses: supercharge/mongodb-github-action@1.10.0
with:
mongodb-version: "7.0.2"

# ----------------------------------------------
# Install poetry and project dependencies
# ----------------------------------------------

- name: Setup Python and Poetry
Expand Down
12 changes: 11 additions & 1 deletion .github/workflows/create-release.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,6 @@ on:

env:
VALENTINA_LOG_FILE: "/tmp/valentina.log" # pass tests
VALENTINA_DB_PATH: "/tmp/valentina.sqlite" # pass tests

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
Expand Down Expand Up @@ -39,15 +38,26 @@ jobs:
pypi.org:443
python-poetry.org:443
uploads.github.com:443
docker.io:443
githubapp.com:443
docker.com:443
- name: Checkout repository
uses: actions/checkout@v4
with:
fetch-depth: 0

# ----------------------------------------------
# Install poetry and project dependencies
# ----------------------------------------------

- name: Setup Python and Poetry
uses: ./.github/actions/setup-poetry

# ----------------------------------------------
# Grab version number
# ----------------------------------------------

- name: Add version to environment vars
run: |
PROJECT_VERSION=$(poetry version --short)
Expand Down
14 changes: 1 addition & 13 deletions .github/workflows/publish-to-ghcr.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@ env:
REGISTRY: ghcr.io
IMAGE_NAME: ${{ github.repository }}
VALENTINA_LOG_FILE: "/tmp/valentina.log" # Pass tests
VALENTINA_DB_PATH: "/tmp/valentina.sqlite" # Pass tests

jobs:
build-and-push-image:
Expand Down Expand Up @@ -52,17 +51,6 @@ jobs:
- name: Checkout repository
uses: actions/checkout@v4

- name: Setup Python and Poetry
uses: ./.github/actions/setup-poetry

# ----------------------------------------------
# Test before building and uploading the container
# ----------------------------------------------
- name: run poetry checks
run: |
poetry run poetry check
poetry run coverage run
# ----------------------------------------------
# Publish the container
# ----------------------------------------------
Expand All @@ -80,7 +68,7 @@ jobs:
type=semver,pattern={{major}}
- name: Set up QEMU
uses: docker/setup-qemu-action@v2
uses: docker/setup-qemu-action@v3

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
Expand Down
8 changes: 4 additions & 4 deletions .pre-commit-config.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ default_stages: [commit, manual]
fail_fast: true
repos:
- repo: "https://github.com/commitizen-tools/commitizen"
rev: 3.10.1
rev: 3.12.0
hooks:
- id: commitizen
# - id: commitizen-branch
Expand Down Expand Up @@ -59,10 +59,10 @@ repos:
entry: yamllint --strict --config-file .yamllint.yml

- repo: "https://github.com/charliermarsh/ruff-pre-commit"
rev: "v0.0.292"
rev: "v0.1.3"
hooks:
- id: ruff
args: ["--extend-ignore", "I001,D301,D401", "--preview"]
args: ["--extend-ignore", "I001,D301,D401"]
exclude: tests/

- repo: "https://github.com/jendrikseipp/vulture"
Expand All @@ -71,7 +71,7 @@ repos:
- id: vulture

- repo: "https://github.com/crate-ci/typos"
rev: v1.16.19
rev: v1.16.22
hooks:
- id: typos

Expand Down
49 changes: 29 additions & 20 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -68,26 +68,24 @@ Before running Valentina, the following must be configured or installed.

#### Environment Variables

| Variable | Default Value | Usage |
| ------------------------------- | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| VALENTINA_AWS_ACCESS_KEY_ID | | Access key for AWS (_Optional: Only needed for image uploads_) |
| VALENTINA_AWS_SECRET_ACCESS_KEY | | Secret access key for AWS (_Optional: Only needed for image uploads_) |
| VALENTINA_S3_BUCKET_NAME | | Name of the S3 bucket to use (_Optional: Only needed for image uploads_) |
| VALENTINA_BACKUP_PATH | `/valentina/backup` | Sets the directory to store database backups.<br />Note, this is the directory used within the Docker container |
| VALENTINA_DAILY_RETENTION | `7` | Sets the number of days to retain database backups. |
| VALENTINA_DB_PATH | `/valentina/db` | Sets the directory to store the database.<br />Note, this is the directory used within the Docker container |
| VALENTINA_DISCORD_TOKEN | | Sets the Discord bot token. This is required to run the bot. |
| VALENTINA_GUILDS | | Sets the Discord guilds the bot is allowed to join. This is a comma separated list of guild IDs. |
| VALENTINA_LOG_FILE | `/valentina/valentina.log` | Sets the file to write logs to.<br />Note, this is the directory used within the Docker container |
| VALENTINA_LOG_LEVEL | `INFO` | Sets master log level. One of `TRACE`, `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL` |
| VALENTINA_LOG_LEVEL_AWS | `INFO` | Sets the log level for AWS S3. One of `TRACE`, `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL` |
| VALENTINA_LOG_LEVEL_DB | `INFO` | Sets the log level for database SQL queries. One of `TRACE`, `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL` |
| VALENTINA_LOG_LEVEL_HTTP | `INFO` | Sets the log level for discord HTTP, gateway, webhook,client events. One of `TRACE`, `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL` |
| VALENTINA_OWNER_CHANNELS | | Sets the Discord channels that are allowed to run bot admin commands. This is a comma separated list of Discord channel IDs. |
| VALENTINA_MONTHLY_RETENTION | `12` | Sets the number of months to retain database backups. |
| VALENTINA_OWNER_IDS | | Sets the Discord user IDs that are allowed to run bot admin commands. This is a comma separated list of Discord user IDs. |
| VALENTINA_WEEKLY_RETENTION | `4` | Sets the number of weeks to retain database backups. |
| VALENTINA_YEARLY_RETENTION | `2` | Sets the number of years to retain database backups. |
| Variable | Default Value | Usage |
| ---------------------------------- | --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| VALENTINA_AWS_ACCESS_KEY_ID | | Access key for AWS (_Optional: Only needed for image uploads_) |
| VALENTINA_AWS_SECRET_ACCESS_KEY | | Secret access key for AWS (_Optional: Only needed for image uploads_) |
| VALENTINA_S3_BUCKET_NAME | | Name of the S3 bucket to use (_Optional: Only needed for image uploads_) |
| VALENTINA_DISCORD_TOKEN | | Sets the Discord bot token. This is required to run the bot. |
| VALENTINA_GUILDS | | Sets the Discord guilds the bot is allowed to join. This is a comma separated string of guild IDs. |
| VALENTINA_LOG_FILE | `/valentina/valentina.log` | Sets the file to write logs to.<br />Note, this is the directory used within the Docker container |
| VALENTINA_LOG_LEVEL | `INFO` | Sets master log level. One of `TRACE`, `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL` |
| VALENTINA_LOG_LEVEL_AWS | `INFO` | Sets the log level for AWS S3. One of `TRACE`, `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL` |
| VALENTINA_LOG_LEVEL_DB | `INFO` | Sets the log level for database SQL queries. One of `TRACE`, `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL` |
| VALENTINA_LOG_LEVEL_HTTP | `INFO` | Sets the log level for discord HTTP, gateway, webhook,client events. One of `TRACE`, `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL` |
| VALENTINA_OWNER_CHANNELS | | Sets the Discord channels that are allowed to run bot admin commands. This is a comma separated string of Discord channel IDs. |
| VALENTINA_OWNER_IDS | | Sets the Discord user IDs that are allowed to run bot admin commands. This is a comma separated string of Discord user IDs. |
| VALENTINA_MONGO_URI | `mongodb://localhost:27017` | Production MongoDB URI |
| VALENTINA_MONGO_DATABASE_NAME | `valentina` | Production Database name |
| VALENTINA_TEST_MONGO_URI | `mongodb://localhost:27017` | URI for a MongoDB used in automated tests URI |
| VALENTINA_TEST_MONGO_DATABASE_NAME | `valentina-test` | Name of the database used in automated tests |

---

Expand Down Expand Up @@ -131,6 +129,17 @@ There are two ways to contribute to this project.
- Run `poetry remove {package}` from within the development environment to uninstall a runtime dependency and remove it from `pyproject.toml` and `poetry.lock`.
- Run `poetry update` from within the development environment to upgrade all dependencies to the latest versions allowed by `pyproject.toml`.

## Testing MongoDB locally

To run the tests associated with the MongoDB database, you must have MongoDB installed locally. The easiest way to do this is with Docker. Set two additional environment variables to allow the tests to connect to the local MongoDB instance.

| Variable | Default Value | Usage |
| -------------------------- | --------------------------- | -------------------------------------------- |
| VALENTINA_TEST_MONGODB_URI | `mongodb://localhost:27017` | URI to the MongoDB instance used for testing |
| VALENTINA_TEST_MONGODB_DB | `valentina-test-db` | Name of the database used for testing |

NOTE: Github CI integrations will ignore these variables and run the tests against a Mongo instance within the workflows.

## Troubleshooting

If connecting to Discord with the bot fails due to a certificate error, run `scripts/install_certifi.py` to install the latest certificate bundle.
Loading

0 comments on commit 7994d1d

Please sign in to comment.