KARL

App: https://karl.qanta.org/ HSQuizbowl forum post: https://hsquizbowl.org/forums/viewtopic.php?f=123&p=379140&sid=8ae602e914bc1e56736a07030176c718

Development

Restrictions

Access to a hosted scheduler for development is restricted. The current workaround is to disable calls to the scheduler during development. When the scheduler portion of KAR³L is open-sourced, a self-hosted version of the scheduler can be hosted locally the port specified by the INTERFACE variable in the .env file.

Env Files

The .env file is the one that contains all your configurations, generated keys and passwords, etc.

An .env file must in the root folder and in frontend/
Use openssl rand -hex 32 to replace [redacted] slots

Example Root `.env` File

DOMAIN=localhost

STACK_NAME=karl

TRAEFIK_PUBLIC_NETWORK=traefik-public
TRAEFIK_TAG=karl
TRAEFIK_PUBLIC_TAG=traefik-public

DOCKER_IMAGE_BACKEND=karl/backend
DOCKER_IMAGE_CELERYWORKER=karl/celeryworker
DOCKER_IMAGE_FRONTEND=karl/frontend

# Backend
BACKEND_CORS_ORIGINS=["http://localhost", "http://localhost:4200", "http://localhost:3000", "http://localhost:8080", "http://localhost:8081", "http://localhost:8082", "https://localhost", "https://localhost:4200", "https://localhost:3000", "https://localhost:8080", "http://dev.test", "https://test", "https://test", "http://local.dockertoolbox.tiangolo.com", "http://localhost.tiangolo.com", "http://dev.karl.qanta.org", "https://stag.karl.qanta.org", "https://karl.qanta.org"]
PROJECT_NAME=KAR³L
SECRET_KEY=[redacted]
FIRST_SUPERUSER=learningwithkarl@gmail.com
FIRST_SUPERUSER_PASSWORD=[redacted]
SMTP_TLS=True
SMTP_PORT=587
SMTP_HOST=smtp.gmail.com
SMTP_USER=[email]
SMTP_PASSWORD=[redacted]

USERS_OPEN_REGISTRATION=True

SENTRY_DSN=
INTERFACE=http://host.docker.internal:4000/

# Flower
FLOWER_BASIC_AUTH=root:[redacted]

# Postgres
POSTGRES_SERVER=db
POSTGRES_USER=postgres
POSTGRES_PASSWORD=[redacted]
POSTGRES_DB=app

# PgAdmin
PGADMIN_LISTEN_PORT=5050
PGADMIN_DEFAULT_EMAIL=[email]
PGADMIN_DEFAULT_PASSWORD=[redacted]

Example `Frontend/` `.env` File

VUE_APP_DOMAIN_DEV=localhost
# VUE_APP_DOMAIN_DEV=local.dockertoolbox.tiangolo.com
# VUE_APP_DOMAIN_DEV=localhost.tiangolo.com
# VUE_APP_DOMAIN_DEV=dev.karl.qanta.org
VUE_APP_DOMAIN_STAG=stag.karl.qanta.org
VUE_APP_DOMAIN_PROD=karl.qanta.org
VUE_APP_NAME=KAR³L
VUE_APP_ENV=development
# VUE_APP_ENV=staging
# VUE_APP_ENV=production

Backend Requirements

Docker.
Docker Compose.
Poetry for Python package and environment management.

Frontend Requirements

Node.js (with npm).

Backend local development

Create an .env file
Start the stack with Docker Compose:

docker compose up -d

Enter the backend container (the default configuration in this project makes use of features documented in detail in the Docker Compose Override section, go there for more details.)

docker compose exec backend bash

Inside the container, run:

bash /start-reload.sh

Now you can open your browser and interact with these URLs:

Frontend, built with Docker, with routes handled based on the path: http://localhost. If you make frontend changes, this will not be reflected unless you rebuild your stack with docker compose up -d --build

Backend, JSON based web API based on OpenAPI: http://localhost/api/

Automatic interactive documentation with Swagger UI (from the OpenAPI backend): http://localhost/docs

Alternative automatic documentation with ReDoc (from the OpenAPI backend): http://localhost/redoc

PGAdmin, PostgreSQL web administration: http://localhost:5050

Flower, administration of Celery tasks: http://localhost:5555

Traefik UI, to see how the routes are being handled by the proxy: http://localhost:8090

Note: The first time you start your stack, it might take a minute for it to be ready. While the backend waits for the database to be ready and configures everything. You can check the logs to monitor it.

To check the logs, run:

docker compose logs

To check the logs of a specific service, add the name of the service, e.g.:

docker compose logs backend

If your Docker is not running in localhost (the URLs above wouldn't work) check the sections below on Development with Docker Toolbox and Development with a custom IP.

Backend local development, additional details

General workflow

By default, the dependencies are managed with Poetry, go there and install it.

From ./backend/app/ you can install all the dependencies with:

$ poetry install

Then you can start a shell session with the new environment with:

$ poetry shell

Next, open your editor at ./backend/app/ (instead of the project root: ./), so that you see an ./app/ directory with your code inside. That way, your editor will be able to find all the imports, etc. Make sure your editor uses the environment you just created with Poetry.

Modify or add SQLAlchemy models in ./backend/app/app/models/, Pydantic schemas in ./backend/app/app/schemas/, API endpoints in ./backend/app/app/api/, CRUD (Create, Read, Update, Delete) utils in ./backend/app/app/crud/. The easiest might be to copy the ones for Items (models, endpoints, and CRUD utils) and update them to your needs.

Add and modify tasks to the Celery worker in ./backend/app/app/worker.py.

If you need to install any additional package to the worker, add it to the file ./backend/app/celeryworker.dockerfile.

Docker Compose Override

During development, you can change Docker Compose settings that will only affect the local development environment, in the file docker-compose.override.yml.

The changes to that file only affect the local development environment, not the production environment. So, you can add "temporary" changes that help the development workflow.

For example, the directory with the backend code is mounted as a Docker "host volume", mapping the code you change live to the directory inside the container. That allows you to test your changes right away, without having to build the Docker image again. It should only be done during development, for production, you should build the Docker image with a recent version of the backend code. But during development, it allows you to iterate very fast.

There is also a command override that runs /start-reload.sh (included in the base image) instead of the default /start.sh (also included in the base image). It starts a single server process (instead of multiple, as would be for production) and reloads the process whenever the code changes. Have in mind that if you have a syntax error and save the Python file, it will break and exit, and the container will stop. After that, you can restart the container by fixing the error and running again:

$ docker compose up -d

There is also a commented out command override, you can uncomment it and comment the default one. It makes the backend container run a process that does "nothing", but keeps the container alive. That allows you to get inside your running container and execute commands inside, for example a Python interpreter to test installed dependencies, or start the development server that reloads when it detects changes, or start a Jupyter Notebook session.

To get inside the container with a bash session you can start the stack with:

$ docker compose up -d

and then exec inside the running container:

$ docker compose exec backend bash

You should see an output like:

root@7f2607af31c3:/app#

that means that you are in a bash session inside your container, as a root user, under the /app directory.

There you can use the script /start-reload.sh to run the debug live reloading server. You can run that script from inside the container with:

$ bash /start-reload.sh

...it will look like:

root@7f2607af31c3:/app# bash /start-reload.sh

and then hit enter. That runs the live reloading server that auto reloads when it detects code changes.

Nevertheless, if it doesn't detect a change but a syntax error, it will just stop with an error. But as the container is still alive and you are in a Bash session, you can quickly restart it after fixing the error, running the same command ("up arrow" and "Enter").

...this previous detail is what makes it useful to have the container alive doing nothing and then, in a Bash session, make it run the live reload server.

Backend tests

To test the backend run:

$ DOMAIN=backend sh ./scripts/test.sh

The file ./scripts/test.sh has the commands to generate a testing docker-stack.yml file, start the stack and test it.

The tests run with Pytest, modify and add tests to ./backend/app/app/tests/.

If you use GitLab CI the tests will run automatically.

Local tests

Start the stack with this command:

DOMAIN=backend sh ./scripts/test-local.sh

The ./backend/app directory is mounted as a "host volume" inside the docker container (set in the file docker-compose.dev.volumes.yml). You can rerun the test on live code:

docker compose exec backend /app/tests-start.sh

Test running stack

If your stack is already up and you just want to run the tests, you can use:

docker compose exec backend /app/tests-start.sh

That /app/tests-start.sh script just calls pytest after making sure that the rest of the stack is running. If you need to pass extra arguments to pytest, you can pass them to that command and they will be forwarded.

For example, to stop on first error:

docker compose exec backend bash /app/tests-start.sh -x

Test Coverage

Because the test scripts forward arguments to pytest, you can enable test coverage HTML report generation by passing --cov-report=html.

To run the local tests with coverage HTML reports:

DOMAIN=backend sh ./scripts/test-local.sh --cov-report=html

To run the tests in a running stack with coverage HTML reports:

docker compose exec backend bash /app/tests-start.sh --cov-report=html

Live development with Python Jupyter Notebooks

If you know about Python Jupyter Notebooks, you can take advantage of them during local development.

The docker-compose.override.yml file sends a variable env with a value dev to the build process of the Docker image (during local development) and the Dockerfile has steps to then install and configure Jupyter inside your Docker container.

So, you can enter into the running Docker container:

docker compose exec backend bash

And use the environment variable $JUPYTER to run a Jupyter Notebook with everything configured to listen on the public port (so that you can use it from your browser).

It will output something like:

root@73e0ec1f1ae6:/app# $JUPYTER
[I 12:02:09.975 NotebookApp] Writing notebook server cookie secret to /root/.local/share/jupyter/runtime/notebook_cookie_secret
[I 12:02:10.317 NotebookApp] Serving notebooks from local directory: /app
[I 12:02:10.317 NotebookApp] The Jupyter Notebook is running at:
[I 12:02:10.317 NotebookApp] http://(73e0ec1f1ae6 or 127.0.0.1):8888/?token=f20939a41524d021fbfc62b31be8ea4dd9232913476f4397
[I 12:02:10.317 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[W 12:02:10.317 NotebookApp] No web browser found: could not locate runnable browser.
[C 12:02:10.317 NotebookApp]

    Copy/paste this URL into your browser when you connect for the first time,
    to login with a token:
        http://(73e0ec1f1ae6 or 127.0.0.1):8888/?token=f20939a41524d021fbfc62b31be8ea4dd9232913476f4397

you can copy that URL and modify the "host" to be localhost or the domain you are using for development (e.g. local.dockertoolbox.tiangolo.com), in the case above, it would be, e.g.:

http://localhost:8888/token=f20939a41524d021fbfc62b31be8ea4dd9232913476f4397

and then open it in your browser.

You will have a full Jupyter Notebook running inside your container that has direct access to your database by the container name (db), etc. So, you can just run sections of your backend code directly, for example with VS Code Python Jupyter Interactive Window or Hydrogen.

Migrations

As during local development your app directory is mounted as a volume inside the container, you can also run the migrations with alembic commands inside the container and the migration code will be in your app directory (instead of being only inside the container). So you can add it to your git repository.

Make sure you create a "revision" of your models and that you "upgrade" your database with that revision every time you change them. As this is what will update the tables in your database. Otherwise, your application will have errors.

Start an interactive session in the backend container:

$ docker compose exec backend bash

If you created a new model in ./backend/app/app/models/, make sure to import it in ./backend/app/app/db/base.py, that Python module (base.py) that imports all the models will be used by Alembic.
After changing a model (for example, adding a column), inside the container, create a revision, e.g.:

$ alembic revision --autogenerate -m "Add column last_name to User model"

Commit to the git repository the files generated in the alembic directory.
After creating the revision, run the migration in the database (this is what will actually change the database):

$ alembic upgrade head

If you don't want to use migrations at all, uncomment the line in the file at ./backend/app/app/db/init_db.py with:

Base.metadata.create_all(bind=engine)

and comment the line in the file prestart.sh that contains:

$ alembic upgrade head

If you don't want to start with the default models and want to remove them / modify them, from the beginning, without having any previous revision, you can remove the revision files (.py Python files) under ./backend/app/alembic/versions/. And then create a first migration as described above.

Development with Docker Toolbox

If you are using Docker Toolbox in Windows or macOS instead of Docker for Windows or Docker for Mac, Docker will be running in a VirtualBox Virtual Machine, and it will have a local IP different than 127.0.0.1, which is the IP address for localhost in your machine.

The address of your Docker Toolbox virtual machine would probably be 192.168.99.100 (that is the default).

As this is a common case, the domain local.dockertoolbox.tiangolo.com points to that (private) IP, just to help with development (actually dockertoolbox.tiangolo.com and all its subdomains point to that IP). That way, you can start the stack in Docker Toolbox, and use that domain for development. You will be able to open that URL in Chrome and it will communicate with your local Docker Toolbox directly as if it was a cloud server, including CORS (Cross Origin Resource Sharing).

If you used the default CORS enabled domains while generating the project, local.dockertoolbox.tiangolo.com was configured to be allowed. If you didn't, you will need to add it to the list in the variable BACKEND_CORS_ORIGINS in the .env file.

To configure it in your stack, follow the section Change the development "domain" below, using the domain local.dockertoolbox.tiangolo.com.

After performing those steps you should be able to open: http://local.dockertoolbox.tiangolo.com and it will be server by your stack in your Docker Toolbox virtual machine.

Check all the corresponding available URLs in the section at the end.

Development in `localhost` with a custom domain

You might want to use something different than localhost as the domain. For example, if you are having problems with cookies that need a subdomain, and Chrome is not allowing you to use localhost.

In that case, you have two options: you could use the instructions to modify your system hosts file with the instructions below in Development with a custom IP or you can just use localhost.tiangolo.com, it is set up to point to localhost (to the IP 127.0.0.1) and all its subdomains too. And as it is an actual domain, the browsers will store the cookies you set during development, etc.

If you used the default CORS enabled domains while generating the project, localhost.tiangolo.com was configured to be allowed. If you didn't, you will need to add it to the list in the variable BACKEND_CORS_ORIGINS in the .env file.

To configure it in your stack, follow the section Change the development "domain" below, using the domain localhost.tiangolo.com.

After performing those steps you should be able to open: http://localhost.tiangolo.com and it will be server by your stack in localhost.

Check all the corresponding available URLs in the section at the end.

Development with a custom IP

If you are running Docker in an IP address different than 127.0.0.1 (localhost) and 192.168.99.100 (the default of Docker Toolbox), you will need to perform some additional steps. That will be the case if you are running a custom Virtual Machine, a secondary Docker Toolbox or your Docker is located in a different machine in your network.

In that case, you will need to use a fake local domain (dev.karl.qanta.org) and make your computer think that the domain is is served by the custom IP (e.g. 192.168.99.150).

If you used the default CORS enabled domains, dev.karl.qanta.org was configured to be allowed. If you want a custom one, you need to add it to the list in the variable BACKEND_CORS_ORIGINS in the .env file.

Open your hosts file with administrative privileges using a text editor:
- Note for Windows: If you are in Windows, open the main Windows menu, search for "notepad", right click on it, and select the option "open as Administrator" or similar. Then click the "File" menu, "Open file", go to the directory c:\Windows\System32\Drivers\etc\, select the option to show "All files" instead of only "Text (.txt) files", and open the hosts file.
- Note for Mac and Linux: Your hosts file is probably located at /etc/hosts, you can edit it in a terminal running sudo nano /etc/hosts.
Additional to the contents it might have, add a new line with the custom IP (e.g. 192.168.99.150) a space character, and your fake local domain: dev.karl.qanta.org.

The new line might look like:

192.168.99.100    dev.karl.qanta.org

Save the file.
- Note for Windows: Make sure you save the file as "All files", without an extension of .txt. By default, Windows tries to add the extension. Make sure the file is saved as is, without extension.

...that will make your computer think that the fake local domain is served by that custom IP, and when you open that URL in your browser, it will talk directly to your locally running server when it is asked to go to dev.karl.qanta.org and think that it is a remote server while it is actually running in your computer.

To configure it in your stack, follow the section Change the development "domain" below, using the domain dev.karl.qanta.org.

After performing those steps you should be able to open: http://dev.karl.qanta.org and it will be server by your stack in localhost.

Check all the corresponding available URLs in the section at the end.

Change the development "domain"

If you need to use your local stack with a different domain than localhost, you need to make sure the domain you use points to the IP where your stack is set up. See the different ways to achieve that in the sections above (i.e. using Docker Toolbox with local.dockertoolbox.tiangolo.com, using localhost.tiangolo.com or using dev.karl.qanta.org).

To simplify your Docker Compose setup, for example, so that the API docs (Swagger UI) knows where is your API, you should let it know you are using that domain for development. You will need to edit 1 line in 2 files.

Open the file located at ./.env. It would have a line like:

DOMAIN=localhost

Change it to the domain you are going to use, e.g.:

DOMAIN=localhost.tiangolo.com

That variable will be used by the Docker Compose files.

Now open the file located at ./frontend/.env. It would have a line like:

VUE_APP_DOMAIN_DEV=localhost

Change that line to the domain you are going to use, e.g.:

VUE_APP_DOMAIN_DEV=localhost.tiangolo.com

That variable will make your frontend communicate with that domain when interacting with your backend API, when the other variable VUE_APP_ENV is set to development.

After changing the two lines, you can re-start your stack with:

docker compose up -d

and check all the corresponding available URLs in the section at the end.

Working with Celeryworker in Dev

If you make changes to backend/app/app/worker.py (which contains celery tasks), you have to restart the celeryworker docker compose restart celeryworker.

Database Back Ups

The following backup script identifies the docker container and performs a backup of the entire Postgres server, naming it using the current timestamp.

./scripts/backup.sh

The following load backup script takes in a relative file path to the file dump and restores it.

./scripts/load-backup.sh [Path to dump] [./.env location]

Database Backup for local analysis

To add to your local database, you should backup only the app database. The following script does this:

./scripts/backup_app.sh

Then, on your local postgres (not in docker), you can run the following commands to add the data:

gunzip -k dump_app_2023-12-11_07_36_31.gz
createdb -U postgres karlapp126new
psql -U postgres -d karlapp1211new -f dump_app_2023-12-11_07_36_31

Analysis

SELECT h.*, sd.*
FROM history h
JOIN session_deck sd ON sd.studyset_id = (h.details::json->>'studyset_id')::int
WHERE h.time > '2022-11-19'
AND h.details::json->>'set_type' IN ('test', 'post_test');

Frontend development

Enter the frontend directory, install the NPM packages and start the live server using the npm scripts:

cd frontend
npm install
npm run serve

Then open your browser at http://localhost:8080

Notice that this live server is not running inside Docker, it is for local development, and that is the recommended workflow. Once you are happy with your frontend, you can build the frontend Docker image and start it, to test it in a production-like environment. But compiling the image at every change will not be as productive as running the local development server with live reload.

Check the file package.json to see other available options.

If you have Vue CLI installed, you can also run vue ui to control, configure, serve, and analyze your application using a nice local web user interface.

If you are only developing the frontend (e.g. other team members are developing the backend) and there is a staging environment already deployed, you can make your local development code use that staging API instead of a full local Docker Compose stack.

To do that, modify the file ./frontend/.env, there's a section with:

VUE_APP_ENV=development
# VUE_APP_ENV=staging

Switch the comment, to:

# VUE_APP_ENV=development
VUE_APP_ENV=staging

Docker Compose files and env vars

There is a main docker-compose.yml file with all the configurations that apply to the whole stack, it is used automatically by docker compose.

And there's also a docker-compose.override.yml with overrides for development, for example to mount the source code as a volume. It is used automatically by docker compose to apply overrides on top of docker-compose.yml.

These Docker Compose files use the .env file containing configurations to be injected as environment variables in the containers.

They also use some additional configurations taken from environment variables set in the scripts before calling the docker compose command.

It is all designed to support several "stages", like development, building, testing, and deployment. Also, allowing the deployment to different environments like staging and production (and you can add more environments very easily).

They are designed to have the minimum repetition of code and configurations, so that if you need to change something, you have to change it in the minimum amount of places. That's why files use environment variables that get auto-expanded. That way, if for example, you want to use a different domain, you can call the docker compose command with a different DOMAIN environment variable instead of having to change the domain in several places inside the Docker Compose files.

Also, if you want to have another deployment environment, say preprod, you just have to change environment variables, but you can keep using the same Docker Compose files.

URLs

These are the URLs used and generated by the project.

Production URLs

Production URLs, from the branch production.

Frontend: https://karl.qanta.org

Backend: https://karl.qanta.org/api/

Automatic Interactive Docs (Swagger UI): https://karl.qanta.org/docs

Automatic Alternative Docs (ReDoc): https://karl.qanta.org/redoc

PGAdmin: https://pgadmin.karl.qanta.org

Flower: https://flower.karl.qanta.org

Development URLs

Development URLs, for local development.

Frontend: http://localhost

Backend: http://localhost/api/

Automatic Interactive Docs (Swagger UI): https://localhost/docs

Automatic Alternative Docs (ReDoc): https://localhost/redoc

PGAdmin: http://localhost:5050

Flower: http://localhost:5555

Traefik UI: http://localhost:8090

Development with Docker Toolbox URLs

Development URLs, for local development.

Frontend: http://local.dockertoolbox.tiangolo.com

Backend: http://local.dockertoolbox.tiangolo.com/api/

Automatic Interactive Docs (Swagger UI): https://local.dockertoolbox.tiangolo.com/docs

Automatic Alternative Docs (ReDoc): https://local.dockertoolbox.tiangolo.com/redoc

PGAdmin: http://local.dockertoolbox.tiangolo.com:5050

Flower: http://local.dockertoolbox.tiangolo.com:5555

Traefik UI: http://local.dockertoolbox.tiangolo.com:8090

Development with a custom IP URLs

Development URLs, for local development.

Frontend: http://dev.karl.qanta.org

Backend: http://dev.karl.qanta.org/api/

Automatic Interactive Docs (Swagger UI): https://dev.karl.qanta.org/docs

Automatic Alternative Docs (ReDoc): https://dev.karl.qanta.org/redoc

PGAdmin: http://dev.karl.qanta.org:5050

Flower: http://dev.karl.qanta.org:5555

Traefik UI: http://dev.karl.qanta.org:8090

Development in localhost with a custom domain URLs

Development URLs, for local development.

Frontend: http://localhost.tiangolo.com

Backend: http://localhost.tiangolo.com/api/

Automatic Interactive Docs (Swagger UI): https://localhost.tiangolo.com/docs

Automatic Alternative Docs (ReDoc): https://localhost.tiangolo.com/redoc

PGAdmin: http://localhost.tiangolo.com:5050

Flower: http://localhost.tiangolo.com:5555

Traefik UI: http://localhost.tiangolo.com:8090

Credits

Full Stack FastAPI PostgreSQL

This project and readme were partially generated using https://github.com/tiangolo/full-stack-fastapi-postgresql with:

pip install cookiecutter
cookiecutter https://github.com/tiangolo/full-stack-fastapi-postgresql

Name		Name	Last commit message	Last commit date
Latest commit History 911 Commits
backend		backend
frontend		frontend
scripts		scripts
.gitattributes		.gitattributes
.gitignore		.gitignore
.gitlab-ci.yml		.gitlab-ci.yml
Attributions.md		Attributions.md
LICENSE		LICENSE
README.md		README.md
SECURITY.md		SECURITY.md
docker-compose.override.yml		docker-compose.override.yml
docker-compose.yml		docker-compose.yml

License

Pinafore/karl-flashcards-web-app

Folders and files

Latest commit

History

Repository files navigation

KARL

Development

Restrictions

Env Files

Example Root .env File

Example Frontend/ .env File

Backend Requirements

Frontend Requirements

Backend local development

Backend local development, additional details

General workflow

Docker Compose Override

Backend tests

Local tests

Test running stack

Test Coverage

Live development with Python Jupyter Notebooks

Migrations

Development with Docker Toolbox

Development in localhost with a custom domain

Development with a custom IP

Change the development "domain"

Working with Celeryworker in Dev

Database Back Ups

Database Backup for local analysis

Analysis

Frontend development

Docker Compose files and env vars

URLs

Production URLs

Development URLs

Development with Docker Toolbox URLs

Development with a custom IP URLs

Development in localhost with a custom domain URLs

Credits

Full Stack FastAPI PostgreSQL

About

Topics

Resources

License

Security policy

Stars

Watchers

Forks

Releases

Packages 0

Contributors 4

Languages

Example Root `.env` File

Example `Frontend/` `.env` File

Development in `localhost` with a custom domain

Packages