impactedsurveyanalysis

Overview

Analysis of ImpactEd's longitudinal survey.

Requirements

To run the code in this GitHub repository, please make sure your system meets the following requirements:

• Unix-like operating system (macOS, Linux, …);
• direnv installed, including shell hooks;
• .envrc allowed/trusted by direnv to use the environment variables - see below;
• If missing, create a .secrets file to store untracked secrets;
• Python 3.5 or above; and
• Python packages installed from the requirements.txt file.

Note there may be some Python IDE-specific requirements around loading environment variables, which are not considered here.

Allowing/trusting .envrc

To allow/trust the .envrc run the allow command using direnv at the top level of this repository.

direnv allow

Creating a .secrets file

Secrets used by this repository can be stored in a .secrets file. This is not tracked by Git, and so secrets will not be committed onto your remote.

In your shell terminal, at the top level of the repository, create a .secrets file.

touch .secrets

Open this new .secrets file using a text editor, and add any secrets as environmental variables. For example, to add a JSON credentials file for Google BigQuery, add the following code to .secrets.

export GOOGLE_APPLICATION_CREDENTIALS="path/to/credentials.json"

Installing Python packages

To install required Python packages via pip, first set up a Python virtual environment; this ensures you do not install the packages globally.

Then run the following make command at the top level of this repository:

make requirements

Once you have installed the packages, remember to set up pre-commit hooks.

Creating a Python virtual environment

Creating a Python virtual environment depends on whether you are using base Python or Anaconda as your interpreter.

Base Python interpreter

If you are using base Python, there are multiple ways to create virtual environments in Python using pip, including (but not limited to):

• venv;
• virtualenv;
• pipenv; and
• pyenv with its virtualenv plugin.

Follow the documentation of your chosen method to create a Python virtual environment.

Anaconda interpreter

If you are using Anaconda or conda, following their documentation to set up a conda environment.

Folder structure

An overview of the folder structure, and the top-level files can be found here.

Installing pre-commit hooks

This repo uses the Python package pre-commit to manage pre-commit hooks. Pre-commit hooks are actions which are run automatically, typically on each commit, to perform some common set of tasks. For example, a pre-commit hook might be used to run any code linting automatically, providing any warnings before code is committed, ensuring that all of our code adheres to a certain quality standard.

For this repo, we are using pre-commit for a number of purposes:

• Checking for any secrets being committed accidentally;
• Checking for any large files (over 5MB) being committed; and
• Cleaning Jupyter notebooks, which means removing all outputs and execution counts.

We have configured pre-commit to run automatically on every commit. By running on each commit, we ensure that pre-commit will be able to detect all contraventions and keep our repo in a healthy state.

In order for pre-commit to run, action is needed to configure it on your system.

• Install the pre-commit package into your Python environment; and
• Run pre-commit install to set-up pre-commit to run when code is committed.

Setting up a baseline for the detect-secrets hook (if one doesn't already exist)

The detect-secrets hook requires that you generate a baseline file if one is not already present within the root directory. This is done via running the following at the root of the repo:

detect-secrets scan > .secrets.baseline

Next, audit the baseline that has been generated by running:

detect-secrets audit .secrets.baseline

When you run this command, you'll enter an interactive console and be presented with a list of high-entropy string / anything which could be a secret, and asked to verify whether or not this is the case. By doing this, the hook will be in a position to know if you're later committing any new secrets to the repo and it will be able to alert you accordingly.

If pre-commit detects secrets during commit:

If pre-commit detects any secrets when you try to create a commit, it will detail what it found and where to go to check the secret.

If the detected secret is a false-positive, you should update the secrets baseline through the following steps:

• Run detect-secrets scan --update .secrets.baseline to index the false-positive(s);
• Next, audit all indexed secrets via detect-secrets audit .secrets.baseline (the same as during initial set-up, if a secrets baseline doesn't exist); and
• Finally, ensure that you commit the updated secrets baseline in the same commit as the false-positive(s) it has been updated for.

If the detected secret is actually a secret (or other sensitive information), remove the secret and re-commit. There is no need to update the secrets baseline in this case.

If your commit contains a mixture of false-positives and actual secrets, remove the actual secrets first before updating and auditing the secrets baseline.

Using this with R

Please follow all the above steps.

After that, we use renv to manage package dependencies here. Thus, before writing any code, run the following R code in your R session:

# installs packages for pre-commit hooks in DESCRIPTION file
renv::install()
# installs packages for project captured in renv.lock
renv::restore()

This will install additional packages that are necessary to work alongside the pre-commit hooks for R code.

Note on Jupyter notebook cleaning

It may be necessary or useful to keep certain output cells of a Jupyter notebook, for example charts or graphs visualising some set of data. To do this, add the following comment at the top of the input block:

# [keep_output]

This will tell pre-commit not to strip the resulting output of this cell, allowing it to be committed.