shavar-list-creation

This script fetches blocklist .json from urls (such as shavar-prod-lists) and generates safebrowsing-compatible digest list files to be served by shavar.

Requirements

• python ≥ 3.6
• (optional) virtualenv and/or virtualenvwrapper

Run

1. (optional) Make a virtualenv for the project and activate it:

   virtualenv -p python3.8 shavar-list-creation
   source shavar-list-creation/bin/activate
   

2. Install required libraries:

   pip install -r requirements-test.txt
   

3. Copy the sample_shavar_list_creation.ini file to shavar_list_creation.ini:

   cp sample_shavar_list_creation.ini shavar_list_creation.ini
   

4. Run the unit tests (currently under development):

   python -m pytest -v --cov=. --cov-branch
   

5. Run the lists2safebrowsing.py script:

   ./lists2safebrowsing.py
   

Usage

This is run by a Jenkins deployment job every 30 minutes that:

1. Checks out this repository
2. Checks out the shavar-list-creation-config repository
3. Copies stage.ini or prod.ini to shavar_list_creation.ini
4. Runs python lists2safebrowsing.py, which uploads updated safebrowsing list files to S3 for shavar. ⚠️ Since there is no concept of deploy for the scripts, the changes that are merged to main are immediately accessible Jenkins for both Stage and Production.