Skip to content

Migration TEK to ITK 2022

Jump to bottom
Ryan Hodges edited this page Jul 28, 2022 · 7 revisions

Migrating from TEKDB (2017) to ITKDB (2022)

The TEKDB tool did not ship with a built-in process to export data. In this walkthrough, you will need:

  • A computer with a browser and either a Unix terminal or Putty to access Linux servers via SSH
  • ssh access to your old TEKDB server (launched between 2017 and 2021)
    • This guide takes a guess at what your commands should be, but familiarity with Unix command line is highly recommended.
  • Admin dashboard login credentials to your new ITKDB server (launched in 2022 or later)

The process (in brief): You will

  1. Create a single zip file archive representing all of the data currently on your server
  2. Download that file to your workstation
  3. Import that file from your workstation into the new ITKDB server
  4. Test that your data imported correctly
  5. DELETE THE ZIP FILE

Export your data

Source: TEKDB 2017

  1. Change directory to your project folder and activate your python virtual environment, i.e.: 
    cd /usr/local/apps/TEKDB
source /usr/local/apps/TEKDB/env/bin/activate
  2. Use git to download the script you need to generate and export file: 
    git fetch
git cherry-pick 9256f8b65311e61505003317b8afd5486ae72a02
  3. Run the export_db management command and note down the output result: 
    sudo chmod +r /usr/local/apps/TEKDB/TEKDB/media/*
python /usr/local/apps/TEKDB/TEKDB/manage.py export_db
  4. Download the output zip file to a local computer
    • If on Windows and using putty to connect, you will need another tool to download. We recommend WinSCP
    • If on Unix you can use a command from the terminal like:
    scp {username}@{server}{:port}/tmp/{YYYYMMDD}_backup.zip /path/to/desired/location/

Import your data

Target: ITKDB 2022

Via Admin Dashboard (easiest):

  1. Log in as an admin to your new ITKDB site
  2. using either the user-menu navigation, or by going to "{your.itkdb.site}/admin", access the admin dashboard
  3. At the right side of the screen, under 'Import Database' click on the 'browse' button
    • navigate the box menu to the zip file downloaded in the 'Export' instructions
    • select it and accept or 'OK' to close the dialog box
  4. Click on the green 'Import' button
  5. Wait. Depending on how much data you are migrating, this could take a very long time.
  6. You will be informed when the process finished with a pop-up box.
    • If you get an error, review troubleshooting below, or reach out to ksdev@ecotrust.org for support.
  7. Delete the ZIP FILE from your computer and never share it, email it, or otherwise expose it to the internet or anyone outside of your team

Via ssh terminal/command line (Ubuntu):

Getting a .SQL File

Unfortunately at this time, the .json fixture does not seem compatible for migrating from the old version to the new. Instead, you will also need to run the following on your old server to get a SQL dump of your database:

sudo -u postgres dump_db > /tmp/YYYYMMDD_dbdump.sql

Organizing the data

  1. Be sure to have used scp to place the exported zipfile AND your new .SQL dump somewhere you can find it.
  2. mkdir /usr/local/apps/TEKDB/data
  3. use the mv command to put your .zip and .sql files into /usr/local/apps/TEKDB/data
  4. cd /usr/local/apps/TEKDB/data
  5. sudo apt install unzip
  6. unzip YYYYMMDD_backup.zip (your backup filename)

Loading the .SQL file

If you have already created your database (we'll assume the name 'tekdb') but have not run the manage.py migrate commpand then your database should still be empty and schemaless. If not, delete your new (target) server's database, and recreate it (remembering to assign an owner).

Now you can easily populate the data, then migrate to the modern schema:

Assumptions:

  • db is named tekdb
  • Your dbdump file is named YYYYMMDD_dbdump.sql (it should be more like 20220727_dbdump.sql)
  • Your virtual environment is activated (env) 
sudo -u postgres psql tekdb < YYYYMMDD_dbdump.sql
python /usr/local/apps/TEKDB/TEKDB/manage.py migrate

LOADING MEDIA

  1. clear out the current media folder
    • sudo rm -r /usr/local/apps/TEKDB/TEKDB/media/*
  2. Move your imported (unzipped) media files into it:
    • sudo mv /usr/local/apps/TEKDB/data/media/* /usr/local/apps/TEKDB/TEKDB/media/
  3. If you aren't following along with the install guide, be sure to set the media folder and contents owner to www-data (or group to mediausers). Otherwise this will be covered near the end of the installation.

Final Touch-Ups:

AUTH GROUPS

Once you've imported your data and have the site running, log in as an admin and pop open AUTHENTICATION AND AUTHORIZATION -> Groups

  • Administrator: Be sure Administrator has ALL permissions
  • Editor: Editor should have all permissions starting with:
    • Relationships|
    • TEKDB|
    • and NOTHING else
  • Reader should have NO permissions

PAGE CONTENTS

  • Help: review that all old links are still valid (any pointing to the GitHub repo with a 'master' branch will need updating)

OTHERS

  • Double check that you see the amount of records in each table
  • Double check that place data appears correct on maps
  • Double check that media is accessible (images load, video/audio plays, .pdfs download, etc...)

TROUBLESHOOTING:

I get an error when I try to create new lookups:

  1. Update your Database iterators A problem on older installs is that after importing data into the database, the iterators may not be updated (or maybe are overwritten by the data import). Rather than check that every single iterator sequence in the database matches the IDs that exist in their sister tables, we've written a handy script:

    1. If the database schema has changed since you last generated the sequence updater script, create a new one:

      cd /usr/local/apps/TEKDB/deployment
sudo -u postgres psql -Atq tekdb -f /usr/local/apps/TEKDB/deployment/generate_db_iterator_reset.sql  -o /usr/local/apps/TEKDB/deployment/reset_db_iterators_YYYYMMDD.sql

      Be sure to replace YYYYMMDD with today's date and remove any old versions of the script.

    2. If the 'reset_db_iterators...' script is up to date, run it:

      sudo -u postgres psql tekdb -f /usr/local/apps/TEKDB/deployment/reset_db_iterators_YYYYMMDD.sql
Clone this wiki locally