EODataDown

A tool for automatically downloading Earth Observation imagery and products.

Dependencies

• SQLAlchemy [http://www.sqlalchemy.org]
• RSGISLib [http://www.rsgislib.org]
• ARCSI [http://arcsi.remotesensing.info]
• Requests [http://www.python-requests.org]
• Planet [https://github.com/planetlabs/planet-client-python]
• pycurl [http://pycurl.io]
• wget [https://www.gnu.org/software/wget/]
• postgresql [https://www.postgresql.org]
• psycopg2 [http://initd.org/psycopg/]
• Google Cloud Services [https://github.com/GoogleCloudPlatform/google-cloud-python]
• PyYAML [https://pyyaml.org]

Google Cloud Services

For google cloud services the following dependencies need to installed.

• google-api-core
• google-cloud-core
• google-auth
• google-cloud-bigquery
• google-cloud-storage

[https://github.com/GoogleCloudPlatform/google-cloud-python]

Installation

Using conda and conda-forge the dependencies can be installed within the following command:

conda install -c conda-forge pycurl postgresql psycopg2 sqlalchemy google-cloud-storage google-cloud-bigquery arcsi wget pyyaml planet

Configuration

There are a number of configuration files which need to created for the system to work

Overall System

An example is within the share directory of the source code.

Logging

EODataDown uses the python logging library. A general configuration suitable for most users is installed alongside the source <install_path>/share/eodatadown. This can be edited or the EDD_LOG_CFG variable used to use a different configuration file.

Environmental Variables

• EDD_MAIN_CFG - specify the location of a JSON file configuring the system.
• EDD_LOG_CFG - specify the location of a JSON file configuring the python logging system.
• EDD_NCORES - specify the number of cores to use when running jobs which can use multiple cores.

Notes for Sensors

Landsat and Sentinel-2 from Google.

To use this sensor you need to set up a google account and project. While the data and download is free an account is still required. See https://cloud.google.com/resource-manager/docs/creating-managing-projects

Details of the account will need to be specified in the JSON config file for the LandsatGOOG sensor.

Setting up Google Accounts

See the following page for setting up a project:

https://cloud.google.com/resource-manager/docs/creating-managing-projects

And then the following page for setting up your credentials:

https://cloud.google.com/bigquery/docs/reference/libraries

Setting up Postgresql Database

New database system

To initialise a database use the initdb command.

initdb -D <path to data>

Start database

postgres -D <path to data>

Alternatively, you can start the database as a background service.

pg_ctl -D <path to data> -l logfile start

Create database

createdb eodd_dev_db

Setup User

psql

CREATE USER eodduser WITH PASSWORD '<password>';
GRANT ALL PRIVILEGES ON DATABASE eodd_dev_db TO eodduser;