Skip to content

vightel/ojo-processing

Repository files navigation

Workshop Training

You will learn:

  • How to generate various products from NASA sources (and others) and store them on the cloud (AWS S3).
  • Use the Open GeoSocial API for data distribution, visualization, discovery and sharing via social netowrks

Notes: This is not authoritative but work in progress used for capacity building and examples... This is not operational software!

Algorithms have not been formally validated by the science team!

Please become a collaborator and help us improve this repository.

Copyright

Copyright © 2013 United States Government as represented by the Administrator of the National Aeronautics and Space Administration. All Other Rights Reserved

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Pre-Requisites Prior To Workshop

Workshop Part 1: ojo-processing

Setup

Create a development directory for the workshop and download the code repository

$ mkdir -p ~/Development/workshop
$ cd ~/Development/workshop
$ git clone https://github.com/vightel/ojo-processing.git
$ cd ojo-processing
$ export WORKSHOP_DIR=~/Development/workshop/ojo-processing
  • Create an Heroku Instance and a small Postgres database. We will use an enhanced buildpack used for later and remember your heroku app name
$ cp buildpacks .buildpacks
$ heroku login

$ heroku create
	Creating stark-sands-3006... done, stack is cedar-14
	https://stark-sands-3006.herokuapp.com/ | https://git.heroku.com/stark-sands-3006.git

$ heroku addons:create heroku-postgresql:hobby-dev --app stark-sands-3006
	Creating postgresql-spherical-2968... done, (free)
	Adding postgresql-spherical-2968 to stark-sands-3006... done
	Setting DATABASE_URL and restarting stark-sands-3006... done, v3
	Database has been created and is available

$ heroku config -s --app stark-sands-3006
  • Configure your environment with your own settings and DATABASE_URL. You will need to parse that URL and set the individual database environment variables as well.
$ cp dockerignore .dockerignore
$ cp envs.copy.sh envs.docker.sh
$ vi envs.docker.sh

Create Processing Container

Create a new Docker VM if you have never done so

$ docker-machine create --driver virtualbox default
$ docker-machine env default
$ eval "$(docker-machine env default)"

Build container image from provided docker file

$ docker build -t ojo-processing .

Now we can run and test the image

$ export DATAFOLDER="-v /Users/patricecappelaere/Development/workshop/ojo-processing:/home/workshop/ojo-processing"
$ docker run $DATAFOLDER --name ojo-processing -it --rm ojo-processing /bin/bash -login

Within that new shell, you can check:

$ gdalinfo --version
	GDAL 2.0.0, released 2015/06/14
$ ogrinfo --formats
$ cd $WORKSHOP_DIR/python
$ python check_environment.py

Landslide Nowcast Processing

You need to edit the python/config.py with your own AWS S3 bucket name for your region. Then run the processing script to generate the landslide nowcast for a particular day. The first time around, it will need to generate the last 60 days of precipitation.

Notes:

Central America is region d02 as defined in config.py

Python scripts can be run with -v option for verbose and -f to force the regeneration of products Make sure that you have the right envs for accessing your AWS S3 bucket (and it is public)

$ python landslide_nowcast.py --region d02 --date 2015-09-23 -v

TRMM data (already done in Landslide nowcast

python ./trmm_process.py --region d02 --date 2015-09-23

Quakes from USGS

python ./quake.py --region d02 --date 2015-09-23

MODIS BURNED AREAS from MODIS

python ./modis-burnedareas.py --region d02 --date 2015-09-23

MODIS ACTIVE FIRES from MODIS - PROBLEM with node

python ./modis-active-fires.py --region d02 --date 2015-09-23

GPM Global, 1-d, 3-d and 7-d products

python ./gpm_global.py  --date 2015-09-22

GEOS5 Global

python ./geos5.py  --date 2015-09-23

GPM Global

python ./gpm_global.py  --date 2015-09-22

GFMS Global

python ./gfms_vectorizer.py  --date %s 2015-09-23

NOTES

$ docker ps -a $ docker images $ docker inspect

Cleanup stopped containers and dangling images $ docker rm -v $(docker ps -aq ) $ docker rmi $(docker images -f dangling=true -q)

About

OJO Processing Backend for workshop

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published