Beam Oslo City Bike Project

Beam project using public Oslo City Bike data

Running the code

Software Requirements

• Java 8 (Java 9 or above are not officially supported by Beam yet)
• Maven 3.5.x

Running on your local machine

mvn compile exec:java \
      -Dexec.mainClass=com.mehmandarov.beam.OsloCityBike \
      -Dexec.args="--availabilityInputFile=src/main/resources/bikedata-availability-example.txt --stationMetadataInputFile=src/main/resources/bikedata-stations-example.txt --output=bikedatalocal" \
      -Pdirect-runner

Running on Google Cloud Platform

First you will need to create and download a GCP credentials file.

export GOOGLE_APPLICATION_CREDENTIALS="/example/path/to/your/file/sykkeldata-creds.json"

To run the code, use the following example. Make sure to update --project, --stagingLocation, --output, and --tempLocation. You will also need to make sure that all the Google Storage buckets are in the same region (EU, US, etc.) .

mvn -Pdataflow-runner compile exec:java \
      -Dexec.mainClass=com.mehmandarov.beam.OsloCityBike \
      -Dexec.args="--project=rm-cx-211107 \
      --stagingLocation=gs://my_oslo_bike_data/testing/ \
      --output=gs://my_oslo_bike_data/testing/output \
      --tempLocation=gs://my_oslo_bike_data/testing/ \
      --runner=DataflowRunner \
      --region=europe-west1"