Development using Docker Compose

Requirements

docker-compose -f local/compose.yml up

You can test the example jobs/pyspark_hello_world.py in the following way:

# attach to the container
docker-compose -f local/compose.yml exec -it awsglue /bin/bash

and then run the following command inside the container:

# Run the PySpark job
glue-spark-submit jobs/pyspark_hello_world.py --JOB_NAME job_example --CUSTOM_ARGUMENT custom_value