hands_on_minio.md

Hands-on S3-Virtual-Schema With MinIO

Introduction

Virtual Schema for Document Files in S3 (VSS3) supports using two different variants of S3 buckets:

• Real S3 buckets in the AWS cloud
• S3 buckets as emulated by MinIO's S3-compatible interface

This Hands-On Guide describes how to access an S3 bucket emulated by a MinIO Docker Container, see also MinIO documentation.

Concepts

Architectural Overview

Credentials

Select a user name and a password for the MinIO Docker Container. Throughout this Hands On Guide these strings will be named as specified in column Reference:

MinIO Terminology	AWS terminology	Reference
User name for MinIO Docker Container	<AWS ACCESS KEY ID>	$MINIO_USER
Password for MinIO Docker Container	<AWS SECRET KEY>	$MINIO_PASS

Variables

In the following this Hand-On Guide will use variables for the Host running the MinIO server name and the name of sample S3 bucket:

export MINIO_HOST=localhost
export MINIO_USER=user
export MINIO_PASS=password
export MINIO_BUCKET=my-bucket

Prerequisites and installation

• Docker Server and Client
• Minio Docker Container
• Minio Client
• Exasol Docker Container

Setup MinIO Docker Container and Client

Download MinIO Docker Image and run it as Docker Container

docker run --detach --name minio \
  -p 9000:9000 \
  -p 9001:9001 \
  -e "MINIO_ROOT_USER=$MINIO_USER" \
  -e "MINIO_ROOT_PASSWORD=$MINIO_PASS" \
  -it minio/minio:latest \
  server /data --console-address ":9001"

With the following command you can verify the container is running:

docker ps

In the output of docker ps you can also see the ports forwarded to your localhost:

26a6f597ea85  minio/minio:...fips  0.0.0.0:9000-9001->9000-9001/tcp  minio

Setup MinIO Client mc

For adding content to MinIO you can use the MinIO client mc. Its manual also contains links to download a binary for various platforms.

The following following command defines alias myminio for your MinIO server using the selected credentials:

mc alias set myminio http://$MINIO_HOST:9000/ $MINIO_USER $MINIO_PASS

Emulate an S3 Bucket With Sample Files

Next we will tell MinIO to emulate a sample S3 bucket:

mc mb myminio/$MINIO_BUCKET

Download the sample Json files and upload them to the emulated S3 bucket:

mc cp book-1.json myminio/$MINIO_BUCKET/
mc cp book-2.json myminio/$MINIO_BUCKET/
mc cp book-3.json myminio/$MINIO_BUCKET/

Setting up the Exasol Database

Initial Setup

Run Exasol database, e.g. as Docker-DB in a Docker Container and follow the instructions in the following sections of document Hands-on S3-Virtual-Schema to execute the required SQL statements:

• Section Installation
  • Download VSS3 latest jar and upload it to BucketFS
  • Create Java Adapter Script
  • Create Java Set Script
• Section Creating a Mapping Definition to create a mapping definition and upload it to BucketFS.

Creating the Connection to MinIO

In section Emulate an S3 Bucket With Sample Files we configured MinIO to emulate a sample S3 bucket.

The following SQL statement will create a connection to this bucket:

CREATE OR REPLACE CONNECTION S3_CONNECTION
  TO '' USER ''
  IDENTIFIED BY '{
      "awsAccessKeyId": "$MINIO_USER",
      "awsSecretAccessKey": "$MINIO_PASS",
      "awsRegion": "any-region",
      "s3Bucket": "$MINIO_BUCKET",
      "awsEndpointOverride": "$MINIO_HOST:9000",
      "useSsl": false
  }';

Please note:

• Your SQL client (e.g. DbVisualizer) may not be able to resolve shell variables $MINIO_USER, MINIO_PASS, $MINIO_HOST, $MINIO_BUCKET.
• So please replace these variables by the actual values described in section Variables.

Create a Virtual Schema and Run Your First Query

Follow additional instructions in document Hands-on S3-Virtual-Schema:

• Creating the Virtual Schema
• First Query to execute your first query and watch the results in your SQL client.