Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Export Exasol Table into AWS S3 as Parquet format #16

Merged
merged 21 commits into from
Feb 12, 2019
Merged

Export Exasol Table into AWS S3 as Parquet format #16

merged 21 commits into from
Feb 12, 2019

Commits on Jan 22, 2019

  1. Configuration menu
    Copy the full SHA
    a369225 View commit details
    Browse the repository at this point in the history

Commits on Jan 23, 2019

  1. Start Exasol table to S3 export as parquet format

    This commit only adds the ExportPath class.
    morazow committed Jan 23, 2019
    Configuration menu
    Copy the full SHA
    1d777fe View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    7fab7da View commit details
    Browse the repository at this point in the history

Commits on Jan 25, 2019

  1. Configuration menu
    Copy the full SHA
    c6b2907 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    6faa6c2 View commit details
    Browse the repository at this point in the history

Commits on Jan 29, 2019

  1. Fix type import -> export

    morazow committed Jan 29, 2019
    Configuration menu
    Copy the full SHA
    c268f63 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    a81db06 View commit details
    Browse the repository at this point in the history

Commits on Jan 30, 2019

  1. Configuration menu
    Copy the full SHA
    53dff6b View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    a5ded42 View commit details
    Browse the repository at this point in the history

Commits on Feb 1, 2019

  1. Configuration menu
    Copy the full SHA
    81b119b View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    661e6f6 View commit details
    Browse the repository at this point in the history
  3. Add initial export functionality

    - Adds ParquetWriter
    - Adds RowWriteSupport
    
    However, there are still some more changes needed:
    
      - Decide on decimal to int32 or int64 based on precision
      - Improve the import functionality with date and timestamps
    morazow committed Feb 1, 2019
    Configuration menu
    Copy the full SHA
    3932d77 View commit details
    Browse the repository at this point in the history

Commits on Feb 4, 2019

  1. Configuration menu
    Copy the full SHA
    e9e1dd8 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    b32e100 View commit details
    Browse the repository at this point in the history

Commits on Feb 5, 2019

  1. Add date and timestamp reader functionality

    Should fix #15
    morazow committed Feb 5, 2019
    Configuration menu
    Copy the full SHA
    9f1d1f5 View commit details
    Browse the repository at this point in the history

Commits on Feb 8, 2019

  1. Configuration menu
    Copy the full SHA
    e572bad View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    c6013c2 View commit details
    Browse the repository at this point in the history

Commits on Feb 11, 2019

  1. Do not separate decimal values into int32 or int64 values

    This does not make sense in this case, because both when reading or writing exasol will provide
    correct Java type, e.g, BigDecimal if decimal with precision and scale and regular Integer or Long
    if int32 or int64.
    morazow committed Feb 11, 2019
    Configuration menu
    Copy the full SHA
    a8daf06 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    9540696 View commit details
    Browse the repository at this point in the history
  3. Add fs.LocalFileSystem into configuration

    S3AFileSystem somehow requires local temp directory, before uploading to S3. This currently only
    tested for AWS S3. I will update this for GCP or Azure if they require this filesystem when
    performing tests for those platforms.
    morazow committed Feb 11, 2019
    Configuration menu
    Copy the full SHA
    7466de1 View commit details
    Browse the repository at this point in the history
  4. Fix bug when converting date into days (since epoch) and back

    An example date was: `0001-01-01`, or in general days before 1970.
    
    After writing this `0001-01-01` as a days since epoch and when reading back I was getting
    `0001-12-31`.
    
    The solution was to incorporate the timezone offset millis and take the *floor* of the millis per
    day.
    morazow committed Feb 11, 2019
    Configuration menu
    Copy the full SHA
    5023000 View commit details
    Browse the repository at this point in the history