Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Buck pkg mgmt #645

Merged
merged 76 commits into from
Feb 27, 2018
Merged

Buck pkg mgmt #645

merged 76 commits into from
Feb 27, 2018

Conversation

b-ryan
Copy link
Contributor

@b-ryan b-ryan commented Jan 29, 2018

No description provided.

drewbanin and others added 30 commits September 29, 2017 09:44
* fix for bad env_var exception

* overwrite target with compiled values

* fixes env vars, adds test. Auto-compile profile/target args
* improvements for code that runs in hooks

* fix error message note

* typo
* tojson was added in jinja 2.9

* requirements
* fix package-defined schema test macros

* create a dummy Relation in parsing

* fix for bq quoting (#565)
* bump snowflake, remove pyasn1

* change requirements.txt
* 090 docs

* Update CHANGELOG.md
* Raise CompilationException on duplicate model

Extend tests

* Ignore disabled models in parse_sql_nodes

Extend tests for duplicate model

* Fix preexisting models

* Use double quotes consistently

Rename model-1 to model-disabled

* Fix unit tests

* Raise exception on duplicate model across packages

Extend tests
* Make run_started_at timezone aware

Set run_started_at timezone to UTC
Enable timezone change in models
Extend requirements
Extend tests

* Address comments from code review

Create modules namespace to context
Move pytz to modules
Add new dependencies to setup.py
* Add warning for missing constraints. Fixes #592

* fix unit tests
* fix schema tests used in, or defined in packages

* don't hardcode dbt test namespace

* fix/actually run tests

* rm junk
* run hooks in correct order, fixes #590

* add tests

* fix tests

* pep8
From cffi callback <function _verify_callback at 0x06BF2978>:
Traceback (most recent call last):
  File "c:\projects\dbt\.tox\pywin\lib\site-packages\OpenSSL\SSL.py", line 313, in wrapper
    _lib.X509_up_ref(x509)
AttributeError: module 'lib' has no attribute 'X509_up_ref'
From cffi callback <function _verify_callback at 0x06B8CF60>:
* Fixes already opened transaction issue

For #602

* Fixes #621
Buck Ryan added 3 commits January 30, 2018 17:23
The downloads directory causes problems with the run command because
this directory is not a dbt project. Need to download it elsewhere.
Copy link
Contributor

@drewbanin drewbanin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lots of tiny cosmetic comments, but this PR is so solid. Great work! Let me know if you have questions about any of the comments

index_cached = memoized(index)


def packages(registry_base_url=None):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this used anywhere? The tinyurl link returns a 404

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catch - the hub.getdbt.com repo did not actually serve the packages.json route. In fact it still isn't unless I just have the URL wrong. How do I get the package JSON from the hub?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@b-ryan we're going to get this live this week @ hub.getdbt.com

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is /api/v1/packages.json the correct path for that json?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah - that will be the path once we get the site up!

start_sha = None
if exists:
directory = exists.group(1)
logger.info('Updating existing dependency %s.', directory)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think this is a little too verbose for logger.info -- can we use logger.debug instead? I think that means also changing the checkout function to also use logger.debug which is probably a good idea anyway.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I noticed that too, particularly because since I wrote some of the deps code to do the checkout at a few different places (just to make the code cleaner) it will always print the repo is already up to date. I can change all the info logs in this module to debug unless you think there are any in particular that should be info.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, i think that's a good idea. We'll still want to show some info to the user with info-level logging, but i don't think any of that ui-related code belongs in the git client

dbt/task/deps.py Outdated
logger.error(
'Could not resolve to a single version for Git repo %s!',
self.git)
logger.error(' Requested versions: %s', requested)
raise Exception('bad')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we throw a better exception here 😂

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

git dependencies should contain exactly one version. https://github.com/fishtown-analytics/dbt-utils contains: {'master', '0.0.1'}

dbt/semver.py Outdated

else:
match_found = True

print('going down the stack with {}'.format(new_unmet_dependencies))
print('going down the stack with {}'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we use debug logging for all of these print statements? Or just remove them if not needed anymore

dbt/task/deps.py Outdated
for v in cls.version_to_list(version)]
# VersionSpecifier.from_version_string will return None in case of
# failure, so we need to iterate again to check if any failed.
if any(not isinstance(v, VersionSpecifier) for v in version):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we make this error message contain the errant version string? I imagine this could be hard to track down without it

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would make more sense to have the underlying semver code throw an exception. Right now it's returning None in a place that appeared to me to make more sense as an exception.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dbt/task/deps.py Outdated


def _parse_package(dict_):
if dict_.get('package'):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we also check that only one of these is specified? Something like this:

packages:
  - git: https://github.com/fishtown-analytics/dbt-utils
    hub: fishtown-analytics/redshift
    version: 1.0.0

Will probably be a common failure mode for people unfamiliar with yaml

Copy link
Contributor Author

@b-ryan b-ryan Feb 2, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Packages should not contain more than one of ['package', 'git', 'local']; yours has 2 of them - ['package', 'git']

look ok?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

perfect

package,
cls._convert_version_strings(version_strings))

dbt.exceptions.raise_dependency_error(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice!

dbt/task/deps.py Outdated

packages = _read_packages(self.project)
if not packages:
return
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we log something like:

Warning: No packages found in dbt_project.yml

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure - so long as you want that. Are you figure the most common case will include at least one package?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh my comment was stupid. Nevermind.

requirements.txt Outdated
@@ -8,7 +8,9 @@ csvkit==0.9.1
snowplow-tracker==0.7.2
celery==3.1.23
voluptuous==0.10.5
snowflake-connector-python==1.4.0
snowflake-connector-python==1.4.14
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'll probably want to remove these changes before merging

return None
if not match:
raise dbt.exceptions.SemverException(
'Could not parse version "{}"'.format(version_string))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:)

dbt/semver.py Outdated

from dbt.exceptions import VersionsNotCompatibleException
import dbt.utils

logger = logging.getLogger(__name__)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We usually do:

from dbt.logger import GLOBAL_LOGGER as logger

Is there any difference to grabbing the logger this way?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah I see. Well I have a few thoughts on this. To answer your question, there is a difference. Using __name__ means the logger in this module is scoped to dbt.semver. So you could, for example, use this to filter or format logs for this module differently than everything else. Like

logging.getLogger("dbt.semver").setLevel(logging.ERROR)

But it also helps identify where logs are coming from if you were to put %(name)s in the format of the log messages since the name of the module would be in the logs.

You might not care about this, but one other thing I noticed is GLOBAL_LOGGER does not have a name, it's just:

logger = logging.getLogger()

which is the root logger. I would at least recommend changing this to logger = logging.getLogger("dbt"). Doing so would at least make this a lot simpler, since your logging config could just look like:

root_logger = logging.getLogger()
root_logger.setLevel(logging.CRITICAL)
logger = logging.getLogger("dbt")
logger.setLevel(logging.DEBUG)

and then you can whitelist modules you want to be more chatty instead of blacklisting modules.

I just tested all this out to confirm it's correct and the other thing you'd need to remove is https://github.com/fishtown-analytics/dbt/blob/development/scripts/dbt#L7 -- I find the basicConfig function is usually not helpful for anything other than one-off / simple scripts.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is super helpful, I think we will do this! Thanks for the heads up!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@b-ryan this worked super well, thanks so much for the tip :)

@drewbanin
Copy link
Contributor

@b-ryan this is looking really great. I'm going to continue to test locally over the weekend -- thanks for making those cosmetic changes

drewbanin and others added 15 commits February 9, 2018 20:33
* Fix for redshift varchar bug

* pep8 on a sql string, smh
* Set global variable overrides on the command line with --vars

* pep8

* integration tests for cli vars
* loader for seed data files

* Functioning rework of seed task

* Make CompilerRunner fns private and impl. SeedRunner.compile

Trying to distinguish between the public/private interface for this
class. And the SeedRunner doesn't need the functionality in the compile
function, it just needs a compile function to exist for use in the
compilation process.

* Test changes and fixes

* make the DB setup script usable locally

* convert simple copy test to use seeed

* Fixes to get Snowflake working

* New seed flag and make it non-destructive by default

* Convert update SQL script to another seed

* cleanup

* implement bigquery csv load

* context handling of StringIO

* Better typing

* strip seeder and csvkit dependency

* update bigquery to use new data typing and to fix unicode issue

* update seed test

* fix abstract functions in base adapter

* support time type

* try pinning crypto, pyopenssl versions

* remove unnecessary version pins

* insert all at once, rather than one query per row

* do not quote field names on creation

* bad

* quiet down parsedatetime logger

* pep8

* UI updates + node conformity for seed nodes

* add seed to list of resource types, cleanup

* show option for CSVs

* typo

* pep8

* move agate import to avoid strange warnings

* deprecation warning for --drop-existing

* quote column names in seed files

* revert quoting change (breaks Snowflake). Hush warnings
@drewbanin
Copy link
Contributor

TODO :

  • merge this with the existing PR
  • s/version/revision for Git packages

@drewbanin drewbanin merged commit 4e2cb40 into package-management Feb 27, 2018
@drewbanin drewbanin deleted the buck-pkg-mgmt branch February 27, 2018 00:14
drewbanin pushed a commit that referenced this pull request Feb 27, 2018
* semver resolution

* cleanup

* remove unnecessary comment

* add test for multiples on both sides

* add resolve_to_specific_version

* local registry

* hacking out deps

* Buck pkg mgmt (#645)

* only load hooks and archives once (#540)

* sets schema for node before parsing raw sql (#541)

* Fix/env vars (#543)

* fix for bad env_var exception

* overwrite target with compiled values

* fixes env vars, adds test. Auto-compile profile/target args

* improvements for code that runs in hooks (#544)

* improvements for code that runs in hooks

* fix error message note

* typo

* Update CHANGELOG.md

* bump version (#546)

* add scope to service account json creds initializer (#547)

* bump 0.9.0a3 --> 0.9.0a4 (#548)

* Fix README links (#554)

* Update README.md

* handle empty profiles.yml file (#555)

* return empty string (instead of None) to avoid polluting rendered sql (#566)

* tojson was added in jinja 2.9 (#563)

* tojson was added in jinja 2.9

* requirements

* fix package-defined schema test macros (#562)

* fix package-defined schema test macros

* create a dummy Relation in parsing

* fix for bq quoting (#565)

* bump snowflake, remove pyasn1 (#570)

* bump snowflake, remove pyasn1

* change requirements.txt

* allow macros to return non-text values (#571)

* revert jinja version, implement tojson hack (#572)

* bump to 090a5

* update changelog

* bump (#574)

* 090 docs (#575)

* 090 docs

* Update CHANGELOG.md

* Update CHANGELOG.md

* Raise CompilationException on duplicate model (#568)

* Raise CompilationException on duplicate model

Extend tests

* Ignore disabled models in parse_sql_nodes

Extend tests for duplicate model

* Fix preexisting models

* Use double quotes consistently

Rename model-1 to model-disabled

* Fix unit tests

* Raise exception on duplicate model across packages

Extend tests

* Make run_started_at timezone aware (#553) (#556)

* Make run_started_at timezone aware

Set run_started_at timezone to UTC
Enable timezone change in models
Extend requirements
Extend tests

* Address comments from code review

Create modules namespace to context
Move pytz to modules
Add new dependencies to setup.py

* Add warning for missing constraints. Fixes #592 (#600)

* Add warning for missing constraints. Fixes #592

* fix unit tests

* fix schema tests used in, or defined in packages (#599)

* fix schema tests used in, or defined in packages

* don't hardcode dbt test namespace

* fix/actually run tests

* rm junk

* run hooks in correct order, fixes #590 (#601)

* run hooks in correct order, fixes #590

* add tests

* fix tests

* pep8

* change req for snowflake to fix crypto install issue (#612)

From cffi callback <function _verify_callback at 0x06BF2978>:
Traceback (most recent call last):
  File "c:\projects\dbt\.tox\pywin\lib\site-packages\OpenSSL\SSL.py", line 313, in wrapper
    _lib.X509_up_ref(x509)
AttributeError: module 'lib' has no attribute 'X509_up_ref'
From cffi callback <function _verify_callback at 0x06B8CF60>:

* Update python version in Makefile from 3.5 to 3.6 (#613)

* Fix/snowflake custom schema (#626)

* Fixes already opened transaction issue

For #602

* Fixes #621

* Create schema in archival flow (#625)

* Fix for pre-hooks outside of transactions (#623)

* Fix for pre-hooks outside of transactions #576

* improve tests

* Fixes already opened transaction issue (#622)

For #602

* Accept string for postgres port number (#583) (#624)

* Accept string for postgres port number (#583)

* s/str/basestring/g

* print correct run time (include hooks) (#607)

* add support for late binding views (Redshift) (#614)

* add support for late binding views (Redshift)

* fix bind logic

* wip for get_columns_in_table

* fix get_columns_in_table

* fix for default value in bind config

* pep8

* skip tests that depend on nonexistent or disabled models (#617)

* skip tests that depend on nonexistent or disabled models

* pep8, Fixes #616

* refactor

* fix for adapter macro called within packages (#630)

* fix for adapter macro called within packages

* better error message

* Update CHANGELOG.md (#632)

* Update CHANGELOG.md

* Update CHANGELOG.md

* Bump version: 0.9.0 → 0.9.1

* more helpful exception for registry funcs

* Rework deps to support local & git

* pylint and cleanup

* make modules directory first

* Refactor registry client for cleanliness and better error handling

* init converter script

* create modules directory only if non-existent

* Only check the hub registry for registry packages

* Incorporate changes from Drew's branch

Diff of original changes:
https://github.com/fishtown-analytics/dbt/pull/591/files

* lint

* include a portion of the actual name in destination directory

* Install dependencies using actual name; better exceptions

* Error if two dependencies have same name

* Process dependencies one level at a time

Included in this change is a refactor of the deps run function for
clarity.

Also I changed the resolve_version function to update the object in
place. I prefer the immutability of this function as it was, but the
rest of the code doesn't really operate that way. And I ran into some
bugs due to this discrepancy.

* update var name

* Provide support for repositories in project yml

* Download files in a temp directory

The downloads directory causes problems with the run command because
this directory is not a dbt project. Need to download it elsewhere.

* pin some versions

* pep8-ify

* some PR feedback changes around logging

* PR feedback round 2

* Fix for redshift varchar bug (#647)

* Fix for redshift varchar bug

* pep8 on a sql string, smh

* Set global variable overrides on the command line with --vars (#640)

* Set global variable overrides on the command line with --vars

* pep8

* integration tests for cli vars

* Seed rewrite (#618)

* loader for seed data files

* Functioning rework of seed task

* Make CompilerRunner fns private and impl. SeedRunner.compile

Trying to distinguish between the public/private interface for this
class. And the SeedRunner doesn't need the functionality in the compile
function, it just needs a compile function to exist for use in the
compilation process.

* Test changes and fixes

* make the DB setup script usable locally

* convert simple copy test to use seeed

* Fixes to get Snowflake working

* New seed flag and make it non-destructive by default

* Convert update SQL script to another seed

* cleanup

* implement bigquery csv load

* context handling of StringIO

* Better typing

* strip seeder and csvkit dependency

* update bigquery to use new data typing and to fix unicode issue

* update seed test

* fix abstract functions in base adapter

* support time type

* try pinning crypto, pyopenssl versions

* remove unnecessary version pins

* insert all at once, rather than one query per row

* do not quote field names on creation

* bad

* quiet down parsedatetime logger

* pep8

* UI updates + node conformity for seed nodes

* add seed to list of resource types, cleanup

* show option for CSVs

* typo

* pep8

* move agate import to avoid strange warnings

* deprecation warning for --drop-existing

* quote column names in seed files

* revert quoting change (breaks Snowflake). Hush warnings

* use hub url

* Show installed version, silence semver regex warnings

* sort versions to make tests deterministic. Prefer higher versions

* pep8, fix comparison functions for py3

* make compare function return value in {-1, 0, 1}

* fix for deleting git dirs on windows?

* use system client rmdir instead of shutil directly

* debug logging to identify appveyor issue

* less restrictive error retry

* rm debug logging

* s/version/revision for git packages

* more s/version/revision, deprecation cleanup

* remove unused semver codepath

* plus symlinks!!!

* get rid of reference to removed function
@drewbanin drewbanin added this to the 0.9.2 milestone Mar 2, 2018
iknox-fa pushed a commit that referenced this pull request Feb 8, 2022
* semver resolution

* cleanup

* remove unnecessary comment

* add test for multiples on both sides

* add resolve_to_specific_version

* local registry

* hacking out deps

* Buck pkg mgmt (#645)

* only load hooks and archives once (#540)

* sets schema for node before parsing raw sql (#541)

* Fix/env vars (#543)

* fix for bad env_var exception

* overwrite target with compiled values

* fixes env vars, adds test. Auto-compile profile/target args

* improvements for code that runs in hooks (#544)

* improvements for code that runs in hooks

* fix error message note

* typo

* Update CHANGELOG.md

* bump version (#546)

* add scope to service account json creds initializer (#547)

* bump 0.9.0a3 --> 0.9.0a4 (#548)

* Fix README links (#554)

* Update README.md

* handle empty profiles.yml file (#555)

* return empty string (instead of None) to avoid polluting rendered sql (#566)

* tojson was added in jinja 2.9 (#563)

* tojson was added in jinja 2.9

* requirements

* fix package-defined schema test macros (#562)

* fix package-defined schema test macros

* create a dummy Relation in parsing

* fix for bq quoting (#565)

* bump snowflake, remove pyasn1 (#570)

* bump snowflake, remove pyasn1

* change requirements.txt

* allow macros to return non-text values (#571)

* revert jinja version, implement tojson hack (#572)

* bump to 090a5

* update changelog

* bump (#574)

* 090 docs (#575)

* 090 docs

* Update CHANGELOG.md

* Update CHANGELOG.md

* Raise CompilationException on duplicate model (#568)

* Raise CompilationException on duplicate model

Extend tests

* Ignore disabled models in parse_sql_nodes

Extend tests for duplicate model

* Fix preexisting models

* Use double quotes consistently

Rename model-1 to model-disabled

* Fix unit tests

* Raise exception on duplicate model across packages

Extend tests

* Make run_started_at timezone aware (#553) (#556)

* Make run_started_at timezone aware

Set run_started_at timezone to UTC
Enable timezone change in models
Extend requirements
Extend tests

* Address comments from code review

Create modules namespace to context
Move pytz to modules
Add new dependencies to setup.py

* Add warning for missing constraints. Fixes #592 (#600)

* Add warning for missing constraints. Fixes #592

* fix unit tests

* fix schema tests used in, or defined in packages (#599)

* fix schema tests used in, or defined in packages

* don't hardcode dbt test namespace

* fix/actually run tests

* rm junk

* run hooks in correct order, fixes #590 (#601)

* run hooks in correct order, fixes #590

* add tests

* fix tests

* pep8

* change req for snowflake to fix crypto install issue (#612)

From cffi callback <function _verify_callback at 0x06BF2978>:
Traceback (most recent call last):
  File "c:\projects\dbt\.tox\pywin\lib\site-packages\OpenSSL\SSL.py", line 313, in wrapper
    _lib.X509_up_ref(x509)
AttributeError: module 'lib' has no attribute 'X509_up_ref'
From cffi callback <function _verify_callback at 0x06B8CF60>:

* Update python version in Makefile from 3.5 to 3.6 (#613)

* Fix/snowflake custom schema (#626)

* Fixes already opened transaction issue

For #602

* Fixes #621

* Create schema in archival flow (#625)

* Fix for pre-hooks outside of transactions (#623)

* Fix for pre-hooks outside of transactions #576

* improve tests

* Fixes already opened transaction issue (#622)

For #602

* Accept string for postgres port number (#583) (#624)

* Accept string for postgres port number (#583)

* s/str/basestring/g

* print correct run time (include hooks) (#607)

* add support for late binding views (Redshift) (#614)

* add support for late binding views (Redshift)

* fix bind logic

* wip for get_columns_in_table

* fix get_columns_in_table

* fix for default value in bind config

* pep8

* skip tests that depend on nonexistent or disabled models (#617)

* skip tests that depend on nonexistent or disabled models

* pep8, Fixes #616

* refactor

* fix for adapter macro called within packages (#630)

* fix for adapter macro called within packages

* better error message

* Update CHANGELOG.md (#632)

* Update CHANGELOG.md

* Update CHANGELOG.md

* Bump version: 0.9.0 → 0.9.1

* more helpful exception for registry funcs

* Rework deps to support local & git

* pylint and cleanup

* make modules directory first

* Refactor registry client for cleanliness and better error handling

* init converter script

* create modules directory only if non-existent

* Only check the hub registry for registry packages

* Incorporate changes from Drew's branch

Diff of original changes:
https://github.com/fishtown-analytics/dbt/pull/591/files

* lint

* include a portion of the actual name in destination directory

* Install dependencies using actual name; better exceptions

* Error if two dependencies have same name

* Process dependencies one level at a time

Included in this change is a refactor of the deps run function for
clarity.

Also I changed the resolve_version function to update the object in
place. I prefer the immutability of this function as it was, but the
rest of the code doesn't really operate that way. And I ran into some
bugs due to this discrepancy.

* update var name

* Provide support for repositories in project yml

* Download files in a temp directory

The downloads directory causes problems with the run command because
this directory is not a dbt project. Need to download it elsewhere.

* pin some versions

* pep8-ify

* some PR feedback changes around logging

* PR feedback round 2

* Fix for redshift varchar bug (#647)

* Fix for redshift varchar bug

* pep8 on a sql string, smh

* Set global variable overrides on the command line with --vars (#640)

* Set global variable overrides on the command line with --vars

* pep8

* integration tests for cli vars

* Seed rewrite (#618)

* loader for seed data files

* Functioning rework of seed task

* Make CompilerRunner fns private and impl. SeedRunner.compile

Trying to distinguish between the public/private interface for this
class. And the SeedRunner doesn't need the functionality in the compile
function, it just needs a compile function to exist for use in the
compilation process.

* Test changes and fixes

* make the DB setup script usable locally

* convert simple copy test to use seeed

* Fixes to get Snowflake working

* New seed flag and make it non-destructive by default

* Convert update SQL script to another seed

* cleanup

* implement bigquery csv load

* context handling of StringIO

* Better typing

* strip seeder and csvkit dependency

* update bigquery to use new data typing and to fix unicode issue

* update seed test

* fix abstract functions in base adapter

* support time type

* try pinning crypto, pyopenssl versions

* remove unnecessary version pins

* insert all at once, rather than one query per row

* do not quote field names on creation

* bad

* quiet down parsedatetime logger

* pep8

* UI updates + node conformity for seed nodes

* add seed to list of resource types, cleanup

* show option for CSVs

* typo

* pep8

* move agate import to avoid strange warnings

* deprecation warning for --drop-existing

* quote column names in seed files

* revert quoting change (breaks Snowflake). Hush warnings

* use hub url

* Show installed version, silence semver regex warnings

* sort versions to make tests deterministic. Prefer higher versions

* pep8, fix comparison functions for py3

* make compare function return value in {-1, 0, 1}

* fix for deleting git dirs on windows?

* use system client rmdir instead of shutil directly

* debug logging to identify appveyor issue

* less restrictive error retry

* rm debug logging

* s/version/revision for git packages

* more s/version/revision, deprecation cleanup

* remove unused semver codepath

* plus symlinks!!!

* get rid of reference to removed function


automatic commit by git-black, original commits:
  5fbcd12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants