Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move WoW db config out to a YAML file. #102

Merged
merged 2 commits into from
Feb 11, 2019
Merged

Move WoW db config out to a YAML file. #102

merged 2 commits into from
Feb 11, 2019

Conversation

toolness
Copy link
Contributor

We need to be able to load/update the WoW database from our containerized DB updater at JustFixNYC/nycdb-k8s-loader#22. One simple way to do this is by putting all our database configuration into a YAML file, similar to what NYCDB is already doing.

This PR moves the names of all the NYCDB dataset dependencies and the SQL filenames into a new who-owns-what.yml file. It also makes the following changes:

  • We no longer load the pluto_17v1 dataset; it seems this was entirely superseded by pluto_18v1, although I should check with @sraby to make sure this is OK.

  • The --update option for dbtool.py builddb now wipes all nycdb data instead of just a subset of tables. This simplified the structure of the YAML and seems OK since nycdb-k8s-loader will now be responsible for updating WoW in a more rigorous manner. If we really want to add this feature back in, we might want to add a separate flag that allows the user to manually specify which datasets to refresh.

  • It automatically figures out what the tables in a dataset are by consulting the nycdb module.

@sraby
Copy link
Member

sraby commented Feb 11, 2019

Checked: we only use pluto_18v1 so removing those dependent datasets is great!

@sraby sraby merged commit db45d61 into master Feb 11, 2019
@sraby sraby deleted the yaml-config branch February 11, 2019 17:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants