Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for generating Who Owns What tables and functions #22

Merged
merged 8 commits into from
Feb 11, 2019

Conversation

toolness
Copy link
Contributor

@toolness toolness commented Feb 8, 2019

Fixes #19 by putting Who Owns What data and functions in a wow schema. Note that any clients who want to use functions defined in the schema will need to SET search_path TO wow, public or else the functions will either not be found, or crash.

Like NYCDB, the version of Who Owns What is specified via a git revision in the Dockerfile.

Notes

  • I opted to just add a new wowutil.py tool to deal with WoW data instead of the alternative, which was to decouple our concept of a "dataset" from NYCDB so we could then turn WoW into its own dataset. That would be a ton of work and I'm not sure if it's even the right decision architecturally, since WoW differs in a lot of ways from a standard NYCDB dataset (it has dependencies, it doesn't have any files that need to be downloaded from the internet, etc).

  • Until now, the database wasn't actually being reset between tests, which made them run faster but wasn't good for test isolation. Now the DB is wiped between tests, which unfortunately increases the test suite time from ~30s on CircleCI to ~45s. Oy. But I think it's for the best, and we don't really anticipate this codebase growing a lot in the future (and if it does, I'm sure we can figure out optimizations to make this take less time, e.g. wrapping tests in one big nested transaction or whatever it is that Django does).

  • Until now, functions created in the SQL scripts for datasets weren't making their way into the public schema. They are now!

  • Currently the test suite ensures that the version of NYCDB we're using is compatible with the version that WoW expects, and I think it's enough for now, but if it's not, we might want to revisit this later by e.g. making WoW itself a python package that's pinned to a particular version/range of NYCDB or something.

To do

  • There's constants in wowutil.py that really need to be moved over to the WoW repository and imported from there. Merge Move WoW db config out to a YAML file. who-owns-what#102, which moves the metadata needed to properly install WoW over to the WoW repository. Then point WOW_REV in the Dockerfile to the appropriate revision.
  • Make the test verify that wow_bldgs is created in the wow schema.
  • Make the test verify that functions are created in the wow schema.
  • Document wowutil.py in the README.
  • Add some kind of command to allow the running of wowutil.py build to be scheduled on AWS Fargate. Or perhaps just document how to do this manually for now.

@toolness toolness changed the title [WIP] Add support for generating Who Owns What tables [WIP] Add support for generating Who Owns What tables and functions Feb 8, 2019
@toolness toolness changed the title [WIP] Add support for generating Who Owns What tables and functions Add support for generating Who Owns What tables and functions Feb 11, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant