Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make current dependencies optional #63

Closed
qlquanle opened this issue Aug 9, 2017 · 8 comments
Closed

Make current dependencies optional #63

qlquanle opened this issue Aug 9, 2017 · 8 comments
Assignees
Milestone

Comments

@qlquanle
Copy link
Contributor

qlquanle commented Aug 9, 2017

  • The only true dependencies are Python (2.7.X), GSLab Python (4.1.0), PyYAML, and SCons.
  • Create switches to implement our automated checks of other software and run example scripts.
    • Comment out lines in SConstruct or SConscript.
    • Add instructions on turning on dependencies in readme.
@yuchuan2016
Copy link
Contributor

I think git-lfs is also a true dependency. Without git-lfs, we even cannot clone this repo.

Some other things:

  • Shall we make lyx also a true dependency? I would vote for yes. We need at least one of lyx/tex to run paper_slides directory. We can say in README that users can turn off lyx and turn on tex, but the default one is lyx. (The default software for analysis directory is python)
  • Shall we write check_matlab and check_latex in configuration_test.py now?
  • Shall we write scanner for latex, similar to lyx now?

@stanfordquan @Shun-Yang , FYI if you have any opinion on above.

@Shun-Yang
Copy link

My suggestion is to make git-lfs and lyx as dependency, and postpone writing check_matlab, check_latex and latex_scanner to the end and see if we have time in this sprint to finish them. Of course, if you think they are relatively easy to do, then feel free to get this done now.

@qlquanle
Copy link
Contributor Author

qlquanle commented Aug 28, 2017

I vote making a note and delaying check_matlab and check_latex, same for latex_scanner.

I vote making TeX a dependency over LyX (smaller footprint, and a lot more people use TeX anecdotally). I think PIs may prefer LyX though.

@yuchuan2016
Copy link
Contributor

Another potential problem is, if users simply write a Stata do file without turning on the stata dependency in config_global.yaml (say, if they uncomment the stata block in analysis/source/prepare_data/SConcsript, then configuration_test won't check stata requirements. Then the build may fail. This may violate the spirit that we want to catch all errors before users actually run the code.

@yuchuan2016
Copy link
Contributor

@gentzkow , per team discussion, we have several options to specify software dependencies. We want your opinion over these options.

  • Use the current structure. Specify dependencies in config_global.yaml. Check requirements only for softwares that have an entry Yes. The disadvantage is users may not be conscious enough to change it when using a new software.
  • Add checks as a part of the builders when they are defined in SConstruct by Builder(action = gs.build_xx). Comment out lines that are not required. If a user want to use another builder, they need to uncomment the relevant line, and the builder will automatically do the check.
  • Check the extensions of scripts in the repo that will be executed. Turn of the dependencies accordingly. This may involve a decent amount of work.

@gentzkow
Copy link

Thanks.

The second option sounds good to me if it can work robustly (i.e., if I uncomment a builder and the software isn't installed I'll always get a nice error; if I don't uncomment a builder but try to run steps that require that software I get an error as well (maybe a bit less nice))

@gentzkow
Copy link

(Otherwise, first option is fine)

yuchuan2016 added a commit that referenced this issue Aug 29, 2017
yuchuan2016 added a commit that referenced this issue Aug 30, 2017
@yuchuan2016
Copy link
Contributor

Pull request created in #78.

yuchuan2016 added a commit that referenced this issue Aug 31, 2017
yuchuan2016 added a commit that referenced this issue Aug 31, 2017
yuchuan2016 added a commit that referenced this issue Aug 31, 2017
yuchuan2016 added a commit that referenced this issue Aug 31, 2017
* #63 add builder check

* #63 add autimatic config scripts

* #63 response to PR

* #63 change the default to not upgrade

* #63 force installing of gslab_python

* #63 small change per PR

* #63 small change
@qlquanle qlquanle mentioned this issue Sep 5, 2017
qlquanle pushed a commit that referenced this issue Mar 3, 2018
* Pull request for #62: Separate sub-directory for analysis and paper/slides (#72)

* #62 reorganize repo

* #62 add provenance

* #62 allow provenance to check missing files

* #72 response to PR and comments

* #72 small name change

* Pull request for #67: Create release/lg for large files (#73)

* #67 add code to fake lg dataset, add lg to gitignore

* #67 add lg protocol to readme.md

* Update constants.yaml

* #67 blow up data file to do large release

* #67 change hist in default stata code to limit number of pins

* #67 delete old logs

* #67 readme change

* #67 manually copy new analysis files to paper input

* rerun paper-slides

* Pull request: move top level logs to release (#75)

* #61 first pass

* #61 second pass

* #61 switch to single quote

* #93 aux first pass

* #93 first pass

* #93 second pass

* Pull request for #61: restructure YAML in template  (#74)

* #61 first pass

* #61 second pass

* #61 switch to single quote

* #61 rm config_template

* add instruction on switching required softwares

* rm unused config_user_template

* #97 testing anything template

* gslab-python#97 demonstration scripts

* Pull Request for #66: Force integration with gslab_stata (#77)

* #66 revise stata scripts

* #66 modify README

* #66 small change in readme

* #66 add note in README

* gslab-python#97 change generator syntax

* gslab-python#97 respond to PR

* Pull Request for #64: Port example scripts to multiple languages (#76)

* #64 add examples for python stata matlab R, mute everything except python

* #64 add example of latex

* #64 modify dependencies

* #64 turn on git-lfs

* #64 response to PR

* #64 update readme and modify sample code

* solve conflict after merge

* rm accidental commit of local scripts

* Pull Request for #63: Make current dependencies optional (#78)

* #63 add builder check

* #63 add autimatic config scripts

* #63 response to PR

* #63 change the default to not upgrade

* #63 force installing of gslab_python

* #63 small change per PR

* #63 small change

* Pull request for creating size warning when git-lfs not presented (#79)

* first pass on template

* add in size warning for paper dir

* fix bugs

* add in option for git attributes

* add in paper dir

* Pull request for documenting and organizing debrief  (#81)

* #69 first pass

* implement PR

* small edits

* small edits

* Pull request for gslab_python#99: Add cache reconfiguration code (#83)

* gslab-econ/gslab_python#99 switch to scons local

* gslab-econ/gslab_python#99 delete scons local, lfs track .zip

* gslab-econ/gslab_python#99 commit scons-local-2.5.1

* gslab-econ/gslab_python#99 scons python scripts for each sub

* gslab-econ/gslab_python#99 remove top-level scons.zip

* #83 gslab-econ/gslab_python#99 update commands in readme.

* 83 gslab-econ/gslab_python#99 symlink to scons.py

* #83 gslab-econ/gslab_python#83 scons.py -> run.py, mention global scons in readme

* #83 gslab-econ/gslab_python#99 arg in main call

* Pull request for #82: Update docs for 4.1.0  (#84)

* #82 readme for 4.1.0, fix scons in paper_slides, build_anything in analysis

* #82 update data download procedure

* #82 add jupyter notebooks to .gitignore

* #82 build_anything demonstration

* #82 output

* #82 uncomment builder

* #84 #82 instructions to connect analysis and paper_slides

* #84 #82 respond to PR

* #84 #82 more PR

* change template

* Pull request for #86: Integrate make_provenance into SConstruct (#87)

* #86 incorporate external logging from gslab-econ/gslab_python#131

* #87 #86 PR

* #132 #131 delete stray file

* #87 #86 comment out some builders in analysis/SConstruct

* Pull request for #88: Grab all paths when recording inputs (#89)

* #88 allow inputs specificed as str or dict

* #88 fix sconscript after flatten input in config_global

* pep8 e713 change

* Pull request for #90: Load PYTHONPATH in SCons ENV (#91)

* #90 store pythonpath

* #90 remove stray pythonpath definition

* Fix location of sconstruct.log.

* Capitalization in readme

* Pull request for #92: Turn off default logging during SCons dry-run (#93)

* #92 turn off logging on dry run, uses gslab-econ/gslab_python#141

* #93 #92 move dry-run checking for logs into gslab_python

* Fix typo in readme

* #95 switch to scons 3 (#97)

* Update SCons local version in README

* practice task

* realized my mistake and reverting

* Pull request for #96: Complete MG points 1-5  (#98)

* #95 switch to scons 3

* .#96 update to new builders and config and rerun

* #96 drop _ before anything builder

* #98 #96 rerun with new builders

* #98 #96 typo in README

* #98 #96 better advertise package dependencies in config scripts

* #98 #96 mention reset config_user in README FAQ
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants