Pull request for gslab_python#99: Add cache reconfiguration code #83

arosenbe · 2017-08-31T20:22:48Z

Original issue gslab-econ/gslab_python#99

@yuchuan2016, can you please PR?

yuchuan2016 · 2017-08-31T21:30:07Z

README.md

-        scons
+2. Unzip `scons.zip` to `scons`.
+    ```bash
+    unzip scons.zip -d scons


I think Windows does not support unzip command.

You're right, and I was torn on whether to include code for this step. I like the code because it shows exactly where we unzip to. Maybe a comment that says not windows? I'd also get rid of the code entirely if you think that's smarter.

I think it's good to have the code. Just make a note perhaps.

yuchuan2016 · 2017-08-31T21:33:52Z

@arosenbe , great! I only have one minor issue about unzip command above.

Another question, should we put scons files in config, or make it a separate directory? If a separate directory, we would have four top-level directories (analysis/paper_slides/config/scons), which does not look very nice...

arosenbe · 2017-08-31T21:36:54Z

That's a good point about the directory structure @yuchuan2016. I think config could be a logical place for the scons directory. My worry with putting it there is the length of the command needed to run scons

python ../config/scons/scons.py

What do you think?

arosenbe · 2017-08-31T21:41:54Z

@yuchuan2016,

Another option would be to edit the paths in scons.py to point to a scons-local directory stored in config. Let me show you what that looks like.

yuchuan2016 · 2017-08-31T21:42:57Z

@arosenbe , yes I agree that the command would become longer. Can we store the scons.py directly in config/, and everything else in config/scons?

arosenbe · 2017-08-31T22:09:00Z

@yuchuan2016, I can do that. We could also keep separate scons.py scripts in analysis and paper_slides to cut down on notation even more. Which do you prefer?

yuchuan2016 · 2017-08-31T22:11:59Z

I'm fine either way.

arosenbe · 2017-08-31T23:05:18Z

@yuchuan2016, can you take a look at the new commit? I wrote python scripts called scons.py to call the scons.py included in the scons-local distribution.

yuchuan2016

@arosenbe , I have just tried another way. We can create a symlink in /analysis/ by typing ln -s ../config/scons/scons.py scons.py. This can prevent keeping duplicated files. What do you think?

yuchuan2016 · 2017-08-31T23:12:21Z

analysis/scons.py

+    cl_args = ' '.join(sys.argv)
+
+    # Create call
+    call = 'python ../config/scons/scons.py %s' % cl_args


Shall we make the path an argument?

Not relevent, switching to symlink, but a good suggestion if it stayed relevant.

arosenbe · 2017-08-31T23:44:12Z

@yuchuan2016 I really like the symlink idea. Will implement.

qlquanle · 2017-08-31T23:56:22Z

Are symlinks platform-independent?

arosenbe · 2017-09-01T00:15:21Z

Symlinks are not platform independent 😿

…ns in readme

arosenbe · 2017-09-01T00:47:02Z

@yuchuan2016, since symlinks are not platform independent, I switched back to the wrappers around scons.py. The difference is that these are called run.py. I also put information about how to use a global scons installation in the readme. Can you take a look?

yuchuan2016

I only have one comment below. After that feel free to merge

yuchuan2016 · 2017-09-01T03:45:05Z

analysis/run.py

+    # Execute
+    subprocess.call(call, shell = True)
+
+main()


There should be an argument in the main function. Same for the other.

😱 You're right!

* Pull request for #62: Separate sub-directory for analysis and paper/slides (#72) * #62 reorganize repo * #62 add provenance * #62 allow provenance to check missing files * #72 response to PR and comments * #72 small name change * Pull request for #67: Create release/lg for large files (#73) * #67 add code to fake lg dataset, add lg to gitignore * #67 add lg protocol to readme.md * Update constants.yaml * #67 blow up data file to do large release * #67 change hist in default stata code to limit number of pins * #67 delete old logs * #67 readme change * #67 manually copy new analysis files to paper input * rerun paper-slides * Pull request: move top level logs to release (#75) * #61 first pass * #61 second pass * #61 switch to single quote * #93 aux first pass * #93 first pass * #93 second pass * Pull request for #61: restructure YAML in template (#74) * #61 first pass * #61 second pass * #61 switch to single quote * #61 rm config_template * add instruction on switching required softwares * rm unused config_user_template * #97 testing anything template * gslab-python#97 demonstration scripts * Pull Request for #66: Force integration with gslab_stata (#77) * #66 revise stata scripts * #66 modify README * #66 small change in readme * #66 add note in README * gslab-python#97 change generator syntax * gslab-python#97 respond to PR * Pull Request for #64: Port example scripts to multiple languages (#76) * #64 add examples for python stata matlab R, mute everything except python * #64 add example of latex * #64 modify dependencies * #64 turn on git-lfs * #64 response to PR * #64 update readme and modify sample code * solve conflict after merge * rm accidental commit of local scripts * Pull Request for #63: Make current dependencies optional (#78) * #63 add builder check * #63 add autimatic config scripts * #63 response to PR * #63 change the default to not upgrade * #63 force installing of gslab_python * #63 small change per PR * #63 small change * Pull request for creating size warning when git-lfs not presented (#79) * first pass on template * add in size warning for paper dir * fix bugs * add in option for git attributes * add in paper dir * Pull request for documenting and organizing debrief (#81) * #69 first pass * implement PR * small edits * small edits * Pull request for gslab_python#99: Add cache reconfiguration code (#83) * gslab-econ/gslab_python#99 switch to scons local * gslab-econ/gslab_python#99 delete scons local, lfs track .zip * gslab-econ/gslab_python#99 commit scons-local-2.5.1 * gslab-econ/gslab_python#99 scons python scripts for each sub * gslab-econ/gslab_python#99 remove top-level scons.zip * #83 gslab-econ/gslab_python#99 update commands in readme. * 83 gslab-econ/gslab_python#99 symlink to scons.py * #83 gslab-econ/gslab_python#83 scons.py -> run.py, mention global scons in readme * #83 gslab-econ/gslab_python#99 arg in main call * Pull request for #82: Update docs for 4.1.0 (#84) * #82 readme for 4.1.0, fix scons in paper_slides, build_anything in analysis * #82 update data download procedure * #82 add jupyter notebooks to .gitignore * #82 build_anything demonstration * #82 output * #82 uncomment builder * #84 #82 instructions to connect analysis and paper_slides * #84 #82 respond to PR * #84 #82 more PR * change template * Pull request for #86: Integrate make_provenance into SConstruct (#87) * #86 incorporate external logging from gslab-econ/gslab_python#131 * #87 #86 PR * #132 #131 delete stray file * #87 #86 comment out some builders in analysis/SConstruct * Pull request for #88: Grab all paths when recording inputs (#89) * #88 allow inputs specificed as str or dict * #88 fix sconscript after flatten input in config_global * pep8 e713 change * Pull request for #90: Load PYTHONPATH in SCons ENV (#91) * #90 store pythonpath * #90 remove stray pythonpath definition * Fix location of sconstruct.log. * Capitalization in readme * Pull request for #92: Turn off default logging during SCons dry-run (#93) * #92 turn off logging on dry run, uses gslab-econ/gslab_python#141 * #93 #92 move dry-run checking for logs into gslab_python * Fix typo in readme * #95 switch to scons 3 (#97) * Update SCons local version in README * practice task * realized my mistake and reverting * Pull request for #96: Complete MG points 1-5 (#98) * #95 switch to scons 3 * .#96 update to new builders and config and rerun * #96 drop _ before anything builder * #98 #96 rerun with new builders * #98 #96 typo in README * #98 #96 better advertise package dependencies in config scripts * #98 #96 mention reset config_user in README FAQ

arosenbe added 3 commits August 31, 2017 13:15

gslab-econ/gslab_python#99 switch to scons local

998255d

gslab-econ/gslab_python#99 delete scons local, lfs track .zip

069022e

gslab-econ/gslab_python#99 commit scons-local-2.5.1

9aae76e

arosenbe requested a review from yuchuan2016 August 31, 2017 20:23

arosenbe mentioned this pull request Aug 31, 2017

Add cache reconfiguration code gslab-econ/gslab_python#99

Closed

yuchuan2016 reviewed Aug 31, 2017

View reviewed changes

gslab-econ/gslab_python#99 scons python scripts for each sub

8e4bbf0

gslab-econ/gslab_python#99 remove top-level scons.zip

bf5b883

yuchuan2016 reviewed Aug 31, 2017

View reviewed changes

#83 gslab-econ/gslab_python#99 update commands in readme.

46bd41f

83 gslab-econ/gslab_python#99 symlink to scons.py

1a53f37

arosenbe added 2 commits August 31, 2017 17:36

#83 gslab-econ/gslab_python#83 scons.py -> run.py, mention global sco…

27b0f52

…ns in readme

#83 gslab-econ/gslab_python#99 merge in commit w/ scons.py and rm 'em

90dde74

yuchuan2016 reviewed Sep 1, 2017

View reviewed changes

#83 gslab-econ/gslab_python#99 arg in main call

6fe1434

arosenbe merged commit 8c91db5 into 4.1.0 Sep 1, 2017

arosenbe deleted the gslab_python#99_cache branch September 1, 2017 15:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pull request for gslab_python#99: Add cache reconfiguration code #83

Pull request for gslab_python#99: Add cache reconfiguration code #83

arosenbe commented Aug 31, 2017

yuchuan2016 Aug 31, 2017

arosenbe Aug 31, 2017

yuchuan2016 Aug 31, 2017

yuchuan2016 commented Aug 31, 2017

arosenbe commented Aug 31, 2017

arosenbe commented Aug 31, 2017

yuchuan2016 commented Aug 31, 2017

arosenbe commented Aug 31, 2017

yuchuan2016 commented Aug 31, 2017

arosenbe commented Aug 31, 2017

yuchuan2016 left a comment

yuchuan2016 Aug 31, 2017

arosenbe Aug 31, 2017

arosenbe commented Aug 31, 2017

qlquanle commented Aug 31, 2017

arosenbe commented Sep 1, 2017

arosenbe commented Sep 1, 2017

yuchuan2016 left a comment

yuchuan2016 Sep 1, 2017

arosenbe Sep 1, 2017

Pull request for gslab_python#99: Add cache reconfiguration code #83

Pull request for gslab_python#99: Add cache reconfiguration code #83

Conversation

arosenbe commented Aug 31, 2017

yuchuan2016 Aug 31, 2017

Choose a reason for hiding this comment

arosenbe Aug 31, 2017

Choose a reason for hiding this comment

yuchuan2016 Aug 31, 2017

Choose a reason for hiding this comment

yuchuan2016 commented Aug 31, 2017

arosenbe commented Aug 31, 2017

arosenbe commented Aug 31, 2017

yuchuan2016 commented Aug 31, 2017

arosenbe commented Aug 31, 2017

yuchuan2016 commented Aug 31, 2017

arosenbe commented Aug 31, 2017

yuchuan2016 left a comment

Choose a reason for hiding this comment

yuchuan2016 Aug 31, 2017

Choose a reason for hiding this comment

arosenbe Aug 31, 2017

Choose a reason for hiding this comment

arosenbe commented Aug 31, 2017

qlquanle commented Aug 31, 2017

arosenbe commented Sep 1, 2017

arosenbe commented Sep 1, 2017

yuchuan2016 left a comment

Choose a reason for hiding this comment

yuchuan2016 Sep 1, 2017

Choose a reason for hiding this comment

arosenbe Sep 1, 2017

Choose a reason for hiding this comment