4.1.0 Release (#85)

* Pull request for #62: Separate sub-directory for analysis and paper/slides (#72)

* #62 reorganize repo

* #62 add provenance

* #62 allow provenance to check missing files

* #72 response to PR and comments

* #72 small name change

* Pull request for #67: Create release/lg for large files (#73)

* #67 add code to fake lg dataset, add lg to gitignore

* #67 add lg protocol to readme.md

* Update constants.yaml

* #67 blow up data file to do large release

* #67 change hist in default stata code to limit number of pins

* #67 delete old logs

* #67 readme change

* #67 manually copy new analysis files to paper input

* rerun paper-slides

* Pull request: move top level logs to release (#75)

* #61 first pass

* #61 second pass

* #61 switch to single quote

* #93 aux first pass

* #93 first pass

* #93 second pass

* Pull request for #61: restructure YAML in template  (#74)

* #61 first pass

* #61 second pass

* #61 switch to single quote

* #61 rm config_template

* add instruction on switching required softwares

* rm unused config_user_template

* #97 testing anything template

* gslab-python#97 demonstration scripts

* Pull Request for #66: Force integration with gslab_stata (#77)

* #66 revise stata scripts

* #66 modify README

* #66 small change in readme

* #66 add note in README

* gslab-python#97 change generator syntax

* gslab-python#97 respond to PR

* Pull Request for #64: Port example scripts to multiple languages (#76)

* #64 add examples for python stata matlab R, mute everything except python

* #64 add example of latex

* #64 modify dependencies

* #64 turn on git-lfs

* #64 response to PR

* #64 update readme and modify sample code

* solve conflict after merge

* rm accidental commit of local scripts

* Pull Request for #63: Make current dependencies optional (#78)

* #63 add builder check

* #63 add autimatic config scripts

* #63 response to PR

* #63 change the default to not upgrade

* #63 force installing of gslab_python

* #63 small change per PR

* #63 small change

* Pull request for creating size warning when git-lfs not presented (#79)

* first pass on template

* add in size warning for paper dir

* fix bugs

* add in option for git attributes

* add in paper dir

* Pull request for documenting and organizing debrief  (#81)

* #69 first pass

* implement PR

* small edits

* small edits

* Pull request for gslab_python#99: Add cache reconfiguration code (#83)

* gslab-econ/gslab_python#99 switch to scons local

* gslab-econ/gslab_python#99 delete scons local, lfs track .zip

* gslab-econ/gslab_python#99 commit scons-local-2.5.1

* gslab-econ/gslab_python#99 scons python scripts for each sub

* gslab-econ/gslab_python#99 remove top-level scons.zip

* #83 gslab-econ/gslab_python#99 update commands in readme.

* 83 gslab-econ/gslab_python#99 symlink to scons.py

* #83 gslab-econ/gslab_python#83 scons.py -> run.py, mention global scons in readme

* #83 gslab-econ/gslab_python#99 arg in main call

* Pull request for #82: Update docs for 4.1.0  (#84)

* #82 readme for 4.1.0, fix scons in paper_slides, build_anything in analysis

* #82 update data download procedure

* #82 add jupyter notebooks to .gitignore

* #82 build_anything demonstration

* #82 output

* #82 uncomment builder

* #84 #82 instructions to connect analysis and paper_slides

* #84 #82 respond to PR

* #84 #82 more PR

* change template

* Pull request for #86: Integrate make_provenance into SConstruct (#87)

* #86 incorporate external logging from gslab-econ/gslab_python#131

* #87 #86 PR

* #132 #131 delete stray file

* #87 #86 comment out some builders in analysis/SConstruct

* Pull request for #88: Grab all paths when recording inputs (#89)

* #88 allow inputs specificed as str or dict

* #88 fix sconscript after flatten input in config_global

* pep8 e713 change

* Pull request for #90: Load PYTHONPATH in SCons ENV (#91)

* #90 store pythonpath

* #90 remove stray pythonpath definition

* Fix location of sconstruct.log.

* Capitalization in readme

* Pull request for #92: Turn off default logging during SCons dry-run (#93)

* #92 turn off logging on dry run, uses gslab-econ/gslab_python#141

* #93 #92 move dry-run checking for logs into gslab_python

* Fix typo in readme

* #95 switch to scons 3 (#97)

* Update SCons local version in README

* practice task

* realized my mistake and reverting

* Pull request for #96: Complete MG points 1-5  (#98)

* #95 switch to scons 3

* .#96 update to new builders and config and rerun

* #96 drop _ before anything builder

* #98 #96 rerun with new builders

* #98 #96 typo in README

* #98 #96 better advertise package dependencies in config scripts

* #98 #96 mention reset config_user in README FAQ

.gitattributes

@@ -3,3 +3,5 @@
 *.rds filter=lfs diff=lfs merge=lfs -text
 *.pdf filter=lfs diff=lfs merge=lfs -text
 *.png filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+release/sconstruct.log filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -1,12 +1,19 @@
-user-config.yaml
-/build/
+*config_user.yaml
+analysis/build/
+analysis/release/lg/
+paper_slides/build/
+paper_slides/input/lg
+paper_slides/release/lg/
+external/
+temp/
+scons/
+*.sconsign.dblite
 *.pyc
 *.lyx~
 *.lyx#
+*.lyx.emergency
 *.Rhistory
 *.Rapp.history
-/external/
-/temp/
 *.DS_Store
 *.aux
 *.fls
@@ -17,6 +24,4 @@ user-config.yaml
 *.toc
 *.out
 *.svn
-.sconsign.dblite
-*.lyx.emergency
-
+*.ipynb

analysis/SConstruct

@@ -0,0 +1,52 @@
+# Preliminaries
+import os
+import sys
+import atexit
+import yaml
+
+sys.path.append('../config')
+sys.dont_write_bytecode = True # Don't write .pyc files
+
+# Setup
+from configuration import configuration
+[mode, cache_dir, CONFIG, executable_names, prereqs, pythonpath] = configuration(ARGUMENTS)
+import gslab_scons as gs
+
+# Define the SCons environment
+env = Environment(ENV = {'PATH': os.environ['PATH'], 'PYTHONPATH': pythonpath},
+                  IMPLICIT_COMMAND_DEPENDENCIES = 0,
+                  BUILDERS = {
+                              # 'BuildMatlab': Builder(action = gs.build_matlab),
+                              'BuildPython': Builder(action = gs.build_python),
+                              # 'BuildR'     : Builder(action = gs.build_r),
+                              # 'BuildStata' : Builder(action = gs.build_stata),
+                              })
+# Load environment variables from configuration
+env['CONFIG'] = CONFIG
+env['executable_names'] = executable_names
+# Only computes hash if time-stamp changed
+env.Decider('MD5-timestamp') 
+# Extensions to be used when scanning for source files in BuildLyx.
+env.EXTENSIONS = ['.eps', '.pdf', '.lyx']
+SourceFileScanner.add_scanner('.lyx', Scanner(gs.misc.lyx_scan, recursive = True))
+# Export environment
+Export('env')
+# Additional mode options
+if mode == 'cache':
+    CacheDir(cache_dir)
+
+# Logging (except on dry run)
+# Log build process
+gs.log.start_log(mode, CONFIG['global']['gslab_version'])
+atexit.register(gs.log.end_log)
+# Log input directories
+gs.log_paths_dict(CONFIG)
+# Log final state of repository 
+debrief_args = CONFIG['global']['scons_debrief_args']
+debrief_args['lfs_required'] = bool('git_lfs' in prereqs)
+atexit.register(gs.scons_debrief, args = debrief_args)
+
+# Run sub-trees
+SConscript('source/prepare_data/SConscript')
+SConscript('source/descriptive/SConscript')
+Default('./build', './release')

analysis/config_global.yaml

@@ -0,0 +1,46 @@
+# GSLab Python Version
+gslab_version: 4.1.0
+
+# Build Directories
+build:
+  prepare_data: build/prepare_data
+  descriptive: build/descriptive
+
+# Source Directories
+source:
+  prepare_data: source/prepare_data
+  descriptive: source/descriptive
+
+# Input Directories (will record contents)
+input: None
+
+# Executable names
+executable_names:
+  python: None
+  r: None
+  stata: None
+  matlab: None
+  lyx: none
+  latex: None
+
+# If True, check that application is installed/up-to-date before SCons run
+prereq_checks:
+  git_lfs: Yes
+  gslab_python: Yes
+  python: Yes
+  r: No
+  stata: No
+  matlab: No
+  lyx: No
+  latex: No
+
+# Repository logging at end of SCons run
+scons_debrief_args:
+  look_in: release;source            # Folders to look in for large versioned files. Assumes ; seperator.
+  file_MB_limit_lfs: 2               # Soft limit on file size (w/ LFS)
+  total_MB_limit_lfs: 500            # Soft limit on total size (w/ LFS)  
+  file_MB_limit: 0.5                 # Soft limit on file size (w/o LFS)
+  total_MB_limit: 125                # Soft limit on total size (w/o LFS)
+  git_attrib_path: ../.gitattributes # Location of .gitattributes
+  MAXIT: 10                          # Max number of files in directory to log
+  log: state_of_repo.log             # Log name

analysis/release/create_data.log

@@ -0,0 +1,52 @@
+
+  ___  ____  ____  ____  ____ (R)
+ /__    /   ____/   /   ____/
+___/   /   /___/   /   /___/   15.1   Copyright 1985-2017 StataCorp LLC
+  Statistics/Data Analysis            StataCorp
+                                      4905 Lakeway Drive
+     MP - Parallel Edition            College Station, Texas 77845 USA
+                                      800-STATA-PC        http://www.stata.com
+                                      979-696-4600        stata@stata.com
+                                      979-696-4601 (fax)
+
+Unlimited-user 4-core Stata network license expires 21 Jul 2018:
+       Serial number:  501509201134
+         Licensed to:  Economics
+                       Stanford University
+
+Notes:
+      1.  Stata is running in batch mode.
+      2.  Unicode is supported; see help unicode_advice.
+      3.  More than 2 billion observations are allowed; see help obs_advice.
+      4.  Maximum number of variables is set to 5000; see help set_maxvar.
+
+
+running /Applications/Stata/profile.do ...
+
+. do "source/prepare_data/create_data.do" 
+
+. version 14
+
+. set more off
+
+. preliminaries
+
+. 
+. program main
+  1.     yaml read YAML using config_global.yaml
+  2.     yaml global build = YAML.build.prepare_data
+  3. 
+.     set obs 300000
+  4.     gen x = _n
+  5.     export delimited "$build/data.txt", delimiter("|") replace
+  6. end
+
+. 
+. * EXECUTE
+. main
+yaml_read error: level = 3 but previous level was 1 (prepare_data: build/prepare_data)
+invalid syntax
+r(100);
+
+end of do-file
+r(100);