diff --git a/.gitignore b/.gitignore index ecb8f04..d183b03 100644 --- a/.gitignore +++ b/.gitignore @@ -18,3 +18,5 @@ user-config.yaml *.out *.svn .sconsign.dblite +*.lyx.emergency + diff --git a/README.md b/README.md index a4e7b2c..b0f321d 100644 --- a/README.md +++ b/README.md @@ -1,60 +1,110 @@ -Using the repository template -============================= - -#### Pre-requisites: - - - Windows `cmd.exe`, Mac OS X `bash`, or Linux `bash`. - - [Python 2.X](https://www.python.org) (add to [PATH](https://en.wikipedia.org/wiki/PATH_(variable))) - - [Stata MP](http://www.stata.com/statamp/) (add to [PATH](https://en.wikipedia.org/wiki/PATH_(variable))) - - [R](https://www.r-project.org/) (add to [PATH](https://en.wikipedia.org/wiki/PATH_(variable))) - - [Lyx](https://www.lyx.org/) (add to [PATH](https://en.wikipedia.org/wiki/PATH_(variable))) - - [SCons](http://scons.org/) (Note that version 2.4.0 or later is best if using the [cache](http://scons.org/doc/2.0.1/HTML/scons-user/c4213.html)). - - More information about SCons can be found [here](https://github.com/gslab-econ/ra-manual/wiki/SCons). - - [git-lfs](https://git-lfs.github.com/) - - [gslab_tools](https://github.com/gslab-econ/gslab_python) version 3.0.3 or later - - [GSLab-modified Metropolis beamer theme](https://github.com/gslab-econ/mtheme) - - [YAML](http://yaml.org/)-related packages/modules: - - Stata ado file: [yaml](https://github.com/sergiocorreia/stata-misc/tree/75a8b251bec02ba590c862cc395c4b95077d8a95) - - Python module: [PyYAML](http://pyyaml.org/wiki/PyYAML) - - R package: [yaml](https://cran.r-project.org/web/packages/yaml/yaml.pdf) - - -The easiest way to install some of the applications above is to use [Homebrew](http://brew.sh/) on Mac OS and [Linuxbrew](http://linuxbrew.sh/) on Linux, as they will set up the `PATH` variable for you, e.g., `brew install scons`. - -#### To run: - - The entire directory: - - In the root directory, type `scons` in the command line. This should run everything that is flagged as being modified or with dependencies that have been modified. - - A single directory of targets: - - `scons build/data` will re-build the `build/data` folder if it is out of sync, without rebuilding other files. - - A single target file: - - `scons build/paper/paper.pdf` will re-run only the code needed to update `build/paper/paper.pdf` without rebuilding other files. - -See [here](https://github.com/gslab-econ/gslab_python/tree/master/gslab_scons) for directions on making a 'release'. - -#### Copy the template: -In order to create a new repository using this template, either - -- First, either: - - Fork this repository - - Create an empty repository in GitHub and clone it locally. Copy the contents of this template into the empty repository. Make sure to exclude the `.git` folder, but include the `.gitattributes` and `.gitignore` files. Re-run the entire directory using `Scons`. Commit and push to the new repository. -- Setup a `user-config.yaml` in the root of the directory (note that this file should not be versioned): - - MacOS or Linux minimal working example - ``` - stata_flavor: statamp - cache: /Users/leviboxell/Google Drive/cache/template - ``` - - Windows 10 minimal working example (note the quotation marks) +# GSLab Template + +The GSLab Template is a minimal working demonstration of the tools and organization used by projects in the GSLab. We use SCons and a few custom builders to execute scripts and track dependencies in a portable and flexible manner. + +## Prerequisites + +You'll need the following to run the template. [Homebrew](https://brew.sh/) for Mac and [Linuxbrew](http://linuxbrew.sh/) for Linux make this easier. +* Windows `cmd.exe`, Mac OS X `bash`, or Linux `bash`. +* [Python 2.7.X](https://wiki.python.org/moin/BeginnersGuide/Download) for [Windows](https://docs.python.org/2/using/windows.html), [Mac](https://docs.python.org/2/using/mac.html) or [Linux](https://docs.python.org/2/using/unix.html). + * [gslab_python](https://github.com/gslab-econ/gslab_python) version 4.0.0. + * [PyYAML](http://pyyaml.org/wiki/PyYAML) a Python module for parsing YAML files. +* [SCons](http://scons.org/pages/download.html) version 2.4 or later. +* [git](https://git-scm.com/downloads) for version control. + * [git-lfs](https://git-lfs.github.com/) for versioning large files. + * You'll need both git and git-lfs to clone the repository. +* [LyX](https://www.lyx.org/Download) (with instructions for LaTeX) + * Add LyX to your PATH for [Windows](http://www.computerhope.com/issues/ch000549.htm), [Mac](http://hathaway.cc/post/69201163472/how-to-edit-your-path-environment-variables-on-mac), and [Linux](http://stackoverflow.com/questions/14637979/how-to-permanently-set-path-on-linux). + * The beamer theme [`metropolis`](https://github.com/matze/mtheme). This is part of MikTeX since Dec 2014. +* [Stata](http://www.stata.com/) + * Add Stata to your PATH for [Windows](http://www.computerhope.com/issues/ch000549.htm), [Mac](http://hathaway.cc/post/69201163472/how-to-edit-your-path-environment-variables-on-mac), and [Linux](http://stackoverflow.com/questions/14637979/how-to-permanently-set-path-on-linux). + * [yaml](https://github.com/gslab-econ/stata-misc) a Stata ado file for parsing YAML files. + +## Getting started + +1. Open a shell, clone the repository, and navigate to its root. ``` - stata_flavor: "%STATAEXE%" - cache: C:\Users\Levi Boxell\Google Drive\cache\template + git clone https://github.com/gslab-econ/template.git + cd template ``` - - +2. You're ready to go. We'll prompt you to enter any necessary information and store it in `user-config.yaml` as your scripts run. + * To build everything that has been modified or with dependencies in the repository that have been modified. + ``` + scons + ``` + * To build everything in a single directory of targets that has been modified and all of their dependencies that have been modified. + ``` + scons build/path/to/directory + ``` + * To build a single target that has been modified and all of its dependencies that have been modified. + ``` + scons build/path/to/file.ext + ``` + +## Copying the template + +If you want to create a repository with the same structure as this template you can fork it. If you want a repository without any of our git history, follow these instructions. +* Create an empty repository in GitHub and clone it. +* Copy the contents of this template into the empty repository. Make sure to exclude the `.git` folder, but include the [`.gitattributes`](https://git-scm.com/docs/gitattributes) and [`.gitignore`](https://git-scm.com/docs/gitignore) files. +* Create a `user-config.yaml` file and run `scons`. +* Commit the changes and push to the new repository. + +## FAQ + +##### What is `user-config.yaml`? + +Each user is allowed to have different local specifications: We don't put any restrictions on where you keep large files, what you call your executables, or how you manage shared directories. We do need to find these things, and that's what `user-config.yaml` is for. Each user maintains an unversioned [YAML file](http://yaml.org/) with these sorts of specifications. Each script uses its associated yaml-parsing module to read these specifications each time the script is run. + +##### What do I put in `user-config.yaml`? + +There's no "default" for `user-config.yaml` because it depends on system specifications and user preferences. Three things we do recommend keeping in `user-config.yaml` are the name of your Stata executable, the location of a [SCons cache directory](http://scons.org/doc/2.0.1/HTML/scons-user/c4213.html), and the location of a release directory. These fields don't have to be specified if you're not using them, and we'll prompt you for their values at runtime if you've forgotten to specify them and they're necessary. A Mac example where Example_User is running a factory-fresh StataMP and has local access to a directories named cache/template and release on Dropbox would be + +```YAML +stata_executable: statamp +cache_directory: /Users/Example_User/Dropbox/cache/template +release_directory: /Users/Example_User/Dropbox/release/ +``` + +##### How do I handle data external to my repository? + +We are agnostic about how you incorporate external data into the template. There's no custom builder for these assets, by design. Our suggestions: + +* When a large dataset is stored locally, `user-config.yaml` can include an entry specifying the user-specific path to that dataset. The key of the entry should be constant across users and documented in the top-level readme of the repository. + +* When a large dataset is stored externally, there are a few options. + * The top-level readme can specify manual download and storage instructions. This is simple, easy to customize, and unlikely to cause errors during a SCons build. It does, however, require each user to successfully download the same dataset, perhaps in an unstructured manner. + * The download can be incorporated into the SCons build. We either execute a program to transfer data (e.g., `rsync` or `rclone`) directly in a standard SCons command or from within a script executed by one of our custom builders. These methods have the benefits of automation and dependency tracking, but they can introduce idiosyncratic errors if the download steps are prone to failure. + * Regardless of the download method, the path to the dataset should be added to `constant.yaml` and `.gitignore` if it is stored within the repository and to `user-config.yaml` if it is stored elsewhere. + +##### Can I use other software for data analysis? + +Yes. We have custom builders for Python and R. You can also use them with the same syntax as the Stata builder. If you're using R, make sure it's been added to your PATH and that you have a YAML-parsing package, such as [yaml](https://cran.r-project.org/web/packages/yaml/yaml.pdf). + +##### Can I pass "command line style" arguments to a script? + +You bet. All of our custom builders accept "command line style" arguments with the same method. Enumerate the arguments in a list and pass them to the builder through the `CL_ARG` keyword argument, exactly the same way you specify sources and targets. We'll format this list, and `scons` will pass its contents to the script at runtime. You can reference these arguments when writing a script using the standard practice for its language. + +##### How is the build process logged? + +Each of our custom builders produces a log of its process in the same directory as the first of its targets. Each log is named `sconscript.log` by default, and you can insert custom text between `sconscript` and `.log` by passing it as a string through the builder's `log_ext` keyword argument. It's similar to the way that you specify sources and targets, except that the `log_ext` argument must be a string. You should specify the `log_ext` argument for builders that produce logs in the same directory, otherwise the default `sconscript.log` will be overwritten by each builder. + +After all the steps in the build are completed, we'll comb through the directory and look for for any files named `sconscript*.log`. These logs will be concatenated—with the earliest completed ones first and all logs with errors on top. We'll store this concatenated log at the root of the repository in `sconstruct.log`. + +##### Can I write my paper in LaTeX instead of LyX? + +We don't have a custom builder for LaTeX. You can still write in it, but you will have to use [SCons's native builder](http://www.scons.org/doc/0.96.91/HTML/scons-user/a5334.html). You can still use our custom table builder to fill LaTeX tables. + +##### Can I release my repository? + +Yes, our [custom tool](https://github.com/gslab-econ/gslab_python/tree/master/gslab_scons) allows you to release to GitHub and a local destination specified in `user-config.yaml`. A new release can be transfered to a remote manually (e.g., using `rsync` or `rclone`) or automatically by specifying a local destination that's synced to a remote (e.g., a Dropbox directory). + +Every file intended for release should be added to the `release` directory. Files not intended for release to GitHub should be added to `.gitignore`. Our tool will transfer everything in `release` to the local destination and create a [GitHub release](https://help.github.com/articles/creating-releases/) with all the versioned files—those not added to `.gitignore`—in `release`. + #### License The MIT License (MIT) -Copyright (c) 2016 Matthew Gentzkow, Jesse Shapiro +Copyright (c) 2017 Matthew Gentzkow, Jesse Shapiro Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: diff --git a/SConstruct b/SConstruct index b4c8073..9cac031 100644 --- a/SConstruct +++ b/SConstruct @@ -1,29 +1,36 @@ # Preliminaries import os +import sys +sys.dont_write_bytecode = True # Don't write .pyc files # Test for proper prerequisites and setup -from setup import setup_test -[user_configs, mode, sf, cache_dir] = setup_test(ARGUMENTS) -import gslab_scons +from configuration_test import configuration_test +[mode, stata_executable, cache_dir] = configuration_test(ARGUMENTS, + gslab_python_version = '4.0.0') +import gslab_scons as gs +import gslab_scons.log as log import yaml +import atexit -# Start log -gslab_scons.start_log() +# Start log after getting mode and release version +mode = ARGUMENTS.get('mode', 'develop') +vers = ARGUMENTS.get('version', '') +log.start_log(mode, vers) -# Defines environment +# Define the SCons environment env = Environment(ENV = {'PATH' : os.environ['PATH']}, IMPLICIT_COMMAND_DEPENDENCIES = 0, - BUILDERS = {'Tablefill' : Builder(action = gslab_scons.build_tables), - 'BuildLyx' : Builder(action = gslab_scons.build_lyx), - 'BuildR' : Builder(action = gslab_scons.build_r), - 'BuildStata' : Builder(action = gslab_scons.build_stata), - 'BuildPython' : Builder(action = gslab_scons.build_python)}, - user_flavor = sf) - -env.Decider('MD5-timestamp') # Only computes hash if time-stamp changed -env.EXTENSIONS = ['.eps', '.pdf', '.lyx'] # Extensions to be used when scanning for source files in BuildLyx. -SourceFileScanner.add_scanner('.lyx', Scanner(gslab_scons.misc.lyx_scan, recursive = True)) - + BUILDERS = {'Tablefill': Builder(action = gs.build_tables), + 'BuildLyx': Builder(action = gs.build_lyx), + 'BuildStata': Builder(action = gs.build_stata), + 'BuildPython': Builder(action = gs.build_python)}, + stata_executable = stata_executable) + +# Only computes hash if time-stamp changed +env.Decider('MD5-timestamp') +# Extensions to be used when scanning for source files in BuildLyx. +env.EXTENSIONS = ['.eps', '.pdf', '.lyx'] +SourceFileScanner.add_scanner('.lyx', Scanner(gs.misc.lyx_scan, recursive = True)) # Load paths env['PATHS'] = yaml.load(open("constants.yaml", 'rU')) @@ -31,7 +38,6 @@ env['PATHS'] = yaml.load(open("constants.yaml", 'rU')) Export('env') # Run sub-trees -SConscript('source/data/SConscript') SConscript('source/analysis/SConscript') SConscript('source/tables/SConscript') SConscript('source/paper/SConscript') @@ -42,9 +48,11 @@ Default('./build', './release') if mode == 'cache': CacheDir(cache_dir) -# Print the state of the repo at end of SCons run -finish_command = Command( 'state_of_repo.log', [], gslab_scons.misc.state_of_repo, MAXIT=10) # From http://stackoverflow.com/questions/8901296/how-do-i-run-some-code-after-every-build-in-scons -Depends(finish_command, BUILD_TARGETS) -env.AlwaysBuild(finish_command) -if 'state_of_repo.log' not in BUILD_TARGETS: - BUILD_TARGETS.append('state_of_repo.log') +debrief_env = {'MAXIT' : 10, + # Folders to look in for large versioned files + 'look_in' : 'release;source', + # Soft limits on file sizes + 'file_MB_limit' : 2, + 'total_MB_limit' : 500} +atexit.register(log.end_log) +atexit.register(gs.misc.scons_debrief, target = 'state_of_repo.log', env = debrief_env) diff --git a/configuration_test.py b/configuration_test.py new file mode 100644 index 0000000..19d731b --- /dev/null +++ b/configuration_test.py @@ -0,0 +1,53 @@ +#!/usr/bin/python +import sys +import os +import re +import subprocess +import warnings +from gslab_scons import _exception_classes +from gslab_scons import misc + +def configuration_test(ARGUMENTS, gslab_python_version): + # Determines whether to print traceback messages + debug = ARGUMENTS.get('debug', False) + if not debug: + # Hide traceback for configuration test only + # http://stackoverflow.com/questions/27674602/hide-traceback-unless-a-debug-flag-is-set + sys.tracebacklimit = 0 + + # Checks initial prerequisites + try: + from gslab_scons import configuration_tests as config + except ImportError: + message = 'Your gslab_tools Python modules installation is outdated' + raise Exception(message) + + config.check_python(gslab_python_version = gslab_python_version, + packages = ["yaml", "gslab_scons", "gslab_fill"]) + config.check_lyx() + config.check_lfs() + stata_executable = config.check_stata(["yaml"]) + + # Uncomment if using + # config.check_r(packages = ["yaml"]) + + # Loads arguments and configurations + mode = ARGUMENTS.get('mode', 'develop') # Gets mode; defaults to 'develop' + + # Checks mode/version + if not (mode in ['develop', 'cache']): + message = "Error: %s is not a defined mode" % mode + raise _exception_classes.PrerequisiteError(message) + + # Get return list + if mode == 'cache': + cache_dir = misc.load_yaml_value("user-config.yaml", "cache_directory") + cache_dir = misc.check_and_expand_path(cache_dir) + return_list = [mode, stata_executable, cache_dir] + else: + return_list = [mode, stata_executable, None] + + # Restore default tracebacklimit and return values + sys.tracebacklimit = 1000 + + return return_list diff --git a/constants.yaml b/constants.yaml index d235bb9..096a81e 100644 --- a/constants.yaml +++ b/constants.yaml @@ -1,7 +1,6 @@ # Build Directories build: analysis: build/analysis - data: build/data paper: build/paper tables: build/tables talk: build/talk @@ -9,7 +8,6 @@ build: # Source Directories source: analysis: source/analysis - data: source/data figures: source/figures paper: source/paper raw: source/raw diff --git a/release/paper/ondeck.pdf b/release/paper/ondeck.pdf index ceab352..8106912 100644 Binary files a/release/paper/ondeck.pdf and b/release/paper/ondeck.pdf differ diff --git a/release/paper/online_appendix.pdf b/release/paper/online_appendix.pdf index 2061d7c..c63be99 100644 Binary files a/release/paper/online_appendix.pdf and b/release/paper/online_appendix.pdf differ diff --git a/release/paper/paper.pdf b/release/paper/paper.pdf index 6773d7c..7141842 100644 Binary files a/release/paper/paper.pdf and b/release/paper/paper.pdf differ diff --git a/release/paper/sconscript.log b/release/paper/sconscript.log index b2d3aca..2322c5f 100644 --- a/release/paper/sconscript.log +++ b/release/paper/sconscript.log @@ -1,17 +1,9 @@ -Log created: 2017-02-06 17:13:52 -Log completed: 2017-02-06 17:13:54 - -This is pdfTeX, Version 3.14159265-2.6-1.40.17 (TeX Live 2016) (preloaded format=pdflatex) +*** Builder log created: {2017-07-27 13:02:13} +*** Builder log completed: {2017-07-27 13:02:14} + This is pdfTeX, Version 3.14159265-2.6-1.40.17 (TeX Live 2016) (preloaded format=pdflatex) restricted \write18 enabled. entering extended mode -(./paper.tex -LaTeX2e <2016/03/31> -Babel <3.9r> and hyphenation patterns for 83 language(s) loaded. - -This is pdfTeX, Version 3.14159265-2.6-1.40.17 (TeX Live 2016) (preloaded format=pdflatex) - restricted \write18 enabled. -entering extended mode -(./paper.tex +(./text.tex LaTeX2e <2016/03/31> Babel <3.9r> and hyphenation patterns for 83 language(s) loaded. diff --git a/release/paper/text.pdf b/release/paper/text.pdf index 2d79592..bfae272 100644 Binary files a/release/paper/text.pdf and b/release/paper/text.pdf differ diff --git a/release/talk/sconscript.log b/release/talk/sconscript.log index e5773d2..9f4ac73 100644 --- a/release/talk/sconscript.log +++ b/release/talk/sconscript.log @@ -1,7 +1,6 @@ -Log created: 2017-02-06 17:13:54 -Log completed: 2017-02-06 17:13:57 - -This is pdfTeX, Version 3.14159265-2.6-1.40.17 (TeX Live 2016) (preloaded format=pdflatex) +*** Builder log created: {2017-07-27 13:02:14} +*** Builder log completed: {2017-07-27 13:02:18} + This is pdfTeX, Version 3.14159265-2.6-1.40.17 (TeX Live 2016) (preloaded format=pdflatex) restricted \write18 enabled. entering extended mode (./slides.tex diff --git a/release/talk/slides.pdf b/release/talk/slides.pdf index a76d406..7716643 100644 Binary files a/release/talk/slides.pdf and b/release/talk/slides.pdf differ diff --git a/sconstruct.log b/sconstruct.log index 603de79..b85fb3d 100644 --- a/sconstruct.log +++ b/sconstruct.log @@ -1,11 +1,143 @@ +*** New build: {2017-07-27 13:02:04} *** scons: done reading SConscript files. scons: Building targets ... -build_stata(["build/analysis/plot.eps", "build/analysis/table.txt"], ["source/analysis/analysis.do", "build/data/data.txt"]) +build_stata(["build/analysis/plot.pdf", "build/analysis/table.txt"], ["source/analysis/analysis.do"]) +build_lyx(["build/paper/ondeck.pdf"], ["source/paper/ondeck.lyx"]) +build_lyx(["build/paper/online_appendix.pdf"], ["source/paper/online_appendix.lyx"]) build_tables(["build/tables/table.lyx"], ["source/tables/table.lyx", "build/analysis/table.txt"]) source/tables/table.lyx filled successfully by tablefill build_lyx(["build/paper/paper.pdf"], ["source/paper/paper.lyx"]) +build_lyx(["build/paper/text.pdf"], ["source/paper/text.lyx"]) build_lyx(["build/talk/slides.pdf"], ["source/talk/slides.lyx"]) Install directory: "build/paper" as "release/paper" Install directory: "build/talk" as "release/talk" -state_of_repo(["state_of_repo.log"], []) scons: done building targets. +*** Build completed: {2017-07-27 13:02:18} *** + + +./release/talk/sconscript.log +*** Builder log created: {2017-07-27 13:02:14} +*** Builder log completed: {2017-07-27 13:02:18} + This is pdfTeX, Version 3.14159265-2.6-1.40.17 (TeX Live 2016) (preloaded format=pdflatex) + restricted \write18 enabled. +entering extended mode +(./slides.tex +LaTeX2e <2016/03/31> +Babel <3.9r> and hyphenation patterns for 83 language(s) loaded. + +This is pdfTeX, Version 3.14159265-2.6-1.40.17 (TeX Live 2016) (preloaded format=pdflatex) + restricted \write18 enabled. +entering extended mode +(./slides.tex +LaTeX2e <2016/03/31> +Babel <3.9r> and hyphenation patterns for 83 language(s) loaded. + +./build/talk/sconscript.log +*** Builder log created: {2017-07-27 13:02:14} +*** Builder log completed: {2017-07-27 13:02:18} + This is pdfTeX, Version 3.14159265-2.6-1.40.17 (TeX Live 2016) (preloaded format=pdflatex) + restricted \write18 enabled. +entering extended mode +(./slides.tex +LaTeX2e <2016/03/31> +Babel <3.9r> and hyphenation patterns for 83 language(s) loaded. + +This is pdfTeX, Version 3.14159265-2.6-1.40.17 (TeX Live 2016) (preloaded format=pdflatex) + restricted \write18 enabled. +entering extended mode +(./slides.tex +LaTeX2e <2016/03/31> +Babel <3.9r> and hyphenation patterns for 83 language(s) loaded. + +./build/paper/sconscript.log +*** Builder log created: {2017-07-27 13:02:13} +*** Builder log completed: {2017-07-27 13:02:14} + This is pdfTeX, Version 3.14159265-2.6-1.40.17 (TeX Live 2016) (preloaded format=pdflatex) + restricted \write18 enabled. +entering extended mode +(./text.tex +LaTeX2e <2016/03/31> +Babel <3.9r> and hyphenation patterns for 83 language(s) loaded. + +./release/paper/sconscript.log +*** Builder log created: {2017-07-27 13:02:13} +*** Builder log completed: {2017-07-27 13:02:14} + This is pdfTeX, Version 3.14159265-2.6-1.40.17 (TeX Live 2016) (preloaded format=pdflatex) + restricted \write18 enabled. +entering extended mode +(./text.tex +LaTeX2e <2016/03/31> +Babel <3.9r> and hyphenation patterns for 83 language(s) loaded. + +./build/tables/sconscript.log +*** Builder log created: {2017-07-27 13:02:12} +*** Builder log completed: {2017-07-27 13:02:12} + source/tables/table.lyx filled successfully by tablefill + +./build/analysis/sconscript_analysis_do.log +*** Builder log created: {2017-07-27 13:02:04} +*** Builder log completed: {2017-07-27 13:02:06} + + ___ ____ ____ ____ ____ (R) + /__ / ____/ / ____/ +___/ / /___/ / /___/ 14.2 Copyright 1985-2015 StataCorp LLC + Statistics/Data Analysis StataCorp + 4905 Lakeway Drive + MP - Parallel Edition College Station, Texas 77845 USA + 800-STATA-PC http://www.stata.com + 979-696-4600 stata@stata.com + 979-696-4601 (fax) + +Unlimited-user 4-core Stata network license expires 21 Jul 2018: + Serial number: 501509201134 + Licensed to: Quan Le + Stanford University + +Notes: + 1. Stata is running in batch mode. + 2. Unicode is supported; see help unicode_advice. + 3. More than 2 billion observations are allowed; see help obs_advice. + 4. Maximum number of variables is set to 5000; see help set_maxvar. + + +running /Applications/Stata/profile.do ... + +. do "source/analysis/analysis.do" + +. version 14 + +. clear all + +. set more off + +. +. program main + 1. yaml read YAML using constants.yaml + 2. yaml local build_analysis=YAML.build.analysis + 3. +. set obs 20 + 4. egen x = fill(1 2 3 4 5 6 7 8) + 5. hist x, discrete width(0.5) + 6. graph export "`build_analysis'/plot.pdf", replace + 7. +. sum x + 8. file open outfile using "`build_analysis'/table.txt", write replace + 9. file write outfile "" _n + 10. file write outfile (r(mean)) _n (r(sd)) _n (r(max)) _n (r(min)) + 11. file close outfile + 12. end + +. +. * EXECUTE +. main +number of observations (_N) was 0, now 20 +(start=1, width=.5) +(file build/analysis/plot.pdf written in PDF format) + + Variable | Obs Mean Std. Dev. Min Max +-------------+--------------------------------------------------------- + x | 20 10.5 5.91608 1 20 +(note: file build/analysis/table.txt not found) + +. +end of do-file diff --git a/setup.py b/setup.py deleted file mode 100644 index 5426930..0000000 --- a/setup.py +++ /dev/null @@ -1,138 +0,0 @@ -import sys -import os -import re -import subprocess -import pkg_resources -import warnings - -def setup_test(ARGUMENTS): - check_python() - check_r() - check_lyx() - check_metropolis() - check_gitlfs() - check_yamls() - - # Loads arguments and configurations - import yaml - user_configs = yaml.load(open("user-config.yaml", 'rU')) - mode = ARGUMENTS.get('mode', 'develop') # Gets mode; defaults to 'develop' - sf = ARGUMENTS.get('sf', user_configs['stata_flavor']) # Gets user supplied stata from command line or yaml - cache_dir = user_configs['cache'] - if mode in ['cache', 'release']: - check_cache(cache) - - # Check stata after loading sf - check_stata(sf) - - # Checks mode/version - if not (mode in ['develop', 'cache']): - raise PrerequisiteError("Error: %s is not a defined mode" % mode) - - # Returns arguments and configurations - return [user_configs, mode, sf, cache_dir] - -def check_python(): - if sys.version_info[0] != 2: - raise PrerequisiteError('Not the right version of python') - check_python_packages() - -def check_python_packages(): - try: - import gslab_scons - import gslab_make - import gslab_fill - except: - raise PrerequisiteError('Missing gslab_tools python module') - - try: - import yaml - except: - raise PrerequisiteError('Missing PyYAML python module') - - if pkg_resources.get_distribution('gslab_tools').version < '3.0.3': - raise PrerequisiteError('Wrong version of gslab_tools') - -def check_r(): - from gslab_scons.misc import is_in_path - if is_in_path('R.exe') is None and is_in_path('R') is None: - raise PrerequisiteError('R is not installed or excecutable is not added to path') - check_r_packages() - -def check_r_packages(): - for pkg in ["yaml"]: - subprocess.check_output('R -q -e "library(%s)"' % pkg, shell = True) # http://stackoverflow.com/questions/6701230/call-r-function-in-linux-command-line and http://stackoverflow.com/questions/18962785/oserror-errno-2-no-such-file-or-directory-while-using-python-subprocess-in-dj - -def check_lyx(): - from gslab_scons.misc import is_in_path - if is_in_path('lyx.exe') is None and is_in_path('lyx') is None: - raise PrerequisiteError('Lyx is not installed or excecutable is not added to path') - -def check_metropolis(): - if sys.platform == 'win32': - warnings.warn('It has not been tested whether Metropolis beamer is installed or not.' + \ - 'Please make sure it has been installed before running the repo.') - else: - metropolis_path = '/usr/local/texlive/texmf-local/tex/latex/' + \ - 'beamer/themes/gslab/beamerthememetropolis_gslab.sty' - if os.path.isfile(metropolis_path): - pass - else: - raise PrerequisiteError('Metropolis beamer not found at %s' % metropolis_path) - -def check_gitlfs(): - from gslab_scons.misc import check_lfs - check_lfs() - -def check_yamls(): - for f in ["constants.yaml", "user-config.yaml"]: - if not os.path.isfile(f): - raise PrerequisiteError("%s file does not exist. Please create it." % f) - -def check_cache(cache): - if not os.path.isdir(cache): - raise PrerequisiteError("Cache directory (%s) is not created." + \ - "Please manually create before running." % cache) - -def check_stata(sf): - import gslab_scons.misc as misc - command = '' - flavors = ['stata-mp', 'stata-se', 'stata'] - if sf is not None: - flavors = [sf] + flavors - if misc.is_unix(): - for flavor in flavors: - if misc.is_in_path(flavor): - command = misc.stata_command_unix(flavor) - break - elif sys.platform == 'win32': - try: - key_exist = os.environ['STATAEXE'] is not None - command = misc.stata_command_win("%%STATAEXE%%") - except KeyError: - flavors = [(f.replace('-', '') + '.exe') for f in flavors] - if misc.is_64_windows(): - flavors = [f.replace('.exe', '-64.exe') for f in flavors] - for flavor in flavors: - if misc.is_in_path(flavor): - command = misc.stata_command_win(flavor) - break - if command == '': - raise PrerequisiteError('Stata is not installed or excecutable is not added to path') - - check_stata_packages(command) - -def check_stata_packages(command): - command = command.split("%s")[0] - for pkg in ['yaml']: - subprocess.check_output(command + '"which %s"' % pkg , shell = True) # http://www.stata.com/statalist/archive/2009-12/msg00493.html and http://stackoverflow.com/questions/18962785/oserror-errno-2-no-such-file-or-directory-while-using-python-subprocess-in-dj - with open('stata.log', 'rU') as f: - log = f.read() - if re.search('command %s not found' % pkg, log): - raise PrerequisiteError('Stata package %s is not installed' % pkg) - os.remove('stata.log') - -class PrerequisiteError(Exception): - pass - - diff --git a/source/analysis/SConscript b/source/analysis/SConscript index 9343331..5e8213e 100644 --- a/source/analysis/SConscript +++ b/source/analysis/SConscript @@ -3,7 +3,9 @@ Import('env') build = env['PATHS']['build']['analysis'] source = env['PATHS']['source']['analysis'] -env.BuildStata(target = ['#%s/plot.eps' % build, - '#%s/table.txt' % build], - source = ['#%s/analysis.do' % source, - '#%s/data.txt' % env['PATHS']['build']['data']]) +env.BuildStata(target = ['#%s/plot.pdf' % build, + '#%s/table.txt' % build], + source = ['#%s/analysis.do' % source], + log_ext = "analysis_do") + + diff --git a/source/analysis/analysis.do b/source/analysis/analysis.do index 8ba1cc2..35c9af5 100644 --- a/source/analysis/analysis.do +++ b/source/analysis/analysis.do @@ -2,15 +2,14 @@ version 14 clear all set more off - program main yaml read YAML using constants.yaml - yaml local build_data=YAML.build.data yaml local build_analysis=YAML.build.analysis - import delimited "`build_data'/data.txt", delimiters("|") varnames(1) clear + set obs 20 + egen x = fill(1 2 3 4 5 6 7 8) hist x, discrete width(0.5) - graph export "`build_analysis'/plot.eps", replace + graph export "`build_analysis'/plot.pdf", replace sum x file open outfile using "`build_analysis'/table.txt", write replace @@ -19,6 +18,5 @@ program main file close outfile end - * EXECUTE main diff --git a/source/data/SConscript b/source/data/SConscript deleted file mode 100644 index c4ed754..0000000 --- a/source/data/SConscript +++ /dev/null @@ -1,7 +0,0 @@ -Import('env') - -source = '#%s/build.R' % env['PATHS']['source']['data'] -target_file_name = 'data' -target = ['#%s/%s.txt' % (env['PATHS']['build']['data'], target_file_name)] - -env.BuildR(target, source, CL_ARG = target_file_name) # Demonstrates using command line arguments diff --git a/source/data/build.R b/source/data/build.R deleted file mode 100644 index 3178959..0000000 --- a/source/data/build.R +++ /dev/null @@ -1,11 +0,0 @@ -library(yaml) -PATHS <- yaml.load_file('constants.yaml') -args <- commandArgs(TRUE) - -main <- function(i){ - x <- seq(-10, 10, 0.1) - write.table(x, sprintf("%s/%s.txt", PATHS$build$data, i), sep = "|", - row.names = FALSE, col.names = TRUE, quote = FALSE) -} - -main(args[1]) diff --git a/source/figures/figure.lyx b/source/figures/figure.lyx index 4bdaedf..e17da1f 100644 --- a/source/figures/figure.lyx +++ b/source/figures/figure.lyx @@ -102,7 +102,7 @@ Figure Title \begin_layout Plain Layout \align center \begin_inset Graphics - filename ../../build/analysis/plot.eps + filename ../../build/analysis/plot.pdf scale 70 \end_inset diff --git a/source/paper/SConscript b/source/paper/SConscript index a64383f..960a47c 100644 --- a/source/paper/SConscript +++ b/source/paper/SConscript @@ -15,4 +15,4 @@ env.BuildLyx(target = ['#%s/text.pdf' % build], env.BuildLyx(target = ['#%s/ondeck.pdf' % build], source = ['#%s/ondeck.lyx' % source]) -env.Install('#release', '#%s/' % build) +env.Install('#release', '#%s/' % build) \ No newline at end of file diff --git a/source/talk/slides.lyx b/source/talk/slides.lyx index d822a87..9635f34 100644 --- a/source/talk/slides.lyx +++ b/source/talk/slides.lyx @@ -7,7 +7,7 @@ \textclass beamer \begin_preamble \PassOptionsToPackage{no-math}{fontspec} -\usetheme{metropolis_gslab} +\usetheme{metropolis} \usepackage{graphicx} \usepackage{hyperref} \usepackage{multirow} @@ -243,7 +243,7 @@ Figure \begin_layout Standard \align center \begin_inset Graphics - filename ../../build/analysis/plot.eps + filename ../../build/analysis/plot.pdf scale 50 \end_inset diff --git a/state_of_repo.log b/state_of_repo.log index aa8e44e..b874a8f 100644 --- a/state_of_repo.log +++ b/state_of_repo.log @@ -8,16 +8,17 @@ as it can be edited after state_of_repo.log finishes running =================================== Last commit: -commit b6382091044b5ac8b72ef58d7a22611066a0753c -Author: Levi Boxell -Date: Mon Feb 6 17:00:14 2017 -0800 +commit 0c8dff7f298a771f359a160188f4691bc00ccf59 +Merge: 6f1e5b0 27673f2 +Author: Quan +Date: Tue Jul 25 14:38:00 2017 -0700 - #29 More PR comments + Merge branch 'master' into dev_four Files changed since last commit: -.sconsign.dblite +SConstruct release/paper/ondeck.pdf release/paper/online_appendix.pdf release/paper/paper.pdf @@ -26,7 +27,6 @@ release/paper/text.pdf release/talk/sconscript.log release/talk/slides.pdf sconstruct.log -source/tables/SConscript state_of_repo.log =================================== @@ -34,138 +34,132 @@ state_of_repo.log FILE STATUS =================================== +./change_log.md: + modified on: 05 Jul 2017 11:53:32 + size of file: 494 +./configuration_test.py: + modified on: 20 Jul 2017 19:15:31 + size of file: 1795 ./constants.yaml: - modified on: 28 Jan 2017 11:33:14 - size of file: 324 + modified on: 12 Jun 2017 09:21:15 + size of file: 285 ./README.md: - modified on: 28 Jan 2017 12:11:50 - size of file: 3902 + modified on: 27 Jul 2017 13:00:44 + size of file: 7938 ./SConstruct: - modified on: 06 Feb 2017 16:52:43 - size of file: 1941 + modified on: 27 Jul 2017 13:01:11 + size of file: 2111 ./sconstruct.log: - modified on: 06 Feb 2017 17:13:50 - size of file: 0 -./setup.py: - modified on: 06 Feb 2017 16:59:37 - size of file: 5087 + modified on: 27 Jul 2017 13:02:18 + size of file: 841 ./setup.pyc: - modified on: 06 Feb 2017 16:59:40 - size of file: 5954 + modified on: 21 Apr 2017 19:58:07 + size of file: 5497 ./state_of_repo.log: - modified on: 06 Feb 2017 17:13:57 + modified on: 27 Jul 2017 13:02:18 size of file: 678 +./Untitled.ipynb: + modified on: 24 Jul 2017 14:13:17 + size of file: 1206 ./user-config.yaml: - modified on: 28 Jan 2017 12:11:47 - size of file: 75 -./build/analysis/plot.eps: - modified on: 06 Feb 2017 17:13:52 - size of file: 12829 -./build/analysis/sconscript.log: - modified on: 06 Feb 2017 17:13:52 - size of file: 2211 + modified on: 27 Jul 2017 13:01:37 + size of file: 168 +./build/analysis/plot.pdf: + modified on: 27 Jul 2017 13:02:06 + size of file: 1290 +./build/analysis/sconscript_analysis_do.log: + modified on: 27 Jul 2017 13:02:06 + size of file: 2133 ./build/analysis/table.txt: - modified on: 06 Feb 2017 17:13:52 - size of file: 30 -./build/data/data.txt: - modified on: 06 Feb 2017 16:59:44 - size of file: 1035 -./build/data/sconscript.log: - modified on: 06 Feb 2017 16:59:44 - size of file: 1164 -./build/data/testing/data.txt: - modified on: 04 Feb 2017 15:50:13 - size of file: 1035 -./build/data/testing/data_2.txt: - modified on: 04 Feb 2017 16:41:32 - size of file: 1038 + modified on: 27 Jul 2017 13:02:06 + size of file: 31 ./build/paper/ondeck.pdf: - modified on: 04 Feb 2017 16:27:59 - size of file: 18640 + modified on: 27 Jul 2017 13:02:11 + size of file: 18038 ./build/paper/online_appendix.pdf: - modified on: 04 Feb 2017 16:28:00 - size of file: 45828 + modified on: 27 Jul 2017 13:02:12 + size of file: 45649 ./build/paper/paper.pdf: - modified on: 06 Feb 2017 17:13:54 - size of file: 58176 + modified on: 27 Jul 2017 13:02:13 + size of file: 55661 ./build/paper/sconscript.log: - modified on: 06 Feb 2017 17:13:54 - size of file: 562 + modified on: 27 Jul 2017 13:02:14 + size of file: 342 ./build/paper/text.pdf: - modified on: 04 Feb 2017 16:28:04 - size of file: 18640 + modified on: 27 Jul 2017 13:02:14 + size of file: 18038 +./build/tables/sconscript.log: + modified on: 27 Jul 2017 13:02:12 + size of file: 157 ./build/tables/table.lyx: - modified on: 06 Feb 2017 17:13:52 - size of file: 4432 + modified on: 27 Jul 2017 13:02:12 + size of file: 4082 ./build/talk/sconscript.log: - modified on: 06 Feb 2017 17:13:57 - size of file: 564 + modified on: 27 Jul 2017 13:02:18 + size of file: 589 ./build/talk/slides.pdf: - modified on: 06 Feb 2017 17:13:57 - size of file: 28971 + modified on: 27 Jul 2017 13:02:18 + size of file: 26982 ./release/paper/ondeck.pdf: - modified on: 04 Feb 2017 16:27:59 - size of file: 18640 + modified on: 27 Jul 2017 13:02:11 + size of file: 18038 ./release/paper/online_appendix.pdf: - modified on: 04 Feb 2017 16:28:00 - size of file: 45828 + modified on: 27 Jul 2017 13:02:12 + size of file: 45649 ./release/paper/paper.pdf: - modified on: 06 Feb 2017 17:13:54 - size of file: 58176 + modified on: 27 Jul 2017 13:02:13 + size of file: 55661 ./release/paper/sconscript.log: - modified on: 06 Feb 2017 17:13:54 - size of file: 562 + modified on: 27 Jul 2017 13:02:14 + size of file: 342 ./release/paper/text.pdf: - modified on: 04 Feb 2017 16:28:04 - size of file: 18640 + modified on: 27 Jul 2017 13:02:14 + size of file: 18038 ./release/talk/sconscript.log: - modified on: 06 Feb 2017 17:13:57 - size of file: 564 + modified on: 27 Jul 2017 13:02:18 + size of file: 589 ./release/talk/slides.pdf: - modified on: 06 Feb 2017 17:13:57 - size of file: 28971 + modified on: 27 Jul 2017 13:02:18 + size of file: 26982 ./source/analysis/analysis.do: - modified on: 06 Feb 2017 17:13:35 - size of file: 584 + modified on: 20 Jul 2017 15:28:13 + size of file: 510 ./source/analysis/SConscript: - modified on: 06 Feb 2017 16:51:50 - size of file: 338 -./source/data/build.R: - modified on: 28 Jan 2017 11:08:43 - size of file: 298 -./source/data/SConscript: - modified on: 06 Feb 2017 16:51:50 - size of file: 292 + modified on: 27 Jul 2017 13:00:44 + size of file: 307 ./source/figures/figure.lyx: - modified on: 26 Jan 2017 11:25:16 + modified on: 12 Jun 2017 09:21:21 size of file: 2223 ./source/paper/ondeck.lyx: - modified on: 26 Jan 2017 11:23:54 + modified on: 17 Feb 2017 17:01:41 size of file: 2522 ./source/paper/online_appendix.lyx: - modified on: 26 Jan 2017 11:23:54 + modified on: 17 Feb 2017 17:01:41 size of file: 3610 ./source/paper/paper.lyx: - modified on: 26 Jan 2017 11:23:54 + modified on: 17 Feb 2017 17:01:41 size of file: 5236 ./source/paper/SConscript: - modified on: 28 Jan 2017 11:56:48 - size of file: 559 + modified on: 25 Feb 2017 13:04:15 + size of file: 558 ./source/paper/text.lyx: - modified on: 26 Jan 2017 11:23:54 + modified on: 17 Feb 2017 17:01:41 size of file: 2522 ./source/raw/README.md: - modified on: 28 Jan 2017 11:20:49 + modified on: 17 Feb 2017 17:01:41 size of file: 102 ./source/tables/SConscript: - modified on: 06 Feb 2017 17:13:47 + modified on: 24 Jul 2017 14:05:51 size of file: 245 ./source/tables/table.lyx: - modified on: 26 Jan 2017 11:23:54 + modified on: 10 Jul 2017 15:56:14 size of file: 4082 +./source/tables/table.tex: + modified on: 24 Jul 2017 13:02:00 + size of file: 997 ./source/talk/SConscript: - modified on: 28 Jan 2017 11:58:21 + modified on: 17 Feb 2017 17:02:23 size of file: 229 ./source/talk/slides.lyx: - modified on: 26 Jan 2017 11:23:54 - size of file: 4712 + modified on: 12 Jun 2017 09:21:21 + size of file: 4706