Although we spend alot of time working with data interactively, this sort of hands-on babysitting is not always appropriate. We have a philosophy of “source is real” in this class and that philosophy can be implemented on a grander scale. Just as we save R code in a script so we can replay analytical steps, we can also record how a series of scripts and commands work together to produce a set of analytical results. This is what we mean by automating data analysis or building an analytical pipeline.
Why and how we automate data analyses + examples.
2015-11-17 NOTE: since we have already set up a build environment for R packages, it is my hope that everyone has Make. These instructions were from 2014, when we did everything in a different order. Cross your fingers and ignore!
(If you are running Mac OS or Linux,
make should already be installed.)
Walk before you run! Prove that
make is actually installed and that it can be found and executed from the shell and from RStudio. It is also important to tell RStudio to NOT substitute spaces for tabs when editing a
Makefile (applies to any text editor).
This fully developed example shows you
make, i.e. how to
makefrom the shell
There are three more toy pipelines, using the Lord of the Rings data, that reinforce:
Makefile, as the default way of running an R script or an R Markdown document, leading to pretty HTML reports without any mouse clicks
xkcd comic on automation. ‘Automating’ comes from the roots ‘auto-’ meaning ‘self-’, and ‘mating’, meaning ‘screwing’.
Karl Broman covers GNU Make in his course Tools for Reproducible Research (see first week)
Karl Broman also wrote An introduction to
Make, aimed at stats / data science types
Using Make for reproducible scientific analyses, blog post by Ben Morris
Software Carpentry’s Slides on
Zachary M. Jones wrote GNU Make for Reproducible Data Analysis
Keeping tabs on your data analysis workflow, blog post by Adam Laiacano, who works at Tumblr
Mike Bostock, of D3.js and New York Times fame, explains Why Use Make: “it’s about the benefits of capturing workflows via a file-based dependency-tracking build system”
Other, more modern data-oriented alternatives to
maker, “Make-like build management, re-imagined for R”
Managing Projects with GNU Make, Third Edition By Robert Mecklenburg is a fantastic book but, sadly, is very focused on compiling software
RStudio’s website documenting R Markdown is generated from this repo using this 20 line Makefile, which is sort of amazing. This is why we study regular expressions and follow filename conventions, people!
littler is an R package maintained by Dirk Eddelbuettel that “provides the
r program, a simplified command-line interface for GNU R.”