Data Wrangling With R

UseR !
BradleyC. Boehmke
Data
Wrangling
with R
Use R!
Series Editors:
Robert Gentleman Kurt Hornik Giovanni Parmigiani
More information about this series at http://www.springer.com/series/6991

Use R!
Wickham: ggplot2
Moore: Applied Survival Analysis Using R
Luke: A Users Guide to Network Analysis in R
Monogan: Political Analysis Using R
Cano/M. Moguerza/Prieto Corcoba: Quality Control with R
Schwarzer/Carpenter/Rcker: Meta-Analysis with R
Gondro: Primer to Analysis of Genomic Data Using R
Chapman/Feit: R for Marketing Research and Analytics
Willekens: Multistate Analysis of Life Histories with R
Cortez: Modern Optimization with R
Kolaczyk/Csrdi: Statistical Analysis of Network Data with R
Swenson/Nathan: Functional and Phylogenetic Ecology in R
Nolan/Temple Lang: XML and Web Technologies for Data Sciences with R
Nagarajan/Scutari/Lbre: Bayesian Networks in R
van den Boogaart/Tolosana-Delgado: Analyzing Compositional Data with R
Bivand/Pebesma/Gmez-Rubio: Applied Spatial Data Analysis with R
(2nd ed. 2013)
Eddelbuettel: Seamless R and C++ Integration with Rcpp
Knoblauch/Maloney: Modeling Psychophysical Data in R
Lin/Shkedy/Yekutieli/Amaratunga/Bijnens: Modeling Dose-Response Microarray
Data in Early Drug Development
Experiments Using R
Cano/M. Moguerza/Redchuk: Six Sigma with R
Soetaert/Cash/Mazzia: Solving Differential Equations in R
Bradley C. Boehmke
Data Wrangling with R

Bradley C. Boehmke, Ph.D.
Air Force Institute of Technology
Dayton, OH, USA
ISSN 2197-5736 ISSN 2197-5744 (electronic)

Use R!
ISBN 978-3-319-45598-3 ISBN 978-3-319-45599-0 (eBook)
DOI 10.1007/978-3-319-45599-0
Library of Congress Control Number: 2016953509
Springer International Publishing Switzerland 2016

This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of
the material is concerned, specically the rights of translation, reprinting, reuse of illustrations, recitation,
broadcasting, reproduction on microlms or in any other physical way, and transmission or information
storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology
now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication
does not imply, even in the absence of a specic statement, that such names are exempt from the relevant
protective laws and regulations and therefore free for general use.
The publisher, the authors and the editors are safe to assume that the advice and information in this book
are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the
editors give a warranty, express or implied, with respect to the material contained herein or for any errors
or omissions that may have been made.
Printed on acid-free paper
This Springer imprint is published by Springer Nature

The registered company is Springer International Publishing AG
The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland
Preface
Welcome to Data Wrangling with R! In this book, I will help you learn the essentials
of preprocessing data leveraging the R programming language to easily and quickly
turn noisy data into usable pieces of information. Data wrangling, which is also
commonly referred to as data munging, transformation, manipulation, janitor work,
etc., can be a painstakingly laborious process. In fact, it has been stated that up to
80 % of data analysis is spent on the process of cleaning and preparing data (cf.
Wickham 2014; Dasu and Johnson 2003). However, being a prerequisite to the rest
of the data analysis workow (visualization, modeling, reporting), its essential that
you become uent and efcient in data wrangling techniques.
This book will guide you through the data wrangling process along with giving
you a solid foundation of the basics of working with data in R. My goal is to teach
you how to easily wrangle your data, so you can spend more time focused on under-
standing the content of your data via visualization, modeling, and reporting your
results. By the time you nish reading this book, you will have learned:
How to work with the different types of data such as numerics, characters, regu-
lar expressions, factors, and dates.
The difference between the various data structures and how to create, add addi-
tional components to, and how to subset each data structure.
How to acquire and parse data from locations you may not have been able to
access before such as web scraping or leveraging APIs.
How to develop your own functions and use loop control structures to reduce
code redundancy.
How to use pipe operators to simplify your code and make it more readable.
How to reshape the layout of your data, and manipulate, summarize, and join
data sets.
Not only will you learn many base R functions, youll also learn how to use some
of the latest data wrangling packages such as tidyr, dplyr, httr, stringr,
lubridate, readr, rvest, magrittr, xlsx, readxl and others. In
essence, you will have the data wrangling toolbox required for modern day data
analysis.
v
vi Preface
Who This Book Is for
This book is meant to establish the baseline R vocabulary and knowledge for the
primary data wrangling processes. This captures a wide range of programming
activities which covers the full spectrum from understanding basic data objects in R
to writing your own functions, applying loops, and web scraping. As a result, this
book can be benecial to all levels of R programmers. Beginner R programmers
will gain a basic understanding of the functionality of R along with learning how to
work with data using R. Intermediate and advanced R programmers will likely nd
the early chapters reiterating established knowledge; however, these programmers
will benet from the mid and latter chapters by learning newer and more efcient
data wrangling techniques.
What You Need for This Book
Obviously to gain and retain knowledge from this book, it is highly recommended
that you follow along and practice the code examples yourself. Furthermore, this
book assumes that you will actually be performing data wrangling in R; therefore,
it is assumed that you have or plan to have R installed on your computer. You will nd
the latest version of R for Linux, Mac OS, and Windows at https://cran.r-project.org.
It is also recommended that you use an integrated development environment (IDE)
as it will simplify and organize your coding environment greatly. There are several
to choose from; however, I highly recommend the RStudio IDE which you can
download at https://www.rstudio.com.
Reader Feedback
Reader comments are greatly appreciated. Please send any feedback regarding
typos, mistakes, confusing statements, or opportunities for improvement to wran-
glingdata@gmail.com.
Bibliography
Dasu, T., & Johnson, T. (2003). Exploratory Data Mining and Data Cleaning (Vol. 479). John
Wiley & Sons.
Wickham, H. (2014). Tidy data. Journal of Statistical Software, 59 (i10).
Contents
Part I Introduction
1 The Role of Data Wrangling .................................................................. 3
2 Introduction to R..................................................................................... 7
2.1 Open Source ..................................................................................... 7
2.2 Flexibility ......................................................................................... 8
2.3 Community ...................................................................................... 9
3 The Basics ................................................................................................ 11
3.1 Installing R and RStudio .................................................................. 11
3.2 Understanding the Console .............................................................. 13
3.2.1 Script Editor ......................................................................... 13
3.2.2 Workspace Environment ...................................................... 13
3.2.3 Console ................................................................................ 15
3.2.4 Misc. Displays...................................................................... 15
3.2.5 Workspace Options and Shortcuts ....................................... 15
3.3 Getting Help ..................................................................................... 16
3.3.1 General Help ........................................................................ 16
3.3.2 Getting Help on Functions ................................................... 16
3.3.3 Getting Help from the Web .................................................. 17
3.4 Working with Packages .................................................................... 17
3.4.1 Installing Packages............................................................... 18
3.4.2 Loading Packages ................................................................ 18
3.4.3 Getting Help on Packages .................................................... 19
3.4.4 Useful Packages ................................................................... 19
3.5 Assignment and Evaluation ............................................................. 19
3.6 R as a Calculator .............................................................................. 21
3.6.1 Vectorization ........................................................................ 22
vii
viii Contents
3.7 Styling Guide ................................................................................... 24

3.7.1 Notation and Naming ........................................................... 24
3.7.2 Organization ......................................................................... 25
3.7.3 Syntax .................................................................................. 26
Part II Working with Different Types of Data in R

4 Dealing with Numbers ............................................................................ 31
4.1 Integer vs. Double ............................................................................ 31
4.1.1 Creating Integer and Double Vectors ................................... 31
4.1.2 Converting Between Integer and Double Values ................. 32
4.2 Generating Sequence of Non-random Numbers .............................. 32
4.2.1 Specifying Numbers Within a Sequence ............................. 32
4.2.2 Generating Regular Sequences ............................................ 33
4.3 Generating Sequence of Random Numbers ..................................... 33
4.3.1 Uniform Numbers ................................................................ 34
4.3.2 Normal Distribution Numbers ............................................. 34
4.3.3 Binomial Distribution Numbers ........................................... 35
4.3.4 Poisson Distribution Numbers ............................................. 36
4.3.5 Exponential Distribution Numbers ...................................... 36
4.3.6 Gamma Distribution Numbers ............................................. 37
4.4 Setting the Seed for Reproducible Random Numbers ..................... 37
4.5 Comparing Numeric Values ............................................................. 37
4.5.1 Comparison Operators ......................................................... 38
4.5.2 Exact Equality ...................................................................... 39
4.5.3 Floating Point Comparison .................................................. 39
4.6 Rounding Numbers .......................................................................... 39
5 Dealing with Character Strings ............................................................. 41
5.1 Character String Basics .................................................................... 41
5.1.1 Creating Strings ................................................................... 41
5.1.2 Converting to Strings ........................................................... 42
5.1.3 Printing Strings .................................................................... 43
5.1.4 Counting String Elements and Characters ........................... 45
5.2 String Manipulation with Base R..................................................... 46
5.2.1 Case Conversion................................................................... 46
5.2.2 Simple Character Replacement ............................................ 46
5.2.3 String Abbreviations ............................................................ 47
5.2.4 Extract/Replace Substrings .................................................. 47
5.3 String Manipulation with stringr ............................................... 49
5.3.1 Basic Operations .................................................................. 49
5.3.2 Duplicate Characters Within a String .................................. 51
5.3.3 Remove Leading and Trailing Whitespace .......................... 51
5.3.4 Pad a String with Whitespace .............................................. 52
Contents ix
5.4 Set Operatons for Character Strings ................................................ 52

5.4.1 Set Union ............................................................................. 52
5.4.2 Set Intersection..................................................................... 52
5.4.3 Identifying Different Elements ............................................ 53
5.4.4 Testing for Element Equality ............................................... 53
5.4.5 Testing for Exact Equality ................................................... 53
5.4.6 Identifying If Elements Are Contained in a String .............. 54
5.4.7 Sorting a String .................................................................... 54
6 Dealing with Regular Expressions ......................................................... 55
6.1 Regex Syntax ................................................................................... 55
6.1.1 Metacharacters ..................................................................... 56
6.1.2 Sequences............................................................................. 56
6.1.3 Character Classes ................................................................. 57
6.1.4 POSIX Character Classes .................................................... 58
6.1.5 Quantiers ............................................................................ 59
6.2 Regex Functions ............................................................................... 60
6.2.1 Main Regex Functions in R ................................................. 60
6.2.2 Regex Functions in stringr ............................................. 63
6.3 Additional Resources ....................................................................... 66
7 Dealing with Factors ............................................................................... 67
7.1 Creating, Converting and Inspecting Factors ................................... 67
7.2 Ordering Levels................................................................................ 68
7.3 Revalue Levels ................................................................................. 69
7.4 Dropping Levels ............................................................................... 69
8 Dealing with Dates .................................................................................. 71
8.1 Getting Current Date and Time ........................................................ 71
8.2 Converting Strings to Dates ............................................................. 72
8.2.1 Convert Strings to Dates ...................................................... 72
8.2.2 Create Dates by Merging Data ............................................. 73
8.3 Extract and Manipulate Parts of Dates............................................. 73
8.4 Creating Date Sequences ................................................................. 75
8.5 Calculations with Dates ................................................................... 76
8.6 Dealing with Time Zones and Daylight Savings ............................. 77
8.7 Additional Resources ....................................................................... 78
Part III Managing Data Structures in R

9 Data Structure Basics ............................................................................. 81
9.1 Identifying the Structure .................................................................. 81
9.2 Attributes.......................................................................................... 82
x Contents
10 Managing Vectors.................................................................................... 85
10.1 Creating Vectors ............................................................................. 85
10.2 Adding On To Vectors.................................................................... 86
10.3 Adding Attributes to Vectors.......................................................... 87
10.4 Subsetting Vectors.......................................................................... 88
10.4.1 Subsetting with Positive Integers ..................................... 88
10.4.2 Subsetting with Negative Integers .................................... 88
10.4.3 Subsetting with Logical Values ........................................ 89
10.4.4 Subsetting with Names..................................................... 89
10.4.5 Simplifying vs. Preserving ............................................... 89
11 Managing Lists ........................................................................................ 91
11.1 Creating Lists ................................................................................. 91
11.2 Adding On To Lists ........................................................................ 92
11.3 Adding Attributes to Lists .............................................................. 93
11.4 Subsetting Lists .............................................................................. 95
11.4.1 Subset List and Preserve Output as a List ........................ 95
11.4.2 Subset List and Simplify Output ...................................... 96
11.4.3 Subset List to Get Elements Out of a List ........................ 96
11.4.4 Subset List with a Nested List.......................................... 96
12 Managing Matrices ................................................................................. 99
12.1 Creating Matrices ........................................................................... 99
12.2 Adding On To Matrices.................................................................. 100
12.3 Adding Attributes to Matrices........................................................ 101
12.4 Subsetting Matrices........................................................................ 103
13 Managing Data Frames .......................................................................... 105
13.1 Creating Data Frames .................................................................... 105
13.2 Adding On To Data Frames ........................................................... 107
13.3 Adding Attributes to Data Frames ................................................. 109
13.4 Subsetting Data Frames ................................................................. 111
14 Dealing with Missing Values .................................................................. 113
14.1 Testing for Missing Values............................................................. 113
14.2 Recoding Missing Values............................................................... 114
14.3 Excluding Missing Values.............................................................. 114
Part IV Importing, Scraping, and Exporting Data with R

15 Importing Data ........................................................................................ 119
15.1 Reading Data from Text Files ........................................................ 119
15.1.1 Base R Functions.............................................................. 119
15.1.2 readr Package ................................................................... 122
15.2 Reading Data from Excel Files ...................................................... 123
15.2.1 xlsx Package ..................................................................... 123
15.2.2 readxl Package ................................................................. 125
Contents xi
15.3 Load Data from Saved R Object File ............................................. 127

15.4 Additional Resources ..................................................................... 127
16 Scraping Data .......................................................................................... 129
16.1 Importing Tabular and Excel Files Stored Online.......................... 129
16.2 Scraping HTML Text ..................................................................... 134
16.2.1 Scraping HTML Nodes .................................................... 135
16.2.2 Scraping Specic HTML Nodes ...................................... 139
16.2.3 Cleaning Up ..................................................................... 141
16.3 Scraping HTML Table Data........................................................... 143
16.3.1 Scraping HTML Tables with rvest ................................... 143
16.3.2 Scraping HTML Tables with XML .................................. 146
16.4 Working with APIs......................................................................... 150
16.4.1 Prerequisites? ................................................................... 150
16.4.2 Existing API Packages ..................................................... 151
16.4.3 httr for All Things Else .................................................... 158
17 Exporting Data ........................................................................................ 163
17.1 Writing Data to Text Files .............................................................. 163
17.1.1 Base R Functions.............................................................. 163
17.1.2 readr Package ................................................................... 164
17.2 Writing Data to Excel Files............................................................ 165
17.2.1 xlsx Package ..................................................................... 165
17.2.2 r2excel Package ................................................................ 167
17.3 Saving Data as an R Object File .................................................... 169
Part V Creating Efficient and Readable Code in R

18 Functions.................................................................................................. 173
18.1 Function Components .................................................................... 173
18.2 Arguments ...................................................................................... 174
18.3 Scoping Rules ................................................................................ 175
18.4 Lazy Evaluation ............................................................................. 177
18.5 Returning Multiple Outputs from a Function ................................ 177
18.6 Dealing with Invalid Parameters .................................................... 178
18.7 Saving and Sourcing Functions...................................................... 179
19 Loop Control Statements........................................................................ 183
19.1 Basic Control Statements (i.e. if, for, while, etc.) ............ 183
19.1.1 if Statement ...................................................................... 183
19.1.2 ifelse Statement ............................................................ 184
19.1.3 for Loop............................................................................ 186
19.1.4 while Loop ....................................................................... 187
19.1.5 repeat Loop....................................................................... 189
xii Contents
19.1.6 break Function to Exit a Loop.......................................... 189

19.1.7 next Function to Skip an Iteration in a Loop.................... 190
19.2 Apply Family ................................................................................. 190
19.2.1 apply() for Matrices and Data Frames ............................. 191
19.2.2 lapply() for ListsOutput as a List ................................. 192
19.2.3 sapply() for ListsOutput Simplied ............................. 193
19.2.4 tapply() for Vectors .......................................................... 194
19.3 Other Useful Loop-Like Functions ............................................ 195
20 Simplify Your Code with %>% ............................................................. 199
20.1 Pipe (%>%) Operator..................................................................... 199
20.1.1 Nested Option................................................................... 200
20.1.2 Multiple Object Option .................................................... 200
20.1.3 %>% Option ..................................................................... 201
20.2 Additional Functions...................................................................... 203
20.3 Additional Pipe Operators.............................................................. 204
Part VI Shaping and Transforming Your Data with R

21 Reshaping Your Data with tidyr......................................................... 211
21.1 Making Wide Data long ................................................................. 212
21.2 Making Long Data wide ................................................................ 213
21.3 Splitting a Single Column into Multiple Columns ........................ 213
21.4 Combining Multiple Columns into a Single Column .................... 214
21.5 Additional tidyr Functions......................................................... 215
21.6 Sequencing Your tidyr Operations............................................. 217
22 Transforming Your Data with dplyr ................................................... 219
22.1 Selecting Variables of Interest ....................................................... 220
22.2 Filtering Rows ................................................................................ 221
22.3 Grouping Data by Categorical Variables ....................................... 222
22.4 Performing Summary Statistics on Variables ................................ 223
22.5 Arranging Variables by Value ........................................................ 225
22.6 Joining Data Sets............................................................................ 226
22.7 Creating New Variables ................................................................. 228
Index ................................................................................................................. 233

Part I
Introduction
With nothing but the power of your own mind, you operate
on the symbols before you in such a way that you gradually
lift yourself from a state of understanding less to one of
understanding more.
Mortimer J. Adler
Data. Our world has become increasingly reliant upon, and awash in, this resource.
Businesses are increasingly seeking to capitalize on data analysis as a means for
gaining competitive advantages. Government agencies are using more types of data
to improve operations and efficiencies. Sports entities are increasing the range of
data applications, from how teams are using data and analytics to how data are
impacting the experience for the fan base. Journalism is increasing the role that
numerical data are used in the production and distribution of information as evi-
denced by the emerging field of data journalism. In fact, the need to work with data
has become so prevalent that the U.S. alone is expected to have a shortage of
140,000190,000 data analysts by 2018.1 Consequently, it is safe to say there is a
need for becoming fluent with the data analysis process. And Im assuming thats
why you are reading this book.
Fluency in data analysis captures a wide range of activities. At its most basic
structure, data analysis fluency includes the ability to get, clean, transform, visual-
ize, and model data along with communicating your results as depicted in the fol-
lowing illustration.
Knowledge generation & extraction
Visualize
Get Clean Transform Communicate
Model
A modified version of Hadley Wickhams analytic process
Fig. 1 Analytic Process
From project to project, no analytic process will be the same. Each specific
instance of data analysis includes unique, different, and often multiple requirements
regarding the specific processes required for each stage. For instance, getting data
1
Manyika et al. (2011).
2 Part I Introduction
may include simply accessing an Excel file, scraping data from an HTML table, or
using an application programming interface (API) to access a database. Cleaning
data may include reshaping data from a wide to long format, parsing or manipulat-
ing variables to different formats. Transforming data may include filtering, sum-
marizing, and applying common or uncommon functions to data along with joining
multiple datasets. Visualizing data may range from common static exploratory data
analysis plots to dynamic, interactive data visualizations in web browsers. And
modeling data can be even more diverse covering the range of descriptive, predic-
tive, and prescriptive analytic techniques.
Consequently, the road to becoming an expert in data analysis can be daunting.
And, in fact, obtaining expertise in the wide range of data analysis processes uti-
lized in your own respective field is a career long process. However, the goal of this
book is to help you take a step closer to fluency in the early stages of the analytic
process. Why? Because before using statistical literate programming to report your
results, before developing an optimization or predictive model, before performing
exploratory data analysis, and before visualizing your data, you need to be able
to manage your data. You need to be able to import your data. You need to be able to
work with the different data types. You need to be able to subset and parse your data.
You need to be able to manipulate and transform your data. You need to be able to
wrangle your data!
Chapter 1
The Role of Data Wrangling
Water, water, everywhere, nor any a drop to drink

Samuel Taylor Coleridge
Synonymous to Samuel Taylor Coleridges quote in Rime of the Ancient Mariner, the
degree to which data are useful is largely determined by an analysts ability to wrangle
data. In spite of advances in technologies for working with data, analysts still spend an
inordinate amount of time obtaining data, diagnosing data quality issues and pre-pro-
cessing data into a usable form. Research has illustrated that this portion of the data
analysis process is the most tedious and time consuming component; often consuming
5080 % of an analysts time (cf. Wickham 2014; Dasu and Johnson 2003). Despite the
challenges, data wrangling remains a fundamental building block that enables visual-
ization and statistical modeling. Only through data wrangling can we make data useful.
Consequently, ones ability to perform data wrangling tasks effectively and efficiently
is fundamental to becoming an expert data analyst in their respective domain.
So what exactly is this thing called data wrangling? Its the ability to take a
messy, unrefined source of data and wrangle it into something useful. Its the art of
using computer programming to extract raw data and creating clear and actionable
bits of information for your analysis. Data wrangling is the entire front end of the
analytic process and requires numerous tasks that can be categorized within the get,
clean, and transform components (Fig. 1.1).
However, learning how to wrangle your data does not necessarily follow a linear
progression as suggested by Fig. 1.1. In fact, you need to start from scratch to under-
stand how to work with data in R. Consequently, this book takes a meandering route
through the data wrangling process to help build a solid data wrangling
foundation.
First, modern day data wrangling requires being comfortable writing code. If
you are new to writing code, R, or RStudio you need to understand some of the
basics of working in the command line environment. The next two chapters in this
part will introduce you to R, discuss the benefits it provides, and then start to get you
comfortable at the command line by walking you through the process of assigning
and evaluating expressions, using vectorization, getting help, managing your
workspace, and working with packages. Lastly, I offer some basic styling guidelines
to help you write code that is easier to digest by others.
Springer International Publishing Switzerland 2016 3

B.C. Boehmke, Data Wrangling with R, Use R!, DOI 10.1007/978-3-319-45599-0_1
4 1 The Role of Data Wrangling
Fig. 1.1 Data Wrangling
Second, data wrangling requires the ability to work with different forms of data.
Analysts and organizations are finding new and unique ways to leverage all forms
of data so its important to be able to work not only with numbers but also with
character strings, categorical variables, logical variables, regular expression, and
dates. Part II explains how to work with these different classes of data so that when
you start to learn how to manage the different data structures, which combines these
data classes into multiple dimensions, you will have a strong knowledge base.
Third, modern day datasets often contain variables of different lengths and
classes. Furthermore, many statistical and mathematical calculations operate on dif-
ferent types of data structures. Consequently, data wrangling requires a strong
knowledge of the different structures to hold your datasets. Part III covers the differ-
ent types of data structures available in R, how they differ by dimensionality and
how to create, add to, and subset the various data structures. Lastly, I cover how to
deal with missing values in data structures. Consequently, this part provides a robust
understanding of managing various forms of datasets.
Fourth, data are arriving from multiple sources at an alarming rate and analysts
and organizations are seeking ways to leverage these new sources of information.
Consequently, analysts need to understand how to get data from these sources.
Furthermore, since analysis is often a collaborative effort, analysts also need to know
how to share their data. Part IV covers the basics of importing tabular and spread-
sheet data, scraping data stored online, and exporting data for sharing purposes.
Fifth, minimizing duplication and writing simple and readable code is important
to becoming an effective and efficient data analyst. Moreover, clarity should always
be a goal throughout the data analysis process. Part V introduces the art of writing
functions and using loop control statements to reduce redundancy in code. I also
discuss how to simplify your code using pipe operators to make your code more
readable. Consequently, this part will help you to perform data wrangling tasks
more effectively, efficiently, and with more clarity.
Last, data wrangling is all about getting your data into the right form in order to
feed it into the visualization and modeling stages. This typically requires a large
amount of reshaping and transforming of your data. Part VI introduces some of the
fundamental functions for tidying your data and for manipulating, sorting, sum-
marizing, and joining your data. These tasks will help to significantly reduce the
time you spend on the data wrangling process.
Individually, each part will provide you important tools for performing individual
data wrangling tasks. Combined, these tools will help to make you more effective
and efficient in the front end of the data analysis process so that you can spend more
of your time visualizing and modeling your data and communicating your results!
Bibliography 5
Bibliography
Dasu, T., & Johnson, T. (2003). Exploratory Data Mining and Data Cleaning (Vol. 479). John
Wiley & Sons.
Manyika, J., Chui, M., Brown, B., Bughin, J., Dobbs, R., Roxburgh, C., et al. (2011). Big data: The
next frontier for innovation, competition, and productivity. McKinsey.
Wickham, H. (2014). Tidy data. Journal of Statistical Software , 59 (i10).
Chapter 2
Introduction to R
A language for data analysis and graphics. This denition of R was used by Ross
Ihaka and Robert Gentleman in the title of their 1996 paper (Ihaka and Gentleman
1996) outlining their experience of designing and implementing the R software. Its
safe to say this remains the essence of what R is; however, its tough to encapsulate
such a diverse programming language into a single phrase.
During the last decade, the R programming language has become one of the most
widely used tools for statistics and data science. Its application runs the gamut from
data preprocessing, cleaning, web scraping and visualization to a wide range of
analytic tasks such as computational statistics, econometrics, optimization, and
natural language processing. In 2012 R had over two million users and continues to
grow by double-digit percentage points every year. R has become an essential ana-
lytic software throughout industry; being used by organizations such as Google,
Facebook, New York Times, Twitter, Etsy, Department of Defense, and even in
presidential political campaigns. So what makes R such a popular tool?
2.1 Open Source
R is an open source software created over 20 years ago by Ihaka and Gentleman at
the University of Auckland, New Zealand. However, its history is even longer as its
lineage goes back to the S programming language created by John Chambers out of
Bell Labs back in the 1970s.1 R is actually a combination of S with lexical scoping
semantics inspired by Scheme (Morandat and Hill 2012). Whereas the resulting
language is very similar in appearance to S, the underlying implementation and
semantics are derived from Scheme. Unbeknownst to many the S language has been
a popular vehicle for research in statistical methodology, and R provides an open
source route to participate in that activity.
1
Consequently, R is named partly after its authors (Ross and Robert) and partly as a play on the
name of S.

8 2 Introduction to R
Although the history of S and R is interesting,2 the principal artifact to observe is

that R is an open source software. Although some contest that open-source software
is merely a craze,3 most evidence suggests that open-source is here to stay and
represents a new4 norm for programming languages. Open-source software such as
R blurs the distinction between developer and user, which provides the ability to
extend and modify the analytic functionality to your, or your organizations needs.
The data analysis process is rarely restricted to just a handful of tasks with predict-
able input and outputs that can be pre-dened by a xed user interface as is common
in proprietary software. Rather, as previously mentioned in the introduction, data
analyses include unique, different, and often multiple requirements regarding the
specic tasks involved. Open source software allows more exibility for you, the
data analyst, to manage how data are being transformed, manipulated, and modeled
under the hood of software rather than relying on stiff point and click software
interfaces. Open source also allows you to operate on every major platform rather
than be restricted to what your personal budget allows or the idiosyncratic pur-
chases of organizations.
This invariably leads to new expectations for data analysts; however, organiza-
tions are proving to greatly value the increased technical abilities of open source
data analysts as evidenced by a recent OReilly survey revealing that data analysts
focusing on open source technologies make more money than those still dealing in
proprietary technologies.
2.2 Flexibility
Another benet of open source is that anybody can access the source code, modify
and improve it. As a result, many excellent programmers contribute to improving
existing R code and developing new capabilities. Researchers from all walks of life
(academic institutions, industry, and focus groups such as RStudio5 and rOpenSci6)
are contributing to advancements of Rs capabilities and best practices. This has
resulted in some powerful tools that advance both statistical and non-statistical
modeling capabilities that are taking data analysis to new levels.
2
See Roger Pengs R programming for Data Science for further, yet concise, details on S and Rs
history.
3
This was recently argued by Pollack, Klimberg, and Boklage (2015) which was appropriately
rebutted by Boehmke and Jackson (2016).
4
Open-source is far from new as it has been around for decades (i.e. A-2 in the 1950s, IBMs ACP
in the 60s, Tiny BASIC in the 70s) but has gained prominence since the late 1990s.
5
https://www.rstudio.com
6
https://ropensci.org/packages
2.3 Community 9
Many researchers in academic institutions are using and developing R code to

develop the latest techniques in statistics and machine learning. As part of their
research, they often publish an R package to accompany their research articles.7
This provides immediate access to the latest analytic techniques and implementa-
tions. And this research is not soley focused on generalized algorithms as many new
capabilities are in the form of advancing analytic algorithms for tasks in specic
domains. A quick assessment of the different task domains8 for which code is being
developed illustrates the wide spectrumeconometrics, nance, chemometrics and
computational physics, pharmacokinetics, social sciences, etc.
Powerful tools are also being developed to perform many tasks that greatly aid
the data analysis process. This is not limited to just new ways to wrangle your data
but also new ways to visualize and communicate data. R packages are now making
it easier than ever to create interactive graphics and websites and produce sophisti-
cated HTML and PDF reports. R packages are also integrating communication with
high-performance programming languages such as C, Fortran, and C++ making
data analysis more powerful, efcient, and posthaste than ever.
So although the analytic mantra use the right tool for the problem should
always be in our prefrontal cortex, the advancements and exibility of R is making
it the right tool for many problems.
2.3 Community
The R community is fantastically diverse and engaged. On a daily basis, the R com-
munity generates opportunities and resources for learning about R. These cover the
full spectrum of trainingbooks, online courses, R user groups, workshops, confer-
ences, etc. And with over two million users and developers, nding help and techni-
cal expertise is only a simple click away. Support is available through R mailing
lists, Q&A websites, social media networks, and numerous blogs.
So now that you know how awesome R is, its time to learn how to use it.
Bibliography
Ihaka, Ross, and Robert Gentleman. R: A language for data analysis and graphics. Journal of
Computational and Graphical Statistics 5, no. 3 (1996):299314.
Morandat, Floral, Brandon Hill, Leo Osvald, and Jan Vitek. Evaluating the design of the R lan-
guage. In European Conference on Object-Oriented Programming, pp. 104131. Springer
Berlin Heidelberg, 2012.
Pollack, R. D., Klimberg, R. K., and Boklage, S.H. The true cost of free statistical software.
OR/MS Today, vol. 42, no. 5 (2015):3435.
Boehmke, Bradley C. and Jackson, Ross A. Unpacking the true cost of free statistical software.
OR/MS Today, vol. 43, no. 1 (2016):2627.
7
See The Journal of Statistical Software and The R Journal.
8
https://cran.r-project.org/web/views/
Chapter 3
The Basics
Programming is like kicking yourself in the face, sooner or later

your nose will bleed.
Kyle Woodbury
A computer language is described by its syntax and semantics; where syntax is

about the grammar of the language and semantics the meaning behind the sentence.
And jumping into a new programming language correlates to visiting a foreign
country with only that ninth grade Spanish 101 class under your belt; there is no
better way to learn than to immerse yourself in the environment! Although itll be
painful early on and your nose will surely bleed, eventually youll learn the dialect
and the quirks that come along with it.
Throughout this book youll learn much of the fundamental syntax and seman-
tics of the R programming language; and hopefully with minimal face kicking
involved. However, this chapter serves to introduce you to many of the basics of R
to get you comfortable. This includes installing R and RStudio, understanding the
console, how to get help, how to work with packages, understanding how to assign
and evaluate expressions, and the idea of vectorization. Finally, I offer some basic
styling guidelines to help you write code that is easier to digest by others.
3.1 Installing R and RStudio
First, you need to download and install R, a free software environment for statistical
computing and graphics from CRAN, the Comprehensive R Archive Network. It is
highly recommended to install a precompiled binary distribution for your operating
system; follow these instructions:
1. Go to https://cran.r-project.org/
2. Click Download R for Mac/Windows
3. Download the appropriate le:
(a) Windows users click Base, and download the installer for the latest R
version
(b) Mac users select the le R-3.X.X.pkg that aligns with your OS version

12 3 The Basics
4. Follow the instructions of the installer

Next, you can download RStudios IDE (integrated development environment),
a powerful user interface for R. RStudio includes a text editor, so you do not have to
install another stand-alone editor. Follow these instructions:
1. Go to RStudio for desktop https://www.rstudio.com/products/rstudio/download/
2. Select the install le for your OS
3. Follow the instructions of the installer.
There are other R IDEs available: Emacs, Microsoft R Open, Notepad++, etc;
however, I have found RStudio to be my preferred route. When you are done install-
ing RStudio click on the icon that looks like (Fig. 3.1):
Fig. 3.1 RStudio Icon
and you should get a window that looks like the following (Fig. 3.2):
You are now ready to start programming!
Fig. 3.2 RStudio Console

3.2 Understanding the Console 13
3.2 Understanding the Console
The RStudio console is where all the action happens. There are four fundamental
windows in the console, each with their own purpose. I discuss each briey below
but I highly suggest Oscar Torres-Reynas Introduction to RStudio1 for a thorough
understanding of the console (Fig. 3.3).
3.2.1 Script Editor
The top left window is where your script les will display. There are multiple forms
of script les but the basic one to start with is the .R le. To create a new le you use
the File New File menu. To open an existing le you use either the File Open
File menu or the Recent Files menu to select from recently opened les.
RStudios script editor includes a variety of productivity enhancing features includ-
ing syntax highlighting, code completion, multiple-le editing, and nd/replace.
A good introduction to the script editor was written by RStudios Josh Paulson.2
3.2.2 Workspace Environment
The top right window is the workspace environment which captures much of your
current R working environment and includes any user-dened objects (vectors,
matrices, data frames, lists, functions). When saving your R working session, these
Fig. 3.3 Four fundamental windows of the RStudio console
1
You can access this tutorial at http://dss.princeton.edu/training/RStudio101.pdf
2
You can assess the script editor tutorial at https://support.rstudio.com/hc/en-us/articles/
200484448-Editing-and-Executing-Code
14 3 The Basics
are the components along with the script les that will be saved in your working
directory, which is the default location for all le inputs and outputs. To get or set
your working directory so you can direct where your les are saved use getwd and
setwd in the console (note that you can type any comments in your code by pre-
ceding the comment with the hashtag (#) symbol; any values, symbols, and texts
following # will not be evaluated.).
# returns path for the current working directory
getwd()
# set the working directory to a specified directory

setwd(directory_name)
For example, if I call getwd() the le path /Users/bradboehmke/Desktop/

Personal/Data Wrangling is returned. If I want to set the working directory to the
Workspace folder within the Data Wrangling directory I would
use setwd("Workspace"). Now if I call getwd() again it returns /Users/
bradboehmke/ Desktop/Personal/Data Wrangling/Workspace.
The workspace environment will also list your user-dened objects such as vec-
tors, matrices, data frames, lists, and functions. To identify or remove the objects
(i.e. vectors, data frames, user dened functions, etc.) in your current R
environment:
# list all objects
ls()
# identify if an R object with a given name is present

exists("object_name")
# remove defined object from the environment

rm("object_name")
# you can remove multiple objects by using the c() function

rm(c("object1", "object2"))
# basically removes everything in the working environment -- use with

# caution!
rm(list = ls())
You can also view previous commands in the workspace environment by clicking
the History tab, by simply pressing the up arrow on your keyboard, or by typing into
the console:
# default shows 25 most recent commands
history()
# show 100 most recent commands

history(100)
# show entire saved history

history(Inf)
3.2 Understanding the Console 15
You can also save and load your workspaces. Saving your workspace will save all
R les and objects within your workspace to a .RData le in your working directory
and loading your workspace will load any .RData les in your working directory.
# save all items in workspace to a .RData file
save.image()
# save specified objects to a .RData file

save(object1, object2, file = "myfile.RData")
# load workspace into current session
load("myfile.RData")
Note that saving the workspace without specifying the working directory will
default to saving in the current directory. You can further specify where to save the
.RData by including the path: save(object1, object2, file = "/users/
name/folder/myfile.RData"). More information regarding saving and
loading R objects such as .RData les will be discussed in Part IV of this book.
3.2.3 Console
The bottom left window contains the console. You can code directly in this window
but it will not save your code. It is best to use this window when you are simply
performing calculator type functions. This is also where your outputs will be pre-
sented when you run code in your script.
3.2.4 Misc. Displays
The bottom right window contains multiple tabs. The Files tab allows you to see
which les are available in your working directory. The Plots tab will display any
visualizations that are produced by your code. The Packages tab will list all pack-
ages downloaded to your computer and also the ones that are loaded (more on this
concept of packages shortly). And the Help tab allows you to search for topics you
need help on and will also display any help responses (more on this later as well).
3.2.5 Workspace Options and Shortcuts
There are multiple options available for you to set and customize your console. You
can view and set options for the current R session:
# learn about available options
help(options)
16 3 The Basics
# view current option settings

options()
# change a specific option (i.e. number of digits to print on output)

options(digits=3)
As with most computer programs, there are numerous keyboard shortcuts for
working with the console. To access a menu displaying all the shortcuts in RStudio
you can use option + shift + k. Within RStudio you can also access them in the Help
menu Keyboard Shortcuts. You can also nd the RStudio console cheatsheet by
going to Help menu Cheatsheets.
3.3 Getting Help
Learning any new language requires lots of help. Luckily, the help documentation
and support in R is comprehensive and easily accessible from the command line. To
leverage general help resources you can use the following:
3.3.1 General Help
To leverage general help resources you can use:

# provides general help links
help.start()
# searches the help system for documentation matching a given character

# string
help.search("text")
Note that the help.search("some text here") function requires a

character string enclosed in quotation marks. So if you are in search of time series
functions in R, using help.search("time series") will pull up a healthy
list of vignettes and code demonstrations that illustrate packages and functions that
work with time series data.
3.3.2 Getting Help on Functions
For more direct help on functions that are installed on your computer:
# provides details for specific function
help(functionname)
3.4 Working with Packages 17
# provides same information as help(functionname)

?functionname
# provides examples for said function

example(functionname)
Note that the help() and ? functions only work for functions within loaded
packages. If you want to see details on a function in a package that is installed on
your computer but not loaded in the active R session you can use
help(functionname, package = "packagename"). Another alterna-
tive is to use the :: operator as in help(packagename::functionname).
3.3.3 Getting Help from the Web
Typically, a problem you may be encountering is not new and others have faced,
solved, and documented the same issue online. The following resources can be used
to search for online help. Although, I typically just google the problem and nd
answers relatively quickly.
RSiteSearch("key phrase"): searches for the key phrase in help manuals
and archived mailing lists on the R Project website at http://search.r-project.org/.
Stack Overow: a searchable Q&A site oriented toward programming issues.
75 % of my answers typically come from Stack Overow questions tagged for R
at http://stackoverow.com/questions/tagged/r.
Cross Validated: a searchable Q&A site oriented toward statistical analysis.
Many questions regarding specic statistical functions in R are tagged for R at
http://stats.stackexchange.com/questions/tagged/r.
R-seek: a Google custom search that is focused on R-specic websites. Located
at http://rseek.org/
R-bloggers: a central hub of content collected from over 500 bloggers who pro-
vide news and tutorials about R. Located at http://www.r-bloggers.com/
3.4 Working with Packages
In R, the fundamental unit of shareable code is the package. A package bundles

together code, data, documentation, and tests and provides an easy method to share
with others. As of June 2016 there were over 8000 packages available on CRAN,
1000 on Bioconductor, and countless more available through GitHub. This huge
variety of packages is one of the reasons that R is so successful: chances are that
someone has already solved a problem that youre working on, and you can benet
from their work by downloading their package.
18 3 The Basics
3.4.1 Installing Packages
Your primary source to obtain packages will likely be from CRAN. To install pack-
ages from CRAN:
# install packages from CRAN
install.packages("packagename")
As previously stated, packages are also available through Bioconductor and

GitHub. To download Bioconductor packages:
# link to Bioconductor URL
source("http://bioconductor.org/biocLite.R")
# install core Bioconductor packages

biocLite()
# install specific Bioconductor package

biocLite("packagename")
And to download GitHub packages:

# the devtools package provides a simply function to download GitHub
# packages
install.packages("devtools")
# install package which exists at github.com/username/packagename

devtools::install_github("username/packagename")
3.4.2 Loading Packages
Once the package is downloaded to your computer you can access the functions and
resources provided by the package in two different ways:
# load the package to use in the current R session
library(packagename)
# use a particular function within a package without loading the package

packagename::functionname
For instance, if you want to have full access to the tidyr package you would
use library(tidyr); however, if you just wanted to use the gather() func-
tion without loading the tidyr package you can use tidyr::gather(function
arguments).
3.5 Assignment and Evaluation 19
3.4.3 Getting Help on Packages
For help on packages that are installed on your computer:

# provides details regarding contents of a package
help(package = "packagename")
# see all packages installed

library()
# see packages currently loaded

search()
# list vignettes available for a specific package

vignette(package = "packagename")
# view specific vignette

vignette("vignettename")
# view all vignettes on your computer

vignette()
Note that some packages will have multiple vignettes. For instance
vignette(package = "grid") will list the 13 vignettes available for the
grid package. To access one of the specic vignettes you simply use vignette
("vignettename").
3.4.4 Useful Packages
There are thousands of helpful R packages for you to use, but navigating them all can
be a challenge. To help you out, RStudio compiled a guide3 to some of the best pack-
ages for loading, manipulating, visualizing, analyzing, and reporting data. In addi-
tion, their list captures packages that specialize in spatial data, time series and nancial
data, increasing speed and performance, and developing your own R packages.
3.5 Assignment and Evaluation
The rst operator youll run into is the assignment operator. The assignment opera-
tor is used to assign a value. For instance we can assign the value 3 to the vari-
able x using the <- assignment operator. We can then evaluate the variable by simply
typing x at the command line which will return the value of x. Note that prior to the
value returned youll see ## [1] in the command line. This simply implies that the
output returned is the rst output.
3
https://support.rstudio.com/hc/en-us/articles/201057987-Quick-list-of-useful-R-packages
20 3 The Basics
# assignment
x <- 3
# evaluation
x
## [1] 3
Interestingly, R actually allows for ve assignment operators:

# leftward assignment
x <- value
x = value
x <<- value
# rightward assignment
value -> x
value ->> x
The original assignment operator in R was <- and has continued to be the pre-
ferred among R users. The = assignment operator was added in 20014 primarily
because it is the accepted assignment operator in many other languages and begin-
ners to R coming from other languages were so prone to use it. However, R uses = to
associate function arguments with values (i.e. f(x = 3) explicitly means to call func-
tion f and set the argument x to 3). Consequently, most R programmers prefer to
keep = reserved for argument association and use <- for assignment.
The operator <<- is normally only used in functions which we will not get into
the details. And the rightward assignment operators perform the same as their left-
ward counterparts; they just assign the value in an opposite direction.
Overwhelmed yet? Dont be. This is just meant to show you that there are options
and you will likely come across them sooner or later. My suggestion is to stick with
the tried and true <- operator. This is the most conventional assignment operator
used and is what you will nd in all the base R source codewhich means it should
be good enough for you.
Lastly, note that R is a case sensitive programming language. Meaning all vari-
ables, functions, and objects must be called by their exact spelling:
x <- 1
y <- 3
z <- 4
x * y * z
## [1] 12
x * Y * z
## Error in eval(expr, envir, enclos): object 'Y' not found
4
See http://developer.r-project.org/equalAssign.html for more details.
3.6 R as a Calculator 21
3.6 R as a Calculator
At its most basic function R can be used as a calculator. When applying basic arith-
metic, the PEMBDAS order of operations applies: parentheses rst followed
by exponentiation, multiplication and division, and nally addition and subtraction.
8 + 9 / 5 ^ 2
## [1] 8.36
8 + 9 / (5 ^ 2)
## [1] 8.36
8 + (9 / 5) ^ 2
## [1] 11.24
(8 + 9) / 5 ^ 2
## [1] 0.68
By default R will display seven digits but this can be changed using options()
as previously outlined.
1 / 7
## [1] 0.1428571
options(digits = 3)
1 / 7
## [1] 0.143
Also, large numbers will be expressed in scientic notation which can also be
adjusted using options().
888888 * 888888
## [1] 7.9e+11
options(digits = 10)
888888 * 888888
## [1] 790121876544
Note that the largest number of digits that can be displayed is 22. Requesting any
larger number of digits will result in an error message.
pi
## [1] 3.141592654
pi
## [1] 3.141592653589793115998
22 3 The Basics
## Error in options(digits = 23): invalid 'digits' parameter, allowed 022

pi
## [1] 3.141592653589793115998
When performing undened calculations R will produce Inf and NaN outputs.
1 / 0 # infinity
## [1] Inf
Inf - Inf # infinity minus infinity

## [1] NaN
-1 / 0 # negative infinity
## [1] -Inf
0 / 0 # not a number
## [1] NaN
sqrt(-9) # square root of -9

## Warning in sqrt(-9): NaNs produced
## [1] NaN
The last two functions to mention are the integer divide (%/%) and modulo (%%)
functions. The integer divide function will give the integer part of a fraction while
the modulo will provide the remainder.
42 / 4 # regular division
## [1] 10.5
42 %/% 4 # integer division

## [1] 10
42 %% 4 # modulo (remainder)
## [1] 2
3.6.1 Vectorization
A key difference between R and many other languages is a topic known as vector-
ization. What does this mean? It means that many functions that are to be applied
individually to each element in a vector of numbers require a loop assessment to
evaluate; however, in R many of these functions have been coded in C to perform
much faster than a for loop would perform. For example, lets say you want to add
the elements of two separate vectors of numbers (x and y).
x <- c(1, 3, 4)
y <- c(1, 2, 4)
x ## [1] 1 3 4
y ## [1] 1 2 4
3.6 R as a Calculator 23
In other languages you might have to run a loop to add two vectors together. In
this for loop I print each iteration to show that the loop calculates the sum for the
rst elements in each vector, then performs the sum for the second elements, etc.
# empty vector
z <- as.vector(NULL)
# for loop to add corresponding elements in each vector

for (i in seq_along(x)) {
z[i] <- x[i] + y[i]
print(z)
}
## [1] 2
## [1] 2 5
## [1] 2 5 8
Instead, in R, + is a vectorized function which can operate on entire vectors at once.

So rather than creating for loops for many functions, you can just use simple syntax:
x + y
## [1] 2 5 8
x * y
## [1] 1 6 16
x > y
## [1] FALSE TRUE FALSE
When performing vector operations in R, it is important to know about recycling.

When performing an operation on two or more vectors of unequal length, R will
recycle elements of the shorter vector(s) to match the longest vector. For example:
long <- 1:10
short <- 1:5
long
## [1] 1 2 3 4 5 6 7 8 9 10
short
## [1] 1 2 3 4 5
long + short
## [1] 2 4 6 8 10 7 9 11 13 15
The elements of long and short are added together starting from the rst ele-
ment of both vectors. When R reaches the end of the short vector, it starts again at
the rst element of short and continues until it reaches the last element of
the long vector. This functionality is very useful when you want to perform the
same operation on every element of a vector. For example, say we want to multiply
every element of our long vector by 3:
24 3 The Basics
long <- 1:10

c <- 3
long * c
## [1] 3 6 9 12 15 18 21 24 27 30
Remember there are no scalars in R, so c is actually a vector of length 1; in order

to add its value to every element of long, it is recycled to match the length of long.
When the length of the longer object is a multiple of the shorter object length, the
recycling occurs silently. When the longer object length is not a multiple of the
shorter object length, a warning is given:
even_length <- 1:10
odd_length <- 1:3
even_length + odd_length
## Warning in even_length + odd_length: longer object length is not a
## multiple of shorter object length
## [1] 2 4 6 5 7 9 8 10 12 11
3.7 Styling Guide
Good coding style is like using correct punctuation. You can manage without it, but it sure
makes things easier to read.Hadley Wickham
As a medium of communication, its important to realize that the readability of

code does in fact make a difference. Well-styled code has many benets to include
making it easy to read, extend, and debug. Unfortunately, R does not come with
ofcial guidelines for code styling but such is an inconvenient truth of most open
source software. However, this should not lead you to believe there is no style to be
followed and over time implicit guidelines for proper code styling have been docu-
mented. What follows are guidelines that have been widely accepted as good prac-
tice in the R community and are based on Googles and Hadley Wickhams R style
guides.5
3.7.1 Notation and Naming
File names should be meaningful and end with a .R extension.

# Good
weather-analysis.R
emerson-text-analysis.R
# Bad
basic-stuff.r
detail.r
5
Googles style guide can be found at https://google.github.io/styleguide/Rguide.xml and Hadley
Wickhams can be found at http://adv-r.had.co.nz/Style.html
3.7 Styling Guide 25
If les need to be run in sequence, prex them with numbers:

0-download.R
1-preprocessing.R
2-explore.R
3-fit-model.R
In R, naming conventions for variables and function are famously muddled.

They include the following:
namingconvention # all lower case; no separator

naming.convention # period separator
naming_convention # underscore separator
namingConvention # lower camel case
NamingConvention # upper camel case
Historically, there has been no clearly preferred approach with multiple naming
styles sometimes used within a single package. Bottom line, your naming conven-
tion will be driven by your preference but the ultimate goal should be consistency.
My personal preference is to use all lowercase with an underscore (_) to separate
words within a name. This follows Hadley Wickhams suggestions in his style
guide. Furthermore, variable names should be nouns and function names should be
verbs to help distinguish their purpose. Also, refrain from using existing names of
functions (i.e. mean, sum, true).
3.7.2 Organization
Organization of your code is also important. Theres nothing like trying to decipher
2000 lines of code that has no organization. The easiest way to achieve organization
is to comment your code. The general commenting scheme I use is the following.
I break up principal sections of my code that have a common purpose with:
#################
# Download Data #
#################
lines of code here
###################
# Preprocess Data #
###################
lines of code here
########################
# Exploratory Analysis #
########################
lines of code here
26 3 The Basics
Then comments for specic lines of code can be done as follows:

code_1 # short comments can be placed to the right of code
code_2 # blah
code_3 # blah
# or comments can be placed above a line of code

code_4
# Or extremely long lines of commentary that go beyond the suggested 80

# characters per line can be broken up into multiple lines. Just don't
# forget to use the hash on each.
code_5
3.7.3 Syntax
The maximum number of characters on a single line of code should be 80 or less. If

you are using RStudio you can have a margin displayed so you know when you need
to break to a new line.6 This allows your code to be printed on a normal 8.5 11 page
with a reasonably sized font. Also, when indenting your code use two spaces rather
than using tabs. The only exception is if a line break occurs inside parentheses. In
this case align the wrapped line with the rst character inside the parenthesis:
super_long_name <- seq(ymd_hm("2015-1-1 0:00"),
ymd_hm("2015-1-1 12:00"),
by = "hour")
Proper spacing within your code also helps with readability. The following pulls
straight from Hadley Wickhams suggestions.7 Place spaces around all inx opera-
tors (=, +, -, <-, etc.). The same rule applies when using = in function calls. Always
put a space after a comma, and never before.
# Good
average <- mean(feet / 12 + inches, na.rm = TRUE)
# Bad
average<-mean(feet/12+inches,na.rm=TRUE)
Theres a small exception to this rule: :, :: and ::: dont need spaces around them.
6
Go to RStudio on the menu bar then Preferences > Code > Display and you can select the show
margin option and set the margin to 80.
7
http://adv-r.had.co.nz/Style.html
3.7 Styling Guide 27
# Good
x <- 1:10
base::get
# Bad
x <- 1 : 10
base :: get
It is important to think about style when communicating any form of language.

Writing code is no exception and is especially important if others will read your
code. Following these basic style guides will get you on the right track for writing
code that can be easily communicated to others.
Part II
Working with Different Types of Data in R
Wait, there are different types of data?

R is a flexible language that allows you to work with many different forms of data.
This includes numeric, character, categorical, dates, and logical. Technically, R
classifies all the different types of data into five classes:
integer
numeric
character
complex
logical
Modern day analysis typically deals with every class so its important to gain
fluency in dealing with these data forms. This section covers the fundamentals of
handling the different data classes. First I cover the basics of dealing with numbers
so you understand the different classes of numbers, how to generate number
sequences, compare numeric values, and round. I then provide an introduction to
working with characters to get you comfortable with character string manipulation
and set operations. This prepares you to then learn about regular expressions which
deals with search patterns for character classes. Next I introduce factors, also refer-
red to as categorical variables, and how to create, convert, order, and re-level this
data class. Lastly, I cover how to manage dates as this can be a persnickety type of
variable when performing data analysis. Throughout several of these chapters youll
also gain an understanding of the TRUE/FALSE logical variables.
Together, this will give you a solid foundation for dealing with the basic data
classes in R so that when you start to learn how to manage the different data struc-
tures, which combines these data classes into multiple dimensions, you will have a
strong base from which to start.
Chapter 4
Dealing with Numbers
In this chapter you will learn the basics of working with numbers in R. This includes
understanding how to manage the numeric type (integer vs. double), the different
ways of generating non-random and random numbers, how to set seed values for
reproducible random number generation, and the different ways to compare and
round numeric values.
4.1 Integer vs. Double
The two most common numeric classes used in R are integer and double (for
double precision floating point numbers). R automatically converts between these
two classes when needed for mathematical purposes. As a result, its feasible to
use R and perform analyses for years without specifying these differences.
To check whether a pre-existing vector is made up of integer or double values you
can use typeof(x) which will tell you if the vector is a double, integer, logical,
or character type.
4.1.1 Creating Integer and Double Vectors
By default, when you create a numeric vector using the c() function it will produce
a vector of double precision numeric values. To create a vector of integers using
c() you must specify explicity by placing an L directly after each number.

32 4 Dealing with Numbers
# create a string of double-precision values

dbl_var <- c(1, 2.5, 4.5)
dbl_var
## [1] 1.0 2.5 4.5
# placing an L after the values creates a string of integers
int_var <- c(1L, 6L, 10L)
int_var
## [1] 1 6 10
4.1.2 Converting Between Integer and Double Values
By default, if you read in data that has no decimal points or you create numeric
values using the x <- 1:10 method the numeric values will be coded as integer.
If you want to change a double to an integer or vice versa you can specify one of the
following:
# converts integers to double-precision values

as.double(int_var)
## [1] 1 6 10
# identical to as.double()
as.numeric(int_var)
## [1] 1 6 10
# converts doubles to integers
as.integer(dbl_var)
## [1] 1 2 4
4.2 Generating Sequence of Non-random Numbers
There are a few R operators and functions that are especially useful for creating
vectors of non-random numbers. These functions provide multiple ways for gener-
ating sequences of numbers.
4.2.1 Specifing Numbers Within a Sequence
To explicitly specify numbers in a sequence you can use the colon : operator to
specify all integers between two specified numbers or the combine c() function to
explicity specify all numbers in the sequence.
4.3 Generating Sequence of Random Numbers 33
# create a vector of integers between 1 and 10

1:10
## [1] 1 2 3 4 5 6 7 8 9 10
# create a vector consisting of 1, 5, and 10
c(1, 5, 10)
## [1] 1 5 10
# save the vector of integers between 1 and 10 as object x
x <- 1:10
x
## [1] 1 2 3 4 5 6 7 8 9 10
4.2.2 Generating Regular Sequences
A generalization of : is the seq() function, which generates a sequence of num-

bers with a specified arithmetic progression.
# generate a sequence of numbers from 1 to 21 by increments of 2
seq(from = 1, to = 21, by = 2)
## [1] 1 3 5 7 9 11 13 15 17 19 21
# generate a sequence of numbers from 1 to 21 that has 15 equal
# incremented numbers
seq(0, 21, length.out = 15)
## [1] 0.0 1.5 3.0 4.5 6.0 7.5 9.0 10.5 12.0 13.5 15.0 16.5
## [13] 18.0 19.5 21.0
The rep() function allows us to conveniently repeat specified constants into

long vectors. This function allows for collated and non-collated repetitions.
# replicates the values in x a specified number of times
rep(1:4, times = 2)
## [1] 1 2 3 4 1 2 3 4
# replicates the values in x in a collated fashion
rep(1:4, each = 2)
## [1] 1 1 2 2 3 3 4 4
4.3 Generating Sequence of Random Numbers
Simulation is a common practice in data analysis. Sometimes your analysis requires

the implementation of a statistical procedure that requires random number
generation or sampling (i.e. Monte Carlo simulation, bootstrap sampling, etc).
R comes with a set of pseudo-random number generators that allow you to simulate
the most common probability distributions such as Uniform, Normal, Binomial,
Poisson, Exponential and Gamma.
4.3.1 Uniform Numbers
To generate random numbers from a uniform distribution you can use the runif()
function. Alternatively, you can use sample() to take a random sample using with
or without replacements.
# generate n random numbers between the default values of 0 and 1
runif(n)
# generate n random numbers between 0 and 25
runif(n, min = 0, max = 25)
# generate n random numbers between 0 and 25 (with replacement)
sample(0:25, n, replace = TRUE)
# generate n random numbers between 0 and 25 (without replacement)
sample(0:25, n, replace = FALSE)
For example, to generate 25 random numbers between the values 0 and 10:
runif(25, min = 0, max = 10)

## [1] 6.11473003 9.72918761 0.04977565 0.98291110 8.53146606 1.17408103
## [7] 1.09907810 5.83266343 8.04336903 1.70783108 3.13275943 1.28380380
## [13] 8.67087873 8.02653947 7.23398025 4.62386458 3.03617622 6.10895175
## [19] 6.39970018 9.02183043 3.24990736 4.64181107 5.35496769 9.97374324
## [25] 3.30954880
For each non-uniform probability distribution there are four primary functions
available to generate random numbers, density (aka probability mass function),
cumulative density, and quantiles. The prefixes for these functions are:
r: random number generation
d: density or probability mass function
p: cumulative distribution
q: quantiles
4.3.2 Normal Distribution Numbers
The normal (or Gaussian) distribution is the most common and well known distri-
bution. Within R, the normal distribution functions are written as norm().
4.3 Generating Sequence of Random Numbers 35
# generate n random numbers from a normal distribution with given

# mean and standard deviation
rnorm(n, mean = 0, sd = 1)
# generate CDF probabilities for value(s) in vector q
pnorm(q, mean = 0, sd = 1)
# generate quantile for probabilities in vector p
qnorm(p, mean = 0, sd = 1)
# generate density function probabilites for value(s) in vector x
dnorm(x, mean = 0, sd = 1)
For example, to generate 25 random numbers from a normal distribution with

mean = 100 and standard deviation = 15:
x <- rnorm(25, mean = 100, sd = 15)
x
## [1] 97.43216 98.98658 96.43514 73.77727 100.51316 103.11050 111.36823
## [8] 102.09288 101.16769 114.54549 99.28044 97.51866 110.57522 87.85074
## [15] 86.67675 108.95660 88.45750 106.28923 114.22225 80.17450 110.39667
## [22] 96.87112 112.30709 110.54963 93.24365
summary(x)
## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 73.78 96.44 100.50 100.10 110.40 114.50
You can also pass a vector of values. For instance, say you want to know the
CDF probabilities for each value in the vector x created above:
pnorm(x, mean = 100, sd = 15)
## [1] 0.43203732 0.47306731 0.40607337 0.04021628 0.51364538 0.58213815
## [7] 0.77573919 0.55548261 0.53102479 0.83390182 0.48086992 0.43430567
## [13] 0.75959941 0.20898424 0.18721209 0.72478191 0.22079836 0.66249503
## [19] 0.82847339 0.09313407 0.75588023 0.41738339 0.79402667 0.75906822
## [25] 0.32620260
4.3.3 Binomial Distribution Numbers
This is conventionally interpreted as the number of successes in size = x trials

and with prob = p probability of success:
# generate a vector of length n displaying the number of successes

# from a trial size = 100 with a probability of success = 0.5
rbinom(n, size = 100, prob = 0.5)
# generate CDF probabilities for value(s) in vector q
pbinom(q, size = 100, prob = 0.5)
# generate quantile for probabilities in vector p

qbinom(p, size = 100, prob = 0.5)
dbinom(x, size = 100, prob = 0.5)
4.3.4 Poisson Distribution Numbers
The Poisson distribution is a discrete probability distribution that expresses the

probability of a given number of events occurring in a fixed interval of time and/or
space if these events occur with a known average rate and independently of the time
since the last event.
# generate a vector of length n displaying the random number of
# events occurring when lambda (mean rate) equals 4.
rpois(n, lambda = 4)
# generate CDF probabilities for value(s) in vector q when lambda
# (mean rate) equals 4.
ppois(q, lambda = 4)
# generate quantile for probabilities in vector p when lambda
# (mean rate) equals 4.
qpois(p, lambda = 4)
# when lambda (mean rate) equals 4.
dpois(x, lambda = 4)
4.3.5 Exponential Distribution Numbers
The Exponential probability distribution describes the time between events in a

Poisson process.
# generate a vector of length n with rate = 1
rexp(n, rate = 1)
# generate CDF probabilities for value(s) in vector q when rate = 4.
pexp(q, rate = 1)
# generate quantile for probabilities in vector p when rate = 4.
qexp(p, rate = 1)
# when rate = 4.
dexp(x, rate = 1)
4.5 Comparing Numeric Values 37
4.3.6 Gamma Distribution Numbers
The Gamma probability distribution is related to the Beta distribution and arises
naturally in processes for which the waiting times between Poisson distributed
events are relevant.
# generate a vector of length n with shape parameter = 1

rgamma(n, shape = 1)
# generate CDF probabilities for value(s) in vector q when shape
# parameter = 1.
pgamma(q, shape = 1)
# generate quantile for probabilities in vector p when shape
# parameter = 1.
qgamma(p, shape = 1)
# when shape parameter = 1.
dgamma(x, shape = 1)
4.4 Setting the Seed for Reproducible Random Numbers
If you want to generate a sequence of random numbers and then be able to repro-
duce that same sequence of random numbers later you can set the random number
seed generator with set.seed(). This is a critical aspect of reproducible research.
For example, we can reproduce a random generation of 10 values from a normal
distribution:
set.seed(197)
rnorm(n = 10, mean = 0, sd = 1)
## [1] 0.6091700 -1.4391423 2.0703326 0.7089004 0.6455311 0.7290563
## [7] -0.4658103 0.5971364 -0.5135480 -0.1866703
set.seed(197)
rnorm(n = 10, mean = 0, sd = 1)
## [1] 0.6091700 -1.4391423 2.0703326 0.7089004 0.6455311 0.7290563
## [7] -0.4658103 0.5971364 -0.5135480 -0.1866703
4.5 Comparing Numeric Values
There are multiple ways to compare numeric values and vectors. This includes
logical operators along with testing for exact equality and also near equality.
4.5.1 Comparison Operators
The normal binary operators allow you to compare numeric values and provide the
answer in logical form:
x < y # is x less than y
x > y # is x greater than y
x <= y # is x less than or equal to y
x >= y # is x greater than or equal to y
x == y # is x equal to y
x != y # is x not equal to y
These operations can be used for single number comparison:
x <- 9
y <- 10
x == y
## [1] FALSE
and also for comparison of numbers within vectors:
x <- c(1, 4, 9, 12)

y <- c(4, 4, 9, 13)
x == y
## [1] FALSE TRUE TRUE FALSE
Note that logical values TRUE and FALSE equate to 1 and 0 respectively. So if
you want to identify the number of equal values in two vectors you can wrap the
operation in the sum() function:
# How many pairwise equal values are in vectors x and y

sum(x == y)
## [1] 2
If you need to identify the location of pairwise equalities in two vectors you can
wrap the operation in the which() function:
# Where are the pairwise equal values located in vectors x and y
which(x == y)
## [1] 2 3
4.6 Rounding Numbers 39
4.5.2 Exact Equality
To test if two objects are exactly equal:
x <- c(4, 4, 9, 12)

y <- c(4, 4, 9, 13)
identical(x, y)
## [1] FALSE
x <- c(4, 4, 9, 12)
y <- c(4, 4, 9, 12)
identical(x, y)
## [1] TRUE
4.5.3 Floating Point Comparison
Sometimes you wish to test for near equality. The all.equal() function
allows you to test for equality with a difference tolerance of 1.5e8.
x <- c(4.00000005, 4.00000008)
y <- c(4.00000002, 4.00000006)
all.equal(x, y)
## [1] TRUE
If the difference is greater than the tolerance level the function will return the
mean relative difference:
x <- c(4.005, 4.0008)

y <- c(4.002, 4.0006)
all.equal(x, y)
## [1] "Mean relative difference: 0.0003997102"
4.6 Rounding Numbers
There are many ways of rounding to the nearest integer, up, down, or toward a
specified decimal place. The following illustrates the common ways to round.
x <- c(1, 1.35, 1.7, 2.05, 2.4, 2.75, 3.1, 3.45, 3.8, 4.15,
4.5, 4.85, 5.2, 5.55, 5.9)
# Round to the nearest integer
round(x)
## [1] 1 1 2 2 2 3 3 3 4 4 4 5 5 6 6
# Round up
ceiling(x)
## [1] 1 2 2 3 3 3 4 4 4 5 5 5 6 6 6
# Round down
floor(x)
## [1] 1 1 1 2 2 2 3 3 3 4 4 4 5 5 5
# Round to a specified decimal
round(x, digits = 1)
## [1] 1.0 1.4 1.7 2.0 2.4 2.8 3.1 3.5 3.8 4.2 4.5 4.8 5.2 5.5 5.9
Chapter 5
Dealing with Character Strings
Dealing with character strings is often under-emphasized in data analysis training. The
focus typically remains on numeric values; however, the growth in data collection is
also resulting in greater bits of information embedded in character strings. Consequently,
handling, cleaning and processing character strings is becoming a prerequisite in daily
data analysis. This chapter is meant to give you the foundation of working with char-
acters by covering some basics followed by learning how to manipulate strings using
base R functions along with using the simplied stringr package.
5.1 Character String Basics
In this section youll learn the basics of creating, converting and printing character
strings followed by how to assess the number of elements and characters in a string.
5.1.1 Creating Strings
The most basic way to create strings is to use quotation marks and assign a string to
an object similar to creating number sequences.
a <- "learning to create" # create string a
b <- "character strings" # create string b
The paste() function provides a versatile means for creating and building
strings. It takes one or more R objects, converts them to character, and then it
concatenates (pastes) them to form one or several character strings.
# paste together string a & b
paste(a, b)

42 5 Dealing with Character Strings
## [1] "learning to create character strings"
# paste character and number strings (converts numbers to

# character class)
paste("The life of", pi)
## [1] "The life of 3.14159265358979"
# paste multiple strings

paste("I", "love", "R")
## [1] "I love R"
# paste multiple strings with a separating character

paste("I", "love", "R", sep = "-")
## [1] "I-love-R"
# use paste0() to paste without spaces btwn characters

paste0("I", "love", "R")
## [1] "IloveR"
# paste objects with different lengths

paste("R", 1:5, sep = " v1.")
## [1] "R v1.1" "R v1.2" "R v1.3" "R v1.4" "R v1.5"
5.1.2 Converting to Strings
Test if strings are characters with is.character() and convert strings to char-
acter with as.character() or with toString().
a <- "The life of"
b <- pi
is.character(a)
## [1] TRUE
is.character(b)
## [1] FALSE
c <- as.character(b)
is.character(c)
## [1] TRUE
toString(c("Aug", 24, 1980))
## [1] "Aug, 24, 1980"

5.1 Character String Basics 43
5.1.3 Printing Strings
The common printing methods include:

print(): generic printing
noquote(): print with no quotes
cat(): concatenate and print with no quotes
sprintf(): a wrapper for the C function sprintf, that returns a character
vector containing a formatted combination of text and variable values
The primary printing function in R is print()
x <- "learning to print strings"
# basic printing
print(x)
## [1] "learning to print strings"
# print without quotes

print(x, quote = FALSE)
## [1] learning to print strings
An alternative to printing a string without quotes is to use noquote()

noquote(x)
## [1] learning to print strings
Another very useful function is cat() which allows us to concatenate objects

and print them either on screen or to a le. The output result is very similar to
noquote(); however, cat() does not print the numeric line indicator. As a
result, cat() can be useful for printing nicely formatted responses to users.
# basic printing (similar to noquote)
cat(x)
## learning to print strings
# combining character strings

cat(x, "in R")
## learning to print strings in R
# basic printing of alphabet

cat(letters)
## a b c d e f g h i j k l m n o p q r s t u v w x y z
# specify a separator between the combined characters

cat(letters, sep = "-")
## a-b-c-d-e-f-g-h-i-j-k-l-m-n-o-p-q-r-s-t-u-v-w-x-y-z
# collapse the space between the combine characters

cat(letters, sep = "")
## abcdefghijklmnopqrstuvwxyz
You can also format the line width for printing long strings using the fill argument:
x <- "Today I am learning how to print strings."
y <- "Tomorrow I plan to learn about textual analysis."
z <- "The day after I will take a break and drink a beer."
cat(x, y, z, fill = 0)
## Today I am learning how to print strings. Tomorrow I plan to

learn about textual analysis. The day after I will take a break and
drink a beer.
cat(x, y, z, fill = 5)
## Today I am learning how to print strings.

## Tomorrow I plan to learn about textual analysis.
## The day after I will take a break and drink a beer.
sprintf() is a useful printing function for precise control of the output. It is

a wrapper for the C function sprintf and returns a character vector containing a
formatted combination of text and variable values.To substitute in a string or string
variable, use %s:
x <- "print strings"
# substitute a single string/variable

sprintf("Learning to %s in R", x)
## [1] "Learning to print strings in R"
# substitute multiple strings/variables

y <- "in R"
sprintf("Learning to %s %s", x, y)
## [1] "Learning to print strings in R"
For integers, use %d or a variant:
version <- 3
# substitute integer
sprintf("This is R version:%d", version)
## [1] "This is R version:3"
# print with leading spaces

sprintf("This is R version:%4d", version)
## [1] "This is R version: 3"

5.1 Character String Basics 45
# can also lead with zeros

sprintf("This is R version:%04d", version)
## [1] "This is R version:0003"
For oating-point numbers, use %f for standard notation, and %e or %E for expo-
nential notation:
sprintf("%f", pi) # '%f' indicates 'fixed point' decimal notation
## [1] "3.141593"
sprintf("%.3f", pi) # decimal notation with 3 decimal digits
## [1] "3.142"
sprintf("%1.0f", pi) # 1 integer and 0 decimal digits
## [1] "3"
sprintf("%5.1f", pi) # decimal notation with 5 total decimal digits and
## [1] " 3.1" # only 1 to the right of the decimal point
sprintf("%05.1f", pi) # same as above but fill empty digits with zeros
## [1] "003.1"
sprintf("%+f", pi) # print with sign (positive)
## [1] "+3.141593"
sprintf("% f", pi) # prefix a space
## [1] " 3.141593"
sprintf("%e", pi) # exponential decimal notation 'e'
## [1] "3.141593e+00"
sprintf("%E", pi) # exponential decimal notation 'E'
## [1] "3.141593E+00"
5.1.4 Counting String Elements and Characters
To count the number of elements in a string use length():

length("How many elements are in this string?")
## [1] 1
length(c("How", "many", "elements", "are", "in", "this", "string?"))
## [1] 7
To count the number of characters in a string use nchar():

nchar("How many characters are in this string?")
## [1] 39
nchar(c("How", "many", "characters", "are", "in", "this", "string?"))
## [1] 3 4 10 3 2 4 7
5.2 String Manipulation with Base R
Basic string manipulation typically includes case conversion, simple character and
substring replacement, adding/removing whitespace, and performing set operations to
compare similarities and differences between two character vectors. These operations
can all be performed with base R functions; however, some operations (or at least
their syntax) are simplied with the stringr package which we will discuss in the
next section. This section illustrates the base R string manipulation capabilities.
5.2.1 Case Conversion
To convert all upper case characters to lower case use tolower():

x <- "Learning To MANIPULATE strinGS in R"
tolower(x)
## [1] "learning to manipulate strings in r"
To convert all lower case characters to upper case use toupper():

toupper(x)
## [1] "LEARNING TO MANIPULATE STRINGS IN R"
5.2.2 Simple Character Replacement
To replace a character (or multiple characters) in a string you can use chartr():
# replace 'A' with 'a'
5.2 String Manipulation with Base R 47
x <- "This is A string."

chartr(old = "A", new = "a", x)
## [1] "This is a string."
# multiple character replacements

# replace any 'd' with 't' and any 'z' with 'a'
y <- "Tomorrow I plzn do lezrn zbout dexduzl znzlysis."
chartr(old = "dz", new = "ta", y)
## [1] "Tomorrow I plan to learn about textual analysis."
Note that chartr() replaces every identied letter for replacement so the only
time I use it is when I am certain that I want to change every possible occurrence of
a letter.
5.2.3 String Abbreviations
To abbreviate strings you can use abbreviate():

streets <- c("Main", "Elm", "Riverbend", "Mario", "Frederick")
# default abbreviations
abbreviate(streets)
## Main Elm Riverbend Mario Frederick

## "Main" "Elm" "Rvrb" "Mari" "Frdr"
# set minimum length of abbreviation

abbreviate(streets, minlength = 2)
## Main Elm Riverbend Mario Frederick

## "Mn" "El" "Rv" "Mr" "Fr"
Note that if you are working with U.S. states, R already has a pre-built vector
with state names (state.name). Also, there is a pre-built vector of abbreviated
state names (state.abb).
5.2.4 Extract/Replace Substrings
To extract or replace substrings in a character vector there are three primary base R
functions to use: substr(), substring(), and strsplit(). The purpose of
substr() is to extract and replace substrings with specied starting and stopping
characters:
alphabet <- paste(LETTERS, collapse = "")
# extract 18th character in string

substr(alphabet, start = 18, stop = 18)
## [1] "R"
# extract 18-24th characters in string

substr(alphabet, start = 18, stop = 24)
## [1] "RSTUVWX"
# replace 19-24th characters with `R`

substr(alphabet, start = 19, stop = 24) <- "RRRRRR"
alphabet
## [1] "ABCDEFGHIJKLMNOPQRRRRRRRYZ"
The purpose of substring() is to extract and replace substrings with only a

specied starting point. substring() also allows you to extract/replace in a
recursive fashion:
alphabet <- paste(LETTERS, collapse = "")
# extract 18th through last character

substring(alphabet, first = 18)
## [1] "RSTUVWXYZ"
# recursive extraction; specify start position only

substring(alphabet, first = 18:24)
## [1] "RSTUVWXYZ" "STUVWXYZ" "TUVWXYZ" "UVWXYZ" "VWXYZ" "WXYZ"

## [7] "XYZ"
# recursive extraction; specify start and stop positions

substring(alphabet, first = 1:5, last = 3:7)
## [1] "ABC" "BCD" "CDE" "DEF" "EFG"
To split the elements of a character string use strsplit():

strsplit(z, split = " ")
## [[1]]
## [1] "The" "day" "after" "I" "will" "take" "a" "break"
## [9] "and" "drink" "a" "beer."
a <- "Alabama-Alaska-Arizona-Arkansas-California"
strsplit(a, split = "-")
## [[1]]
## [1] "Alabama" "Alaska" "Arizona" "Arkansas" "California"
5.3 String Manipulation with stringr 49
Note that the output of strsplit() is a list. To convert the output to a simple
atomic vector simply wrap in unlist():
unlist(strsplit(a, split = "-"))
5.3 String Manipulation with stringr
The stringr package was developed by Hadley Wickham to act as simple wrap-
pers that make Rs string functions more consistent, simple, and easier to use. To
replicate the functions in this section you will need to install and load the stringr
package:
# install stringr package
install.packages("stringr")
# load package
library(stringr)
5.3.1 Basic Operations
There are three stringr functions that are closely related to their base R equivalents,
but with a few enhancements:
Concatenate with str_c()
Number of characters with str_length()
Substring with str_sub()
str_c() is equivalent to the paste() functions:
# same as paste0()
str_c("Learning", "to", "use", "the", "stringr", "package")
## [1] "Learningtousethestringrpackage"
# same as paste()
str_c("Learning", "to", "use", "the", "stringr", "package", sep = " ")
## [1] "Learning to use the stringr package"
# allows recycling
str_c(letters, " is for", "")
## [1] "a is for" "b is for" "c is for" "d is for" "e is for"
## [6] "f is for" "g is for" "h is for" "i is for" "j is for"
## [11] "k is for" "l is for" "m is for" "n is for" "o is for"
## [16] "p is for" "q is for" "r is for" "s is for" "t is for"
## [21] "u is for" "v is for" "w is for" "x is for" "y is for"
## [26] "z is for"
str_length() is similar to the nchar() function; however, str_

length() behaves more appropriately with missing (NA) values:
# some text with NA

text = c("Learning", "to", NA, "use", "the", NA, "stringr", "package")
# compare `str_length()` with `nchar()`

nchar(text)
## [1] 8 2 2 3 3 2 7 7
str_length(text)
## [1] 8 2 NA 3 3 NA 7 7
str_sub() is similar to substr(); however, it returns a zero length vector

if any of its inputs are zero length, and otherwise expands each argument to match
the longest. It also accepts negative positions, which are calculated from the left of
the last character.
x <- "Learning to use the stringr package"
# alternative indexing
str_sub(x, start = 1, end = 15)
## [1] "Learning to use"
str_sub(x, end = 15)
str_sub(x, start = 17)
## [1] "the stringr package"
str_sub(x, start = c(1, 17), end = c(15, 35))
## [1] "Learning to use" "the stringr package"
# using negative indices for start/end points from end of string

str_sub(x, start = -1)
## [1] "e"
str_sub(x, start = -19)
## [1] "the stringr package"
str_sub(x, end = -21)

5.3 String Manipulation with stringr 51
# Replacement
str_sub(x, end = 15) <- "I know how to use"
x
## [1] "I know how to use the stringr package"
5.3.2 Duplicate Characters Within a String
A new functionality that stringr provides in which base R does not have a specic
function for is character duplication:
str_dup("beer", times = 3)
## [1] "beerbeerbeer"
str_dup("beer", times = 1:3)
## [1] "beer" "beerbeer" "beerbeerbeer"
# use with a vector of strings

states_i_luv <- state.name[c(6, 23, 34, 35)]
str_dup(states_i_luv, times = 2)
## [1] "ColoradoColorado" "MinnesotaMinnesota"

## [3] "North DakotaNorth Dakota" "OhioOhio"
5.3.3 Remove Leading and Trailing Whitespace
A common task of string processing is that of parsing text into individual words.
Often, this results in words having blank spaces (whitespaces) on either end of the
word. The str_trim() can be used to remove these spaces:
text <- c("Text ", " with", " whitespace ", " on", "both ", " sides ")
# remove whitespaces on the left side

str_trim(text, side = "left")
## [1] "Text " "with" "whitespace " "on" "both "

## [6] "sides "
# remove whitespaces on the right side

str_trim(text, side = "right")
## [1] "Text" " with" " whitespace" " on" "both"

## [6] " sides"
# remove whitespaces on both sides

str_trim(text, side = "both")
## [1] "Text" "with" "whitespace" "on" "both"

## [6] "sides"
5.3.4 Pad a String with Whitespace
To add whitespace, or to pad a string, use str_pad(). You can also use str_
pad() to pad a string with specied characters.
str_pad("beer", width = 10, side = "left")
## [1] " beer"
str_pad("beer", width = 10, side = "both")
## [1] " beer "
str_pad("beer", width = 10, side = "right", pad = "!")
## [1] "beer!!!!!!"
5.4 Set Operatons for Character Strings
There are also base R functions that allow for assessing the set union, intersection,
difference, equality, and membership of two vectors.
5.4.1 Set Union
To obtain the elements of the union between two character vectors use union():
set_1 <- c("lagunitas", "bells", "dogfish", "summit", "odell")
set_2 <- c("sierra", "bells", "harpoon", "lagunitas", "founders")
union(set_1, set_2)
## [1] "lagunitas" "bells" "dogfish" "summit" "odell" "sierra"

## [7] "harpoon" "founders"
5.4.2 Set Intersection
To obtain the common elements of two character vectors use intersect():
intersect(set_1, set_2)
## [1] "lagunitas" "bells"
5.4 Set Operatons for Character Strings 53
5.4.3 Identifying Different Elements
To obtain the non-common elements, or the difference, of two character vectors use
setdiff():
# returns elements in set_1 not in set_2
setdiff(set_1, set_2)
## [1] "dogfish" "summit" "odell"
# returns elements in set_2 not in set_1

setdiff(set_2, set_1)
## [1] "sierra" "harpoon" "founders"
5.4.4 Testing for Element Equality
To test if two vectors contain the same elements regardless of order use
setequal():
set_3 <- c("woody", "buzz", "rex")
set_4 <- c("woody", "andy", "buzz")
set_5 <- c("andy", "buzz", "woody")
setequal(set_3, set_4)
## [1] FALSE
setequal(set_4, set_5)
## [1] TRUE
5.4.5 Testing for Exact Equality
To test if two character vectors are equal in content and order use
identical():

set_7 <- c("andy", "buzz", "woody")
identical(set_6, set_7)
## [1] FALSE
identical(set_6, set_8)
## [1] TRUE
5.4.6 Identifying If Elements Are Contained in a String
To test if an element is contained within a character vector use is.element() or

%in%:
good <- "andy"
bad <- "sid"
is.element(good, set_8)
## [1] TRUE
good %in% set_8
## [1] TRUE
bad %in% set_8
## [1] FALSE
5.4.7 Sorting a String
To sort a character vector use sort():

sort(set_8)
## [1] "andy" "buzz" "woody"
sort(set_8, decreasing = TRUE)
## [1] "woody" "buzz" "andy"
Chapter 6
Dealing with Regular Expressions
A regular expression (aka regex) is a sequence of characters that dene a search

pattern, mainly for use in pattern matching with text strings. Typically, regex pat-
terns consist of a combination of alphanumeric characters as well as special charac-
ters. The pattern can also be as simple as a single character or it can be more complex
and include several characters.
To understand how to work with regular expressions in R, we need to consider
two primary features of regular expressions. One has to do with the syntax, or the
way regex patterns are expressed in R. The other has to do with the functions used
for regex matching in R. In this chapter, we will cover both of these aspects. First,
I cover the syntax that allows you to perform pattern matching functions with meta
characters, character and POSIX classes, and quantiers. This will provide you with
the basic understanding of the syntax required to establish the pattern to nd. Then
I cover the functions you can apply to identify, extract, replace, and split parts of
character strings based on the regex pattern specied.
6.1 Regex Syntax
At rst glance (and second, third,) the regex syntax can appear quite confusing.
This section will provide you with the basic foundation of regex syntax; however,
realize that there is a plethora of resources available that will give you far more
detailed, and advanced, knowledge of regex syntax. To read more about the speci-
cations and technicalities of regex in R you can nd help at help(regex) or
help(regexp).

56 6 Dealing with Regular Expressions
6.1.1 Metacharacters
Metacharacters consist of non-alphanumeric symbols such as:

.\ |()[{$*+?
To match metacharacters in R you need to escape them with a double backslash
\\. The following displays the general escape syntax for the most common
metacharacters (Fig. 6.1):
Fig. 6.1 Escape syntax for Metacharacter Literal Meaning Escape Syntax
common metacharacters . period or dot \\.
$ dollar sign \\$
* asterisk \\*
+ plus sign \\+
? question mark \\?
| vertical bar \\|
\\ double backslash \\\\
^ caret \\^
[ square bracket \\[
{ curly brace \\{
( parenthesis \\(
*adapted from Handling and Processing Strings in R (Sanchez, 2013)
The following provides examples to show how to use the escape syntax to nd
and replace metacharacters. For information on the sub and gsub functions used
in this example visit the main regex functions page.
# substitute $ with !
sub(pattern = "\\$", "\\!", "I love R$")
## [1] "I love R!"
# substitute ^ with carrot

sub(pattern = "\\^", "carrot", "My daughter has a ^ with almost every meal!")
## [1] "My daughter has a carrot with almost every meal!"
# substitute \\ with whitespace

gsub(pattern = "\\\\", " ", "I\\need\\space")
## [1] "I need space"
6.1.2 Sequences
To match a sequence of characters we can apply short-hand notation which captures

the fundamental types of sequences. The following displays the general syntax for
these common sequences (Fig. 6.2):
6.1 Regex Syntax 57
Fig. 6.2 Anchors for Anchor Description

common sequences \\d match a digit character
\\D match a non-digit character
\\s match a space character
\\S match a non-space character
\\w match a word
\\W match a non-word
\\b match a word boundary
\\B match a non-word boundary
\\h match a horizontal space
\\H match a non-horizontal space
\\v match a vertical space
\\V match a non-vertical space
The following provides examples to show how to use the anchor syntax to nd
and replace sequences. For information on the gsub function used in this example
visit the main regex functions page.
# substitute any digit with an underscore
gsub(pattern = "\\d", "_", "I'm working in RStudio v.0.99.484")
## [1] "I'm working in RStudio v._.__.___"
# substitute any non-digit with an underscore

gsub(pattern = "\\D", "_", "I'm working in RStudio v.0.99.484")
## [1] "_________________________0_99_484"
# substitute any whitespace with underscore

gsub(pattern = "\\s", "_", "I'm working in RStudio v.0.99.484")
## [1] "I'm_working_in_RStudio_v.0.99.484"
# substitute any wording with underscore

gsub(pattern = "\\w", "_", "I'm working in RStudio v.0.99.484")
## [1] "_'_ _______ __ _______ _._.__.___"
6.1.3 Character Classes
To match one of several characters in a specied set we can enclose the characters
of concern with square brackets [ ]. In addition, to match any characters not in a
specied character set we can include the caret ^ at the beginning of the set within
the brackets. The following displays the general syntax for common character
classes but these can be altered easily as shown in the examples that follow
(Fig. 6.3):
Anchor Description
[aeiou] match any specified lower case vowel
[AEIOU] match any specified upper case vowel
[0123456789] match any specified numeric value
[0-9] match any range of specified numeric values
[a-z] match any range of lower case letter
[A-Z] match any range of upper case letter
[a-zA-Z0-9] match any of the above
[^aeiou] match anything other than a lowercase vowel
[^0-9] match anything other than the specified numeric values
Fig. 6.3 Anchors for common character classes
The following provides examples to show how to use the anchor syntax to match
character classes. For information on the grep function used in this example visit
the main regex functions page.
x <- c("RStudio", "v.0.99.484", "2015", "09-22-2015", "grep vs. grepl")
# find any strings with numeric values between 0-9

grep(pattern = "[0-9]", x, value = TRUE)
## [1] "v.0.99.484" "2015" "09-22-2015"
# find any strings with numeric values between 6-9

grep(pattern = "[6-9]", x, value = TRUE)
## [1] "v.0.99.484" "09-22-2015"
# find any strings with the character R or r

grep(pattern = "[Rr]", x, value = TRUE)
## [1] "RStudio" "grep vs. grepl"
# find any strings that have non-alphanumeric characters

grep(pattern = "[^0-9a-zA-Z]", x, value = TRUE)
## [1] "v.0.99.484" "09-22-2015" "grep vs. grepl"
6.1.4 POSIX Character Classes
Closely related to regex character classes are POSIX character classes which are
expressed in double brackets [[ ]] (Fig. 6.4).
The following provides examples to show how to use the anchor syntax to match
POSIX character classes. For information on the grep function used in this exam-
ple visit the main regex functions page.
x <- "I like beer! #beer, @wheres_my_beer, I like R (v3.2.2) #rrrrrrr2015"
# remove space or tabs

gsub(pattern = "[[:blank:]]", replacement = "", x)
## [1] "Ilikebeer!#beer,@wheres_my_beer,IlikeR(v3.2.2)#rrrrrrr2015"
6.1 Regex Syntax 59
Anchor Description
[[:lower:]] lower-case letters
[[:upper:]] upper-case letters
[[:alpha:]] alphabetic characters [[:lower:]] + [[:upper:]]
[[:digit:]] numeric values
[[:alnum:]] alphanumeric characters [[:alpha:]] + [[:digit:]]
[[:blank:]] blank characters (space & tab)
[[:cntrl:]] control characters
[[:punct:]] punctuation characters: ! " # % & ' ( ) * + , - . / : ;
[[:space:]] space characters: tab, newline, vertical tab, space, etc
[[:xdigit:]] hexadecimal digits: 0-9 A B C D E F a b c d e f
[[:print:]] printable characters [[:alpha:]] + [[:punct:]] + space
[[:graph:]] graphical characters [[:alpha:]] + [[:punct:]]
Fig. 6.4 Anchors for POSIX character classes
Quantifier Description
? the preceding item is optional and will be matched at most once
* the preceding item will be matched zero or more times
+ the preceding item will be matched one or more times
{n} the preceding item is matched exactly n times
{n,} the preceding item is matched n or more times
{n,m} the preceding item is matched at least n times, but not more than m times
Fig. 6.5 Quantiers
# replace punctuation with whitespace

gsub(pattern = "[[:punct:]]", replacement = " ", x)
## [1] "I like beer beer wheres my beer I like R v3 2 2 rrrrrrr2015"
# remove alphanumeric characters

gsub(pattern = "[[:alnum:]]", replacement = "", x)
## [1] " ! #, @__, (..) #"
6.1.5 Quantifiers
When we want to match a certain number of characters that meet a certain criteria we
can apply quantiers to our pattern searches. The quantiers we can use are (Fig. 6.5):
The following provides examples to show how to use the quantier syntax
to match a certain number of characters patterns. For information on the grep
function used in this example visit the main regex functions page. Note that state.
name is a built in dataset within R that contains all the U.S. state names.
# match states that contain z

grep(pattern = "z+", state.name, value = TRUE)
## [1] "Arizona"
# match states with two s

grep(pattern = "s{2}", state.name, value = TRUE)
## [1] "Massachusetts" "Mississippi" "Missouri" "Tennessee"
# match states with one or two s

grep(pattern = "s{1,2}", state.name, value = TRUE)
## [1] "Alaska" "Arkansas" "Illinois" "Kansas"
## [5] "Louisiana" "Massachusetts" "Minnesota" "Mississippi"
## [9] "Missouri" "Nebraska" "New Hampshire" "New Jersey"
## [13] "Pennsylvania" "Rhode Island" "Tennessee" "Texas"
## [17] "Washington" "West Virginia" "Wisconsin"
6.2 Regex Functions
Now that Ive illustrated how R handles some of the most common regular expres-
sion elements, its time to present the functions you can use for working with regu-
lar expression. R contains a set of functions in the base package that we can use to
nd pattern matches. Alternatively, the R package stringr also provides several
functions for regex operations. We will cover both these alternatives.
6.2.1 Main Regex Functions in R
The primary base R regex functions serve three primary purposes: pattern matching,
pattern replacement, and character splitting.
6.2.1.1 Pattern Matching
There are ve functions that provide pattern matching capabilities. The three func-
tions that I provide examples for (grep(), grepl(), and regexpr()) are ones
that are most common. The primary difference between these three functions is the
output they provide. The two other functions which I do not illustrate are greg-
expr() and regexec(). These two functions provide similar capabilities as
regexpr() but with the output in list form.
To nd a pattern in a character vector and to have the element values or indices
as the output use grep():
6.2 Regex Functions 61
# use the built in data set state.division

head(as.character(state.division))
## [1] "East South Central" "Pacific" "Mountain"
## [4] "West South Central" "Pacific" "Mountain"
# find the elements which match the pattern

grep("North", state.division)
## [1] 13 14 15 16 22 23 25 27 34 35 41 49
# use value = TRUE to show the element value

grep("North", state.division, value = TRUE)
## [1] "East North Central" "East North Central" "West North Central"
## [4] "West North Central" "East North Central" "West North Central"
## [7] "West North Central" "West North Central" "West North Central"
## [10] "East North Central" "West North Central" "East North Central"
# can use the invert argument to show the non-matching elements

grep("North | South", state.division, invert = TRUE)
## [1] 2 3 5 6 7 8 9 10 11 12 19 20 21 26 28 29 30 31 32 33 37 38 39
## [24] 40 44 45 46 47 48 50
To nd a pattern in a character vector and to have logical (TRUE/FALSE) out-

puts use grepl():
grepl("North | South", state.division)
## [1] TRUE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
## [12] FALSE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE TRUE
## [23] TRUE TRUE TRUE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE
## [34] TRUE TRUE TRUE FALSE FALSE FALSE FALSE TRUE TRUE TRUE FALSE
## [45] FALSE FALSE FALSE FALSE TRUE FALSE
# wrap in sum() to get the count of matches

sum(grepl("North | South", state.division))
## [1] 20
To nd exactly where the pattern exists in a string use regexpr():

x <- c("v.111", "0v.11", "00v.1", "000v.", "00000")
regexpr("v.", x)
## [1] 1 2 3 4 -1
## attr(,"match.length")
## [1] 2 2 2 2 -1
## attr(,"useBytes")
## [1] TRUE
The output of regexpr() can be interpreted as follows. The rst element pro-
vides the starting position of the match in each element. Note that the value 1
means there is no match. The second element (attribute match length) provides
the length of the match. The third element (attribute useBytes) has a value TRUE
meaning matching was done byte-by-byte rather than character-by-character.
6.2.1.2 Pattern Replacement Functions
In addition to nding patterns in character vectors, its also common to want to

replace a pattern in a string with a new pattern. Base R regex functions provide two
options for this: (a) replace the rst matching occurrence or (b) replace all
occurrences.
To replace the first matching occurrence of a pattern use sub():
new <- c("New York", "new new York", "New New New York")
new
## [1] "New York" "new new York" "New New New York"
# Default is case sensitive

sub("New", replacement = "Old", new)
## [1] "Old York" "new new York" "Old New New York"
# use 'ignore.case = TRUE' to perform the obvious

sub("New", replacement = "Old", new, ignore.case = TRUE)
## [1] "Old York" "Old new York" "Old New New York"
To replace all matching occurrences of a pattern use gsub():

# Default is case sensitive
gsub("New", replacement = "Old", new)
## [1] "Old York" "new new York" "Old Old Old York"
# use ignore.case = TRUE to perform the obvious

gsub("New", replacement = "Old", new, ignore.case = TRUE)
## [1] "Old York" "Old Old York" "Old Old Old York"
6.2.1.3 Splitting Character Vectors
There will be times when you want to split the elements of a character string into
separate elements. To divide the characters in a vector into individual components
use strsplit():
x <- paste(state.name[1:10], collapse = " ")
# output will be a list

strsplit(x, " ")
## [[1]]
## [6] "Colorado" "Connecticut" "Delaware" "Florida" "Georgia"
# output as a vector rather than a list

unlist(strsplit(x, " "))
## [6] "Colorado" "Connecticut" "Delaware" "Florida" "Georgia"
6.2.2 Regex Functions in stringr
Similar to basic string manipulation, the stringr package also offers regex func-
tionality. In some cases the stringr performs the same functions as certain base
R functions but with more consistent syntax. In other cases stringr offers addi-
tional functionality that is not available in base R functions.
# install stringr package
install.packages("stringr")
# load package
library(stringr)
6.2.2.1 Detecting Patterns
To detect whether a pattern is present (or absent) in a string vector use the str_
detect(). This function is a wrapper for grepl().
# use the built in data set 'state.name'
head(state.name)
## [6] "Colorado"
str_detect(state.name, pattern = "New")

## [1] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
## [23] FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE FALSE
## [45] FALSE FALSE FALSE FALSE FALSE FALSE
# count the total matches by wrapping with sum

sum(str_detect(state.name, pattern = "New"))
## [1] 4
6.2.2.2 Locating Patterns
To locate the occurrences of patterns stringr offers two options: (a) locate the
rst matching occurrence or (b) locate all occurrences. To locate the position of the
rst occurrence of a pattern in a string vector use str_locate(). The output pro-
vides the starting and ending position of the rst match found within each element.
x <- c("abcd", "a22bc1d", "ab3453cd46", "a1bc44d")
# locate 1st sequence of 1 or more consecutive numbers

str_locate(x, "[0-9]+")
## start end
## [1,] NA NA
## [2,] 2 3
## [3,] 3 6
## [4,] 2 2
To locate the positions of all pattern match occurrences in a character vector use
str_locate_all(). The output provides a list the same length as the number
of elements in the vector. Each list item will provide the starting and ending positions
for each pattern match occurrence in its respective element.
# locate all sequences of 1 or more consecutive numbers
str_locate_all(x, "[0-9]+")
## [[1]]
## start end
##
## [[2]]
## start end
## [1,] 2 3
## [2,] 6 6
##
## [[3]]
## start end
## [1,] 3 6
## [2,] 9 10
##
## [[4]]
## start end
## [1,] 2 2
## [2,] 5 6
6.2.2.3 Extracting Patterns
For extracting a string containing a pattern, stringr offers two primary options:
(a) extract the rst matching occurrence or (b) extract all occurrences. To extract the
rst occurrence of a pattern in a character vector use str_extract(). The out-
put will be the same length as the string and if no match is found the output will be
NA for that element.
y <- c("I use R #useR2014", "I use R and love R #useR2015", "Beer")
str_extract(y, pattern = "R")

## [1] "R" "R" NA
To extract all occurrences of a pattern in a character vector use str_extract_

all(). The output provides a list the same length as the number of elements in the
vector. Each list item will provide the matching pattern occurrence within that rela-
tive vector element.
str_extract_all(y, pattern = "[[:punct:]]*[a-zA-Z0-9]*R[a-zA-Z0-9]*")
## [[1]]
## [1] "R" "#useR2014"
##
## [[2]]
## [1] "R" "R" "#useR2015"
##
## [[3]]
## character(0)
6.2.2.4 Replacing Patterns
For extracting a string containing a pattern, stringr offers two options: (a)
replace the rst matching occurrence or (b) replace all occurrences. To replace the
rst occurrence of a pattern in a character vector use str_replace(). This func-
tion is a wrapper for sub().
cities <- c("New York", "new new York", "New New New York")
cities
## [1] "New York" "new new York" "New New New York"
# case sensitive
str_replace(cities, pattern = "New", replacement = "Old")
## [1] "Old York" "new new York" "Old New New York"
# to deal with case sensitivities use Regex syntax in the 'pattern' argument
str_replace(cities, pattern = "[N]*[n]*ew", replacement = "Old")
## [1] "Old York" "Old new York" "Old New New York"
To extract all occurrences of a pattern in a character vector use str_replace_

all(). This function is a wrapper for gsub().
str_replace_all(cities, pattern = "[N]*[n]*ew", replacement = "Old")
## [1] "Old York" "Old Old York" "Old Old Old York"
6.2.2.5 String Splitting
To split the elements of a character string use str_split(). This function is a

wrapper for strsplit().
str_split(z, pattern = " ")
## [[1]]
## [1] "The" "day" "after" "I" "will" "take" "a" "break"
## [9] "and" "drink" "a" "beer."
a <- "Alabama-Alaska-Arizona-Arkansas-California"
str_split(a, pattern = "-")
## [[1]]
Note that the output of strs_plit() is a list. To convert the output to a simple
atomic vector simply wrap in unlist():
unlist(str_split(a, pattern = "-"))
6.3 Additional Resources
Character strings are often considered semi-structured data. Text can be structured
in a specied eld; however, the quality and consistency of the text input can be far
from structured. Consequently, managing and manipulating character strings can be
extremely tedious and unique to each data wrangling process. As a result, taking the
time to learn the nuances of dealing with character strings and regex functions can
provide a great return on investment; however, the functions and techniques required
will likely be greater than what I could offer here. So here are additional resources
that are worth reading and learning from:
Handling and Processing Strings in R1
stringr Package Vignette2
Regular Expressions3
1
http://gastonsanchez.com/Handling_and_Processing_Strings_in_R.pdf
2
https://cran.r-project.org/web/packages/stringr/vignettes/stringr.html
3
http://www.regular-expressions.info/
Chapter 7
Dealing with Factors
Factors are variables in R, which take on a limited number of different values; such
variables are often referred to as categorical variables. One of the most important
uses of factors is in statistical modeling; since categorical variables enter into statis-
tical models such as lm and glm differently than continuous variables, storing data
as factors insures that the modeling functions will treat such data correctly.
One can think of a factor as an integer vector where each integer has a label.1 In
fact, factors are built on top of integer vectors using two attributes: the class()
factor, which makes them behave differently from regular integer vectors, and the
levels(), which defines the set of allowed values.2
In this chapter I will cover the basics of dealing with factors, which includes
Creating, converting and inspecting factors, Ordering levels, Revaluing levels, and
Dropping levels.
7.1 Creating, Converting and Inspecting Factors
Factor objects can be created with the factor() function:

# create a factor string
gender <- factor(c("male", "female", "female", "male", "female"))
gender
## [1] male female female male female
## Levels: female male
# inspect to see if it is a factor class

class(gender)
## [1] "factor"
1
https://leanpub.com/rprogramming
2
http://adv-r.had.co.nz/Data-structures.html

68 7 Dealing with Factors
# show that factors are just built on top of integers

typeof(gender)
## [1] "integer"
# See the underlying representation of factor

unclass(gender)
## [1] 2 1 1 2 1
## attr(,"levels")
## [1] "female" "male"
# what are the factor levels?

levels(gender)
## [1] "female" "male"
# show summary of counts

summary(gender)
## female male
## 3 2
If we have a vector of character strings or integers we can easily convert to

factors:
group <- c("Group1", "Group2", "Group2", "Group1", "Group1")
str(group)
## chr [1:5] "Group1" "Group2" "Group2" "Group1" "Group1"
# convert from characters to factors

as.factor(group)
## [1] Group1 Group2 Group2 Group1 Group1
## Levels: Group1 Group2
7.2 Ordering Levels
When creating a factor we can control the ordering of the levels by using the lev-
els argument:
# when not specified the default puts order as alphabetical
gender <- factor(c("male", "female", "female", "male", "female"))
gender
## Levels: female male
# specifying order
gender <- factor(c("male", "female", "female", "male", "female"),
levels = c("male", "female"))
gender
## Levels: male female
7.4 Dropping Levels 69
We can also create ordinal factors in which a specific order is desired by using
the ordered = TRUE argument. This will be reflected in the output of the levels
as shown below in which low < middle < high:
ses <- c("low", "middle", "low", "low", "low", "low", "middle", "low", "middle",
"middle", "middle", "middle", "middle", "high", "high", "low", "middle",
"middle", "low", "high")
# create ordinal levels

ses <- factor(ses, levels = c("low", "middle", "high"), ordered = TRUE)
ses
## [1] low middle low low low low middle low middle middle
## [11] middle middle middle high high low middle middle low high
## Levels: low < middle < high
# you can also reverse the order of levels if desired

factor(ses, levels = rev(levels(ses)))
## [1] low middle low low low low middle low middle middle
## [11] middle middle middle high high low middle middle low high
## Levels: high < middle < low
7.3 Revalue Levels
To recode factor levels I usually use the revalue() function from the plyr
package.
plyr::revalue(ses, c("low" = "small", "middle" = "medium", "high" = "large"))
## [1] small medium small small small small medium small medium medium
## [11] medium medium medium large large small medium medium small large
## Levels: small < medium < large
Note that Using the :: notation allows you to access the revalue() function
without having to fully load the plyr package.
7.4 Dropping Levels
When you want to drop unused factor levels, use droplevels():

ses2 <- ses[ses != "middle"]
# lets say you have no observations in one level

summary(ses2)
## low middle high
## 8 0 3
# you can drop that level if desired

droplevels(ses2)
## [1] low low low low low low high high low low high
## Levels: low < high
Chapter 8
Dealing with Dates
Real world data are often associated with dates and time; however, dealing with
dates accurately can appear to be a complicated task due to the variety in formats
and accounting for time-zone differences and leap years. R has a range of functions
that allow you to work with dates and times. Furthermore, packages such as lub-
ridate make it easier to work with dates and times.
In this chapter I will introduce you to the basics of dealing with dates. This
includes printing the current date and time stamp, converting strings to dates,
extracting and manipulating parts of dates, creating date sequences, performing cal-
culations with dates, and dealing with time zone and daylight savings differences.
I end with offering additional resources to learn and deal with date and time data.
8.1 Getting Current Date and Time
To get current date and time information:

Sys.timezone()
## [1] "America/New_York"
Sys.Date()
## [1] "2015-09-24"
Sys.time()
## [1] "2015-09-24 15:08:57 EDT"
If using the lubridate package:

library(lubridate)
now()
## [1] "2015-09-24 15:08:57 EDT"

72 8 Dealing with Dates
8.2 Converting Strings to Dates
When date and time data are imported into R they will often default to a character
string. This requires us to convert strings to dates. We may also have multiple strings
that we want to merge to create a date variable.
8.2.1 Convert Strings to Dates
To convert a string that is already in a date format (YYYY-MM-DD) into a date

object use as.Date():
x <- c("2015-07-01", "2015-08-01", "2015-09-01")
as.Date(x)
## [1] "2015-07-01" "2015-08-01" "2015-09-01"
Note that the default date format is YYYY-MM-DD; therefore, if your string is
of different format you must incorporate the format argument. There are multiple
formats that dates can be in; for a complete list of formatting code options in R type
?strftime in your console.
y <- c("07/01/2015", "07/01/2015", "07/01/2015")
as.Date(y, format = "%m/%d/%Y")

## [1] "2015-07-01" "2015-07-01" "2015-07-01"
If using the lubridate package:

library(lubridate)
ymd(x)
## [1] "2015-07-01 UTC" "2015-08-01 UTC" "2015-09-01 UTC"
mdy(y)
## [1] "2015-07-01 UTC" "2015-07-01 UTC" "2015-07-01 UTC"
One of the many benefits of the lubricate package is that it automatically

recognizes the common separators used when recording dates (-, /, ., and ).
As a result, you only need to focus on specifying the order of the date elements to
determine the parsing function applied (Fig. 8.1):
Order of elements in date-time Parse function

year, month, day ymd()
year, day, month ydm()
month, day, year mdy()
day, month, year dmy()
hour, minute hm()
hour, minute, second hms()
year, month, day, hour, minute, second ymd_hms()
*adapted from Dates and Times Made Easy with lubridate (Grolemund & Wickham, 2011)
Fig. 8.1 Parsing functions for lubridate

8.3 Extract and Manipulate Parts of Dates 73
8.2.2 Create Dates by Merging Data
Sometimes your date data are collected in separate elements. To convert these sepa-
rate data into one date object incorporate the ISOdate() function:
yr <- c("2012", "2013", "2014", "2015")
mo <- c("1", "5", "7", "2")
day <- c("02", "22", "15", "28")
# ISOdate converts to a POSIXct object

ISOdate(year = yr, month = mo, day = day)
## [1] "2012-01-02 12:00:00 GMT" "2013-05-22 12:00:00 GMT"
## [3] "2014-07-15 12:00:00 GMT" "2015-02-28 12:00:00 GMT"
# truncate the unused time data by converting with as.Date

as.Date(ISOdate(year = yr, month = mo, day = day))
## [1] "2012-01-02" "2013-05-22" "2014-07-15" "2015-02-28"
Note that ISODate() also has arguments to accept data for hours, minutes,
seconds, and time-zone if you need to merge all these separate components.
8.3 Extract and Manipulate Parts of Dates
To extract and manipulate individual elements of a date I typically use the lub-
ridate package due to its simplistic function syntax. The functions provided by
lubridate to perform extraction and manipulation of dates include (Fig. 8.2):
Fig. 8.2 Accessor Date component Accessor

functions for lubridate Year year()
Month month()
Week week()
Day of year yday()
Day of month mday()
Day of week wday()
Hour hour()
Minute minute()
Second second()
Time zone tz()
*adapted from Dates and Times Made Easy with
lubridate (Grolemund & Wickham, 2011)
To extract an individual element of the date variable you simply use the accessor
function desired. Note that the accessor variables have additional arguments that
can be used to show the name of the date element in full or abbreviated form.
library(lubridate)
x <- c("2015-07-01", "2015-08-01", "2015-09-01")
year(x)
## [1] 2015 2015 2015
# default is numerical value

month(x)
## [1] 7 8 9
# show abbreviated name

month(x, label = TRUE)
## [1] Jul Aug Sep
## 12 Levels: Jan < Feb < Mar < Apr < May < Jun < Jul < Aug < Sep < < Dec
# show unabbreviated name

month(x, label = TRUE, abbr = FALSE)
## [1] July August September
## 12 Levels: January < February < March < April < May < June < < December
wday(x, label = TRUE, abbr = FALSE)

## [1] Wednesday Saturday Tuesday
## 7 Levels: Sunday < Monday < Tuesday < Wednesday < Thursday < < Saturday
To manipulate or change the values of date elements we simply use the accessor
function to extract the element of choice and then use the assignment function to
assign a new value.
# convert to date format

x <- ymd(x)
x
## [1] "2015-07-01 UTC" "2015-08-01 UTC" "2015-09-01 UTC"
# change the days for the dates

mday(x)
## [1] 1 1 1
mday(x) <- c(3, 10, 22)

x
## [1] "2015-07-03 UTC" "2015-08-10 UTC" "2015-09-22 UTC"
# can also use update() function

update(x, year = c(2013, 2014, 2015), month = 9)
## [1] "2013-09-03 UTC" "2014-09-10 UTC" "2015-09-22 UTC"
# can also add/subtract units

x + years(1) - days(c(2, 9, 21))
## [1] "2016-07-01 UTC" "2016-08-01 UTC" "2016-09-01 UTC"
8.4 Creating Date Sequences 75
8.4 Creating Date Sequences
To create a sequence of dates we can leverage the seq() function. As with numeric
vectors, you have to specify at least three of the four arguments (from, to, by, and
length.out).
seq(as.Date("2010-1-1"), as.Date("2015-1-1"), by = "years")
## [1] "2010-01-01" "2011-01-01" "2012-01-01" "2013-01-01" "2014-01-01"
## [6] "2015-01-01"
seq(as.Date("2015/1/1"), as.Date("2015/12/30"), by = "quarter")

## [1] "2015-01-01" "2015-04-01" "2015-07-01" "2015-10-01"
seq(as.Date('2015-09-15'), as.Date('2015-09-30'), by = "2 days")

## [1] "2015-09-15" "2015-09-17" "2015-09-19" "2015-09-21" "2015-09-23"
## [6] "2015-09-25" "2015-09-27" "2015-09-29"
Using the lubridate package is very similar. The only difference is lubridate
changes the way you specify the first two arguments in the seq() function.
library(lubridate)
seq(ymd("2010-1-1"), ymd("2015-1-1"), by = "years")

## [1] "2010-01-01 UTC" "2011-01-01 UTC" "2012-01-01 UTC" "2013-01-01 UTC"
## [5] "2014-01-01 UTC" "2015-01-01 UTC"
seq(ymd("2015/1/1"), ymd("2015/12/30"), by = "quarter")

## [1] "2015-01-01 UTC" "2015-04-01 UTC" "2015-07-01 UTC" "2015-10-01 UTC"
seq(ymd('2015-09-15'), ymd('2015-09-30'), by = "2 days")

## [1] "2015-09-15 UTC" "2015-09-17 UTC" "2015-09-19 UTC" "2015-09-21 UTC"
## [5] "2015-09-23 UTC" "2015-09-25 UTC" "2015-09-27 UTC" "2015-09-29 UTC"
Creating sequences with time is very similar; however, we need to make sure our
date object is POSIXct rather than just a Date object (as produced by as.Date):
seq(as.POSIXct("2015-1-1 0:00"), as.POSIXct("2015-1-1 12:00"), by = "hour")
## [1] "2015-01-01 00:00:00 EST" "2015-01-01 01:00:00 EST"
## [3] "2015-01-01 02:00:00 EST" "2015-01-01 03:00:00 EST"
## [5] "2015-01-01 04:00:00 EST" "2015-01-01 05:00:00 EST"
## [7] "2015-01-01 06:00:00 EST" "2015-01-01 07:00:00 EST"
## [9] "2015-01-01 08:00:00 EST" "2015-01-01 09:00:00 EST"
## [11] "2015-01-01 10:00:00 EST" "2015-01-01 11:00:00 EST"
## [13] "2015-01-01 12:00:00 EST"
# with lubridate
seq(ymd_hm("2015-1-1 0:00"), ymd_hm("2015-1-1 12:00"), by = "hour")
## [1] "2015-01-01 00:00:00 UTC" "2015-01-01 01:00:00 UTC"
## [3] "2015-01-01 02:00:00 UTC" "2015-01-01 03:00:00 UTC"
## [5] "2015-01-01 04:00:00 UTC" "2015-01-01 05:00:00 UTC"
## [7] "2015-01-01 06:00:00 UTC" "2015-01-01 07:00:00 UTC"
## [9] "2015-01-01 08:00:00 UTC" "2015-01-01 09:00:00 UTC"
## [11] "2015-01-01 10:00:00 UTC" "2015-01-01 11:00:00 UTC"
## [13] "2015-01-01 12:00:00 UTC"
8.5 Calculations with Dates
Since R stores date and time objects as numbers, this allows you to perform various
calculations such as logical comparisons, addition, subtraction, and working with
durations.
x <- Sys.Date()
x
## [1] "2015-09-26"
y <- as.Date("2015-09-11")
x > y
## [1] TRUE
x - y
## Time difference of 15 days
The nice thing about the date/time classes is that they keep track of leap years,
leap seconds, daylight savings, and time zones. Use OlsonNames() for a full list
of acceptable time zone specifications.
# last leap year
x <- as.Date("2012-03-1")
y <- as.Date("2012-02-28")
x - y
## Time difference of 2 days
# example with time zones

x <- as.POSIXct("2015-09-22 01:00:00", tz = "US/Eastern")
y <- as.POSIXct("2015-09-22 01:00:00", tz = "US/Pacific")
y == x
## [1] FALSE
y - x
## Time difference of 3 hours
Similarly, the same functionality exists with the lubridate package with the
only difference being the accessor function(s) used.
library(lubridate)
x <- now()
x
## [1] "2015-09-26 10:08:18 EDT"
y <- ymd("2015-09-11")
x > y
## [1] TRUE
8.6 Dealing with Time Zones and Daylight Savings 77
x - y
## Time difference of 15.5891 days
y + days(4)
## [1] "2015-09-15 UTC"
x - hours(4)
## [1] "2015-09-26 06:08:18 EDT"
We can also deal with time spans by using the duration functions in lubridate.
Durations simply measure the time span between start and end dates. Using base R
date functions for duration calculations is tedious and often results in wrong
measurements. lubridate provides simplistic syntax to calculate durations with
the desired measurement (seconds, minutes, hours, etc.).
# create new duration (represented in seconds)
new_duration(60)
## [1] "60s"
# create durations for minutes, hours, years

dminutes(1)
## [1] "60s"
dhours(1)
## [1] "3600 s (~1 hours)"
dyears(1)
## [1] "31536000 s (~365 days)"
# add/subtract durations from date/time object

x <- ymd_hms("2015-09-22 12:00:00")
x + dhours(10)
## [1] "2015-09-22 22:00:00 UTC"
x + dhours(10) + dminutes(33) + dseconds(54)

## [1] "2015-09-22 22:33:54 UTC"
8.6 Dealing with Time Zones and Daylight Savings
To change the time zone for a date/time we can use the with_tz() function which
will also update the clock time to align with the updated time zone:
library(lubridate)
time <- now()

time
## [1] "2015-09-26 10:30:32 EDT"
with_tz(time, tzone = "MST")

## [1] "2015-09-26 07:30:32 MST"
If the time zone is incorrect or for some reason you need to change the time zone
without changing the clock time you can force it with force_tz():
time
## [1] "2015-09-26 10:30:32 EDT"
force_tz(time, tzone = "MST")

## [1] "2015-09-26 10:30:32 MST"
We can also easily work with daylight savings times to eliminate impacts on
date/time calculations:
# most recent daylight savings time
ds <- ymd_hms("2015-03-08 01:59:59", tz = "US/Eastern")
# if we add a duration of 1 sec we gain an extra hour

ds + dseconds(1)
## [1] "2015-03-08 03:00:00 EDT"
# add a duration of 2 hours will reflect actual daylight savings clock time
# that occurred 2 hours after 01:59:59 on 2015-03-08
ds + dhours(2)
## [1] "2015-03-08 04:59:59 EDT"
# add a period of two hours will reflect clock time that normally occurs after
# 01:59:59 and is not influenced by daylight savings time.
ds + hours(2)
## [1] "2015-03-08 03:59:59 EDT"
For additional resources on learning and dealing with dates I recommend the
following:
Dates and times made easy with lubridate1
Date and time classes in R2
1
http://www.jstatsoft.org/article/view/v040i03
2
https://www.r-project.org/doc/Rnews/Rnews_2004-1.pdf
Part III
Managing Data Structures in R
Smart data structures and dumb code works a lot better than
the other way around
Eric S. Raymond
In the previous section I illustrated how to work with different types of data; however,
we primarily focused on data in a one-dimensional structure. In typical data analyses
you often need more than one dimension. Many datasets can contain variables of dif-
ferent length and or types of values (i.e. numeric vs character). Furthermore, many
statistical and mathematical calculations are based on matrices. R provides multiple
types of data structures to deal with these different needs.
The basic data structures in R can be organized by their dimensionality (1D,
2D, , nD) and their likeness (homogenous vs. heterogeneous). This results in
five data structure types most often used in data analysis; and almost all other
objects in R are built from these foundational types:
Basic Data Structures in R
Dimensions Homogenous Heterogeneous

1D Atomic Vector List
2D Matrix Data frame
nD Array
*adapted from Advanced R (H. Wickham 2014)
In this section I will cover the basics of these data structures. I have not had the need
to use multi-dimensional arrays, therefore, the topics I will go into details on will
include vectors, lists, matrices, and data frames. These types represent the most
commonly used data structures for day-to-day analyses. For each data structure I
will illustrate how to create the structure, add additional elements to a pre-existing
structure, add attributes to structures, and how to subset the various data structures.
Lastly, I will cover how to deal with missing values in data structures. Consequently,
this section will provide a robust understanding of managing various forms of data-
sets depending on dimensionality needs.
Chapter 9
Data Structure Basics
Prior to jumping into the data structures, its benecial to understand two components
of data structures - the structure and attributes.
9.1 Identifying the Structure
Given an object, the best way to understand what data structure it represents is to
use the structure function str(). str() stands for structure and provides a
compact display of the internal structure of an R object.
# different data structures
vector <- 1:10
list <- list(item1 = 1:10, item2 = LETTERS[1:18])
matrix <- matrix(1:12, nrow = 4)
df <- data.frame(item1 = 1:18, item2 = LETTERS[1:18])
# identify the structure of each object

str(vector)
## int [1:10] 1 2 3 4 5 6 7 8 9 10
str(list)
## List of 2
## $ item1: int [1:10] 1 2 3 4 5 6 7 8 9 10
## $ item2: chr [1:18] "A" "B" "C" "D"
str(matrix)
## int [1:4, 1:3] 1 2 3 4 5 6 7 8 9 10
str(df)
## 'data.frame': 18 obs. of 2 variables:
## $ item1: int 1 2 3 4 5 6 7 8 9 10
## $ item2: Factor w/ 18 levels "A","B","C","D",..: 1 2 3 4 5 6 7 8 9 10

82 9 Data Structure Basics
9.2 Attributes
R objects can have attributes, which are like metadata for the object. These meta-
data can be very useful in that they help to describe the object. For example, column
names on a data frame help to tell us what data are contained in each of the columns.
Some examples of R object attributes are:
names, dimnames
dimensions (e.g. matrices, arrays)
class (e.g. integer, numeric)
length
other user-dened attributes/metadata
Attributes of an object (if any) can be accessed using the attributes() func-
tion. Not all R objects contain attributes, in which case the attributes() func-
tion returns NULL.
# assess attributes of an object
attributes(df)
## $names
## [1] "item1" "item2"
##
## $row.names
## [1] 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
##
## $class
## [1] "data.frame"
attributes(matrix)
## $dim
## [1] 4 3
# assess names of an object

names(df)
## [1] "item1" "item2"
# assess the dimensions of an object

dim(matrix)
## [1] 4 3
# assess the class of an object

class(list)
## [1] "list"
# access the length of an object

length(vector)
## [1] 10
9.2 Attributes 83
# note that length will measure the number of items in

# a list or number of columns in a data frame
length(list)
## [1] 2
length(df)
## [1] 2
This chapter only shows you functions to assess these attributes. In the chapters
that follow more details are provided on how to view and create attributes for each
type of data structure.
Chapter 10
Managing Vectors
The basic structure in R is the vector. A vector is a sequence of data elements of the
same basic type: integer, double, logical, or character.1 The one-dimensional exam-
ples illustrated in the previous section are considered vectors. In this chapter I will
illustrate how to create vectors, add additional elements to pre-existing vectors, add
attributes to vectors, and subset vectors.
10.1 Creating Vectors
The colon : operator can be used to create a vector of integers between two speci-
fied numbers or the c() function can be used to create vectors of objects by concat-
enating elements together:
# integer vector
w <- 8:17
w
## [1] 8 9 10 11 12 13 14 15 16 17
# double vector
x <- c(0.5, 0.6, 0.2)
x
## [1] 0.5 0.6 0.2
# logical vector
y1 <- c(TRUE, FALSE, FALSE)
y1
## [1] TRUE FALSE FALSE
1
There are two additional vector types which I will not discusscomplex and raw.

86 10 Managing Vectors
# logical vector in shorthand

y2 <- c(T, F, F)
y2
## [1] TRUE FALSE FALSE
# Character vector
z <- c("a", "b", "c")
z
## [1] "a" "b" "c"
You can also use the as.vector() function to initialize vectors or change the
vector type:
v <- as.vector(8:17)
v
## [1] 8 9 10 11 12 13 14 15 16 17
# turn numerical vector to character

as.vector(v, mode = "character")
## [1] "8" "9" "10" "11" "12" "13" "14" "15" "16" "17"
All elements of a vector must be the same type, so when you attempt to combine
different types of elements they will be coerced to the most flexible type possible:
# numerics are turned to characters
str(c("a", "b", "c", 1, 2, 3))
## chr [1:6] "a" "b" "c" "1" "2" "3"
# logical are turned to numerics

str(c(1, 2, 3, TRUE, FALSE))
## num [1:5] 1 2 3 1 0
# or character
str(c("A", "B", "C", TRUE, FALSE))
## chr [1:5] "A" "B" "C" "TRUE" "FALSE"
10.2 Adding On To Vectors
To add additional elements to a pre-existing vector we can continue to leverage the

c() function. Also, note that vectors are always flat so nested c() functions will
not add additional dimensions to the vector:
v1 <- 8:17
c(v1, 18:22)
## [1] 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
# same as
c(v1, c(18, c(19, c(20, c(21:22)))))
## [1] 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
10.3 Adding Attributes to Vectors 87
10.3 Adding Attributes to Vectors
The attributes that you can add to vectors includes names and comments. If we
continue with our vector v1 we can see that the vector currently has no attributes:
attributes(v1)
## NULL
We can add names to vectors using two approaches. The first uses names() to
assign names to each element of the vector. The second approach is to assign names
when creating the vector.
# assigning names to a pre-existing vector
names(v1) <- letters[1:length(v1)]
v1
## a b c d e f g h i j
## 8 9 10 11 12 13 14 15 16 17
attributes(v1)
## $names
## [1] "a" "b" "c" "d" "e" "f" "g" "h" "i" "j"
# adding names when creating vectors

v2 <- c(name1 = 1, name2 = 2, name3 = 3)
v2
## name1 name2 name3
## 1 2 3
attributes(v2)
## $names
## [1] "name1" "name2" "name3"
We can also add comments to vectors to act as a note to the user. This does not change
how the vector behaves; rather, it simply acts as a form of metadata for the vector.
comment(v1) <- "This is a comment on a vector"
v1
## 8 9 10 11 12 13 14 15 16 17
attributes(v1)
## $names
## [1] "a" "b" "c" "d" "e" "f" "g" "h" "i" "j"
##
## $comment
## [1] "This is a comment on a vector"
10.4 Subsetting Vectors
The four main ways to subset a vector include combining square brackets [ ] with:
Positive integers
Negative integers
Logical values
Names
You can also subset with double brackets [[ ]] for simplifying subsets.
10.4.1 Subsetting with Positive Integers
Subsetting with positive integers returns the elements at the specified positions:
v1
## 8 9 10 11 12 13 14 15 16 17
v1[2]
## b
## 9
v1[2:4]
## b c d
## 9 10 11
v1[c(2, 4, 6, 8)]
## b d f h
## 9 11 13 15
# note that you can duplicate index positions

v1[c(2, 2, 4)]
## b b d
## 9 9 11
10.4.2 Subsetting with Negative Integers
Subsetting with negative integers will omit the elements at the specified positions:
v1[-1]
## b c d e f g h i j
## 9 10 11 12 13 14 15 16 17
v1[-c(2, 4, 6, 8)]
## a c e g i j
## 8 10 12 14 16 17
10.4 Subsetting Vectors 89
10.4.3 Subsetting with Logical Values
Subsetting with logical values will select the elements where the corresponding
logical value is TRUE:
v1[c(TRUE, FALSE, TRUE, FALSE, TRUE, TRUE, TRUE, FALSE, FALSE, TRUE)]
## a c e f g j
## 8 10 12 13 14 17
v1[v1 < 12]

## a b c d
## 8 9 10 11
v1[v1 < 12 | v1 > 15]

## a b c d i j
## 8 9 10 11 16 17
# if logical vector is shorter than the length of the vector being

# subsetted, it will be recycled to be the same length
v1[c(TRUE, FALSE)]
## a c e g i
## 8 10 12 14 16
10.4.4 Subsetting with Names
Subsetting with names will return the elements with the matching names specified:
v1["b"]
## b
## 9
v1[c("a", "c", "h")]

## a c h
## 8 10 15
10.4.5 Simplifying vs. Preserving
Its also important to understand the difference between simplifying and preserving
when subsetting. Simplifying subsets returns the simplest possible data structure
that can represent the output. Preserving subsets keeps the structure of the output
the same as the input.
For vectors, subsetting with single brackets [ ] preserves while subsetting with
double brackets [[ ]] simplifies. The change you will notice when simplifying
vectors is the removal of names.
v1[1]
## a
## 8
v1[[1]]
## [1] 8
Chapter 11
Managing Lists
A list is an R structure that allows you to combine elements of different types and
lengths. This can include a list embedded within a list. Many statistical outputs are
provided as a list as well; therefore, its critical to understand how to work with lists.
In this chapter I will illustrate how to create lists, add additional elements to pre-
existing lists, add attributes to lists, and subset lists.
11.1 Creating Lists
To create a list we can use the list() function. Note how each of the four list items
below are of different classes (integer, character, logical, and numeric) and different
lengths.
l <- list(1:3, "a", c(TRUE, FALSE, TRUE), c(2.5, 4.2))
str(l)
## List of 4
## $ : int [1:3] 1 2 3
## $ : chr "a"
## $ : logi [1:3] TRUE FALSE TRUE
## $ : num [1:2] 2.5 4.2
# a list containing a list

l <- list(1:3, list(letters[1:5], c(TRUE, FALSE, TRUE)))
str(l)
## List of 2
## $ : int [1:3] 1 2 3
## $ :List of 2
## ..$ : chr [1:5] "a" "b" "c" "d"
## ..$ : logi [1:3] TRUE FALSE TRUE

92 11 Managing Lists
11.2 Adding On To Lists
To add additional list components to a list we can leverage the list() and
append() functions. We can illustrate with the following list.
l1 <- list(1:3, "a", c(TRUE, FALSE, TRUE))
str(l1)
## List of 3
## $ : int [1:3] 1 2 3
## $ : chr "a"
If we add the new elements with list() it will create a list of two components,
component 1 will be a nested list of the original list and component 2 will be the
new elements added:
l2 <- list(l1, c(2.5, 4.2))
str(l2)
## List of 2
## $ :List of 3
## ..$ : int [1:3] 1 2 3
## ..$ : chr "a"
## ..$ : logi [1:3] TRUE FALSE TRUE
## $ : num [1:2] 2.5 4.2
To simply add a fourth list component without creating nested lists we use the
append() function:
l3 <- append(l1, list(c(2.5, 4.2)))
str(l3)
## List of 4
## $ : int [1:3] 1 2 3
## $ : chr "a"
## $ : num [1:2] 2.5 4.2
Alternatively, we can also add a new list component by utilizing the $ sign and
naming the new item:
l3$item4 <- "new list item"
str(l3)
## List of 5
## $ : int [1:3] 1 2 3
## $ : chr "a"
## $ : num [1:2] 2.5 4.2
## $ item4: chr "new list item"
11.3 Adding Attributes to Lists 93
To add individual elements to a specic list component we need to introduce

some subsetting which is further discussed later in the chapter in the Subsetting sec-
tion. Well continue with our original l1 list:
str(l1)
## List of 3
## $ : int [1:3] 1 2 3
## $ : chr "a"
To add additional values to a list item you need to subset for that specic list item
and then you can use the c() function to add the additional elements to that list
item:
l1[[1]] <- c(l1[[1]], 4:6)
str(l1)
## List of 3
## $ : int [1:6] 1 2 3 4 5 6
## $ : chr "a"
l1[[2]] <- c(l1[[2]], c("dding", "to a", "list"))

str(l1)
## List of 3
## $ : int [1:6] 1 2 3 4 5 6
## $ : chr [1:4] "a" "dding" "to a" "list"
11.3 Adding Attributes to Lists
The attributes that you can add to lists include names, general comments, and spe-
cic list item comments. Currently, our l1 list has no attributes:
attributes(l1)
## NULL
We can add names to lists in two ways. First, we can use names() to assign
names to list items in a pre-existing list. Second, we can add names to a list when
we are creating a list.
# adding names to a pre-existing list
names(l1) <- c("item1", "item2", "item3")
str(l1)
## List of 3
## $ item1: int [1:6] 1 2 3 4 5 6
## $ item2: chr [1:4] "a" "dding" "to a" "list"
## $ item3: logi [1:3] TRUE FALSE TRUE
attributes(l1)
## $names
## [1] "item1" "item2" "item3"
# adding names when creating lists

l2 <- list(item1 = 1:3, item2 = letters[1:5], item3 = c(T, F, T, T))
str(l2)
## List of 3
## $ item1: int [1:3] 1 2 3
## $ item2: chr [1:5] "a" "b" "c" "d"
## $ item3: logi [1:4] TRUE FALSE TRUE TRUE
attributes(l2)
## $names
We can also add comments to lists. As previously mentioned, comments act as a

note to the user without changing how the object behaves. With lists, we can add a
general comment to the list using comment() and we can also add comments to
specic list items with attr().
# adding a general comment to list l2 with comment()
comment(l2) <- "This is a comment on a list"
str(l2)
## List of 3
## $ item1: int [1:3] 1 2 3
## $ item2: chr [1:5] "a" "b" "c" "d"
## - attr(*, "comment")= chr "This is a comment on a list"
attributes(l2)
## $names
##
## $comment
## [1] "This is a comment on a list"
# adding a comment to a specific list item with attr()

attr(l2, "item2") <- "Comment for item2"
str(l2)
## List of 3
## $ item1: int [1:3] 1 2 3
## $ item2: chr [1:5] "a" "b" "c" "d"
## - attr(*, "comment")= chr "This is a comment on a list"
## - attr(*, "item2")= chr "Comment for item2"
attributes(l2)
## $names
##
## $comment
## [1] "This is a comment on a list"
##
## $item2
## [1] "Comment for item2"
11.4 Subsetting Lists 95
11.4 Subsetting Lists
If list x is a train carrying objects, then x[[5]] is the object in car 5; x[4:6] is a train of cars
4-6@RLangTip
To subset lists we can utilize the single bracket [ ], double brackets [[ ]], and
dollar sign $ operators. Each approach provides a specic purpose and can be com-
bined in different ways to achieve the following subsetting objectives:
Subset list and preserve output as a list
Subset list and simplify output
Subset list to get elements out of a list
Subset list with a nested list
11.4.1 Subset List and Preserve Output as a List
To extract one or more list items while preserving1 the output in list format use the
[ ] operator:
# extract first list item
l2[1]
## $item1
## [1] 1 2 3
# same as above but using the item's name

l2["item1"]
## $item1
## [1] 1 2 3
# extract multiple list items

l2[c(1,3)]
## $item1
## [1] 1 2 3
##
## $item3
## [1] TRUE FALSE TRUE TRUE
# same as above but using the items' names

l2[c("item1", "item3")]
## $item1
## [1] 1 2 3
##
## $item3
## [1] TRUE FALSE TRUE TRUE
1
Its important to understand the difference between simplifying and preserving subsetting.
Simplifying subsets returns the simplest possible data structure that can represent the output.
Preserving subsets keeps the structure of the output the same as the input. See Hadley Wickhams
section on Simplifying vs. Preserving Subsetting to learn more.
11.4.2 Subset List and Simplify Output
To extract one or more list items while simplifying the output use the [[ ]] or $
operator:
# extract first list item and simplify to a vector
l2[[1]]
## [1] 1 2 3

l2[["item1"]]
## [1] 1 2 3
# same as above but using the `$` operator

l2$item1
## [1] 1 2 3
One thing that differentiates the [[ operator from the $ is that the [[ operator can
be used with computed indices. The $ operator can only be used with literal names.
11.4.3 Subset List to Get Elements Out of a List
To extract individual elements out of a specic list item combine the [[ (or $)
operator with the [ operator:
# extract third element from the second list item
l2[[2]][3]
## [1] "c"

l2[["item2"]][3]
## [1] "c"
# same as above but using the `$` operator

l2$item2[3]
## [1] "c"
11.4.4 Subset List with a Nested List
If you have nested lists you can expand the ideas above to extract items and ele-
ments. Well use the following list l3 which has a nested list in item 2.
l3 <- list(item1 = 1:3,
item2 = list(item2a = letters[1:5],
item3b = c(T, F, T, T)))
str(l3)
11.4 Subsetting Lists 97
## List of 2
## $ item1: int [1:3] 1 2 3
## $ item2:List of 2
## ..$ item2a: chr [1:5] "a" "b" "c" "d"
## ..$ item3b: logi [1:4] TRUE FALSE TRUE TRUE
If the goal is to subset l3 to extract the nested list item item2a from item2,
we can perform this multiple ways.
# preserve the output as a list
l3[[2]][1]
## $item2a
## [1] "a" "b" "c" "d" "e"
# same as above but simplify the output

l3[[2]][[1]]
## [1] "a" "b" "c" "d" "e"
# same as above with names

l3[["item2"]][["item2a"]]
## [1] "a" "b" "c" "d" "e"
# same as above with `$` operator

l3$item2$item2a
## [1] "a" "b" "c" "d" "e"
# extract individual element from a nested list item

l3[[2]][[1]][3]
## [1] "c"
Chapter 12
Managing Matrices
A matrix is a collection of data elements arranged in a two-dimensional rectangular

layout. In R, the elements that make up a matrix must be of a consistent mode (i.e. all
elements must be numeric, or character, etc.). Therefore, a matrix can be thought of as
an atomic vector with a dimension attribute. Furthermore, all columns of a matrix must
be of same length. In this chapter I will illustrate how to create matrices, add additional
elements to pre-existing matrices, add attributes to matrices, and subset matrices.
12.1 Creating Matrices
Matrices are constructed column-wise, so entries can be thought of starting in the

upper left corner and running down the columns. We can create a matrix using the
matrix() function and specifying the values to fill in the matrix and the number
of rows and columns to make the matrix.
# numeric matrix
m1 <- matrix(1:6, nrow = 2, ncol = 3)
m1
## [,1] [,2] [,3]
## [1,] 1 3 5
## [2,] 2 4 6
The underlying structure of this matrix is simply an integer vector with an added
2 3 dimension attribute.
str(m1)
## int [1:2, 1:3] 1 2 3 4 5 6
attributes(m1)
## $dim
## [1] 2 3

100 12 Managing Matrices
Matrices can also contain character values. Whether a matrix contains data that
are of numeric or character type, all the elements must be of the same class.
# a character matrix
m2 <- matrix(letters[1:6], nrow = 2, ncol = 3)
m2
## [,1] [,2] [,3]
## [1,] "a" "c" "e"
## [2,] "b" "d" "f"
# structure of m2 is simply character vector with 2x3 dimension

str(m2)
## chr [1:2, 1:3] "a" "b" "c" "d" "e" "f"
attributes(m2)
## $dim
## [1] 2 3
Matrices can also be created using the column-bind cbind() and row-bind
rbind() functions. However, keep in mind that the vectors that are being binded
must be of equal length and mode.
v1 <- 1:4
v2 <- 5:8
cbind(v1, v2)
## v1 v2
## [1,] 1 5
## [2,] 2 6
## [3,] 3 7
## [4,] 4 8
rbind(v1, v2)
## [,1] [,2] [,3] [,4]
## v1 1 2 3 4
## v2 5 6 7 8
# bind several vectors together

v3 <- 9:12
cbind(v1, v2, v3)

## v1 v2 v3
## [1,] 1 5 9
## [2,] 2 6 10
## [3,] 3 7 11
## [4,] 4 8 12
12.2 Adding On To Matrices
We can leverage the cbind() and rbind() functions for adding onto matrices
as well. Again, its important to keep in mind that the vectors that are being binded
must be of equal length and mode to the pre-existing matrix.
12.3 Adding Attributes to Matrices 101
m1 <- cbind(v1, v2)

m1
## v1 v2
## [1,] 1 5
## [2,] 2 6
## [3,] 3 7
## [4,] 4 8
# add a new column

cbind(m1, v3)
## v1 v2 v3
## [1,] 1 5 9
## [2,] 2 6 10
## [3,] 3 7 11
## [4,] 4 8 12
# or add a new row

rbind(m1, c(4.1, 8.1))
## v1 v2
## [1,] 1.0 5.0
## [2,] 2.0 6.0
## [3,] 3.0 7.0
## [4,] 4.0 8.0
## [5,] 4.1 8.1
12.3 Adding Attributes to Matrices
As previously mentioned, matrices by default will have a dimension attribute as

illustrated in the following matrix m2.
# basic matrix
m2
## [,1] [,2] [,3]
## [1,] 1 5 9
## [2,] 2 6 10
## [3,] 3 7 11
## [4,] 4 8 12
# the dimension attribute shows this matrix has 4 rows and 3 columns
attributes(m2)
## $dim
## [1] 4 3
However, matrices can also have additional attributes such as row names, column
names, and comments. Adding names can be done individually, meaning we can
add row names or column names separately.
# add row names as an attribute

rownames(m2) <- c("row1", "row2", "row3", "row4")
m2
## [,1] [,2] [,3]
## row1 1 5 9
## row2 2 6 10
## row3 3 7 11
## row4 4 8 12
# attributes displayed will now show the dimension, list the row names
# and will show the column names as NULL
attributes(m2)
## $dim
## [1] 4 3
##
## $dimnames
## $dimnames[[1]]
## [1] "row1" "row2" "row3" "row4"
##
## $dimnames[[2]]
## NULL
# add column names

colnames(m2) <- c("col1", "col2", "col3")
m2
## col1 col2 col3
## row1 1 5 9
## row2 2 6 10
## row3 3 7 11
## row4 4 8 12
attributes(m2)
## $dim
## [1] 4 3
##
## $dimnames
## $dimnames[[1]]
## [1] "row1" "row2" "row3" "row4"
##
## $dimnames[[2]]
## [1] "col1" "col2" "col3"
Another option is to use the dimnames() function. To add row names you
assign the names to dimnames(m2)[[1]] and to add column names you assign
the names to dimnames(m2)[[2]].
dimnames(m2)[[1]] <- c("row_1", "row_2", "row_3", "row_4")
m2
## col1 col2 col3
## row_1 1 5 9
## row_2 2 6 10
## row_3 3 7 11
## row_4 4 8 12
12.4 Subsetting Matrices 103
# column names are contained in the second list item

dimnames(m2)[[2]] <- c("col_1", "col_2", "col_3")
m2
## col_1 col_2 col_3
## row_1 1 5 9
## row_2 2 6 10
## row_3 3 7 11
## row_4 4 8 12
Lastly, similar to lists and vectors you can add a comment attribute to a list.
comment(m2) <- "adding a comment to a matrix"
attributes(m2)
## $dim
## [1] 4 3
##
## $dimnames
## $dimnames[[1]]
## [1] "row_1" "row_2" "row_3" "row_4"
##
## $dimnames[[2]]
## [1] "col_1" "col_2" "col_3"
##
##
## $comment
## [1] "adding a comment to a matrix"
12.4 Subsetting Matrices
To subset matrices we use the [ operator; however, since matrices have two dimen-
sions we need to incorporate subsetting arguments for both row and column dimen-
sions. A generic form of matrix subsetting looks like: matrix[rows, columns].
We can illustrate with matrix m2:
m2
## row_1 1 5 9
## row_2 2 6 10
## row_3 3 7 11
## row_4 4 8 12
By using different values in the rows and columns argument of m2[rows,

columns], we can subset m2 in multiple ways.
# subset for rows 1 and 2 but keep all columns
m2[1:2, ]
## row_1 1 5 9
## row_2 2 6 10
# subset for columns 1 and 3 but keep all rows

m2[ , c(1, 3)]
## col_1 col_3
## row_1 1 9
## row_2 2 10
## row_3 3 11
## row_4 4 12
# subset for both rows and columns

m2[1:2, c(1, 3)]
## col_1 col_3
## row_1 1 9
## row_2 2 10
# use a vector to subset

v <- c(1, 2, 4)
m2[v, c(1, 3)]
## col_1 col_3
## row_1 1 9
## row_2 2 10
## row_4 4 12
# use names to subset

m2[c("row_1", "row_3"), ]
## row_1 1 5 9
## row_3 3 7 11
Note that subsetting matrices with the [ operator will simplify the results to the
lowest possible dimension. To avoid this you can introduce the drop = FALSE
argument:
# simplifying results in a named vector
m2[, 2]
## row_1 row_2 row_3 row_4
## 5 6 7 8
# preserving results in a 4x1 matrix

m2[, 2, drop = FALSE]
## col_2
## row_1 5
## row_2 6
## row_3 7
## row_4 8
Chapter 13
Managing Data Frames
A data frame is the most common way of storing data in R and, generally, is the data
structure most often used for data analyses. Under the hood, a data frame is a list of
equal-length vectors. Each element of the list can be thought of as a column and the
length of each element of the list is the number of rows. As a result, data frames can
store different classes of objects in each column (i.e. numeric, character, factor).
In essence, the easiest way to think of a data frame is as an Excel worksheet that
contains columns of different types of data but are all of equal length rows. In this
chapter I will illustrate how to create data frames, add additional elements to
pre-existing data frames, add attributes to data frames, and subset data frames.
13.1 Creating Data Frames
Data frames are usually created by reading in a dataset using read.table() or

read.csv(); this will be covered in the importing and scraping data chapters.
However, data frames can also be created explicitly with the data.frame()
function or they can be coerced from other types of objects like lists. In this case Ill
create a simple data frame df and assess its basic structure:
df <- data.frame(col1 = 1:3,
col2 = c("this", "is", "text"),
col3 = c(TRUE, FALSE, TRUE),
col4 = c(2.5, 4.2, pi))
# assess the structure of a data frame

str(df)
## $ col1: int 1 2 3
## $ col2: Factor w/ 3 levels "is","text","this": 3 1 2
## $ col3: logi TRUE FALSE TRUE
## $ col4: num 2.5 4.2 3.14

106 13 Managing Data Frames
# number of rows
nrow(df)
## [1] 3
# number of columns
ncol(df)
## [1] 4
Note how col2 in df was converted to a column of factors. This is because

there is a default setting in data.frame() that converts character columns to
factors. We can turn this off by setting the stringsAsFactors = FALSE
argument:
df <- data.frame(col1 = 1:3,
col2 = c("this", "is", "text"),
col3 = c(TRUE, FALSE, TRUE),
col4 = c(2.5, 4.2, pi),
stringsAsFactors = FALSE)
# note how col2 now is of a character class

str(df)
## $ col1: int 1 2 3
## $ col2: chr "this" "is" "text"
## $ col4: num 2.5 4.2 3.14
We can also convert pre-existing structures to a data frame. The following illus-
trates how we can turn multiple vectors, a list, or a matrix into a data frame:
v1 <- 1:3
v2 <-c("this", "is", "text")
v3 <- c(TRUE, FALSE, TRUE)
# convert same length vectors to a data frame using data.frame()

data.frame(col1 = v1, col2 = v2, col3 = v3)
## col1 col2 col3
## 1 1 this TRUE
## 2 2 is FALSE
## 3 3 text TRUE
# convert a list to a data frame using as.data.frame()

l <- list(item1 = 1:3,
item2 = c("this", "is", "text"),
item3 = c(2.5, 4.2, 5.1))
l
## $item1
## [1] 1 2 3
##
## $item2
## [1] "this" "is" "text"
##
## $item3
## [1] 2.5 4.2 5.1
13.2 Adding On To Data Frames 107
as.data.frame(l)
## item1 item2 item3
## 1 1 this 2.5
## 2 2 is 4.2
## 3 3 text 5.1
# convert a matrix to a data frame using as.data.frame()

m1
## [,1] [,2] [,3]
## [1,] 1 5 9
## [2,] 2 6 10
## [3,] 3 7 11
## [4,] 4 8 12
as.data.frame(m1)
## V1 V2 V3
## 1 1 5 9
## 2 2 6 10
## 3 3 7 11
## 4 4 8 12
13.2 Adding On To Data Frames
We can leverage the cbind() function for adding columns to a data frame. Note
that one of the objects being combined must already be a data frame otherwise
cbind() could produce a matrix.
df
## col1 col2 col3 col4
## 1 1 this TRUE 2.500000
## 2 2 is FALSE 4.200000
## 3 3 text TRUE 3.141593
# add a new column

v4 <- c("A", "B", "C")
cbind(df, v4)
## col1 col2 col3 col4 v4
## 1 1 this TRUE 2.500000 A
## 2 2 is FALSE 4.200000 B
## 3 3 text TRUE 3.141593 C
We can also use the rbind() function to add data frame rows together.
However, severe caution should be taken because this can cause changes in the
classes of the columns. For instance, our data frame df currently consists of an
integer, character, logical, and numeric variables.
df
## 1 1 this TRUE 2.500000
## 2 2 is FALSE 4.200000
## 3 3 text TRUE 3.141593

str(df)
## $ col1: int 1 2 3
## $ col2: chr "this" "is" "text"
## $ col4: num 2.5 4.2 3.14
If we attempt to add a row using rbind() and c() it converts all columns to a
character class. This is because all elements in the vector created by c() must be of
the same class so they are all coerced to the character class which then coerces all
the variables in the data frame to the character class.
df2 <- rbind(df, c(4, "R", F, 1.1))
df2
## 1 1 this TRUE 2.5
## 2 2 is FALSE 4.2
## 3 3 text TRUE 3.14159265358979
## 4 4 R FALSE 1.1
str(df2)
## $ col1: chr "1" "2" "3" "4"
## $ col2: chr "this" "is" "text" "R"
## $ col3: chr "TRUE" "FALSE" "TRUE" "FALSE"
## $ col4: chr "2.5" "4.2" "3.14159265358979" "1.1"
To add rows appropriately, we need to convert the items being added to a data
frame and make sure the columns are the same class as the original data frame.
adding_df <- data.frame(col1 = 4,
col2 = "R",
col3 = FALSE,
col4 = 1.1,
df3 <- rbind(df, adding_df)

df3
## 1 1 this TRUE 2.500000
## 2 2 is FALSE 4.200000
## 3 3 text TRUE 3.141593
## 4 4 R FALSE 1.100000
str(df3)
## $ col1: num 1 2 3 4
## $ col2: chr "this" "is" "text" "R"
## $ col3: logi TRUE FALSE TRUE FALSE
## $ col4: num 2.5 4.2 3.14 1.1
There are better ways to join data frames together than to use cbind() and
rbind(). These are covered later on in the transforming your data with dplyr
chapter.
13.3 Adding Attributes to Data Frames 109
13.3 Adding Attributes to Data Frames
Similar to matrices, data frames will have a dimension attribute. In addition, data
frames can also have additional attributes such as row names, column names, and
comments. We can illustrate with data frame df.
# basic data frame
df
## 1 1 this TRUE 2.500000
## 2 2 is FALSE 4.200000
## 3 3 text TRUE 3.141593
dim(df)
## [1] 3 4
attributes(df)
## $names
## [1] "col1" "col2" "col3" "col4"
##
## $row.names
## [1] 1 2 3
##
## $class
## [1] "data.frame"
Currently df does not have row names but we can add them with
rownames():
# add row names
rownames(df) <- c("row1", "row2", "row3")
df
## row1 1 this TRUE 2.500000
## row2 2 is FALSE 4.200000
## row3 3 text TRUE 3.141593
attributes(df)
## $names
## [1] "col1" "col2" "col3" "col4"
##
## $row.names
## [1] "row1" "row2" "row3"
##
## $class
## [1] "data.frame"
We can also change the existing column names by using colnames() or

names():
# add/change column names with colnames()
colnames(df) <- c("col_1", "col_2", "col_3", "col_4")
df
## col_1 col_2 col_3 col_4
## row1 1 this TRUE 2.500000
## row2 2 is FALSE 4.200000
## row3 3 text TRUE 3.141593
attributes(df)
## $names
## [1] "col_1" "col_2" "col_3" "col_4"
##
## $row.names
## [1] "row1" "row2" "row3"
##
## $class
## [1] "data.frame"
# add/change column names with names()

names(df) <- c("col.1", "col.2", "col.3", "col.4")
df
## col.1 col.2 col.3 col.4
## row1 1 this TRUE 2.500000
## row2 2 is FALSE 4.200000
## row3 3 text TRUE 3.141593
attributes(df)
## $names
## [1] "col.1" "col.2" "col.3" "col.4"
##
## $row.names
## [1] "row1" "row2" "row3"
##
## $class
## [1] "data.frame"
Lastly, just like vectors, lists, and matrices, we can add a comment to a data
frame without affecting how it operates.
# adding a comment attribute
comment(df) <- "adding a comment to a data frame"
attributes(df)
## $names
## [1] "col.1" "col.2" "col.3" "col.4"
##
## $row.names
## [1] "row1" "row2" "row3"
##
## $class
## [1] "data.frame"
##
## $comment
## [1] "adding a comment to a data frame"
13.4 Subsetting Data Frames 111
13.4 Subsetting Data Frames
Data frames possess the characteristics of both lists and matrices: if you subset with
a single vector, they behave like lists and will return the selected columns with all
rows; if you subset with two vectors, they behave like matrices and can be subset by
row and column:
df
## row1 1 this TRUE 2.500000
## row2 2 is FALSE 4.200000
## row3 3 text TRUE 3.141593
# subsetting by row numbers

df[2:3, ]
## row2 2 is FALSE 4.200000
## row3 3 text TRUE 3.141593
# subsetting by row names

df[c("row2", "row3"), ]
## row2 2 is FALSE 4.200000
## row3 3 text TRUE 3.141593
# subsetting columns like a list

df[c("col.2", "col.4")]
## col.2 col.4
## row1 this 2.500000
## row2 is 4.200000
## row3 text 3.141593
# subsetting columns like a matrix

df[ , c("col.2", "col.4")]
## col.2 col.4
## row1 this 2.500000
## row2 is 4.200000
## row3 text 3.141593
# subset for both rows and columns

df[1:2, c(1, 3)]
## col.1 col.3
## row1 1 TRUE
## row2 2 FALSE
# use a vector to subset

v <- c(1, 2, 4)
df[ , v]
## col.1 col.2 col.4
## row1 1 this 2.500000
## row2 2 is 4.200000
## row3 3 text 3.141593
Note that subsetting data frames with the [ operator will simplify the results to
the lowest possible dimension. To avoid this you can introduce the drop = FALSE
argument:
# simplifying results in a named vector
df[, 2]
## [1] "this" "is" "text"
# preserving results in a 3x1 data frame

df[, 2, drop = FALSE]
## col.2
## row1 this
## row2 is
## row3 text
Chapter 14
Dealing with Missing Values
A common task in data analysis is dealing with missing values. In R, missing values
are often represented by NA or some other value that represents missing values (i.e.
99). We can easily work with missing values and in this chapter I illustrate how to
test for, recode, and exclude missing values in your data.
14.1 Testing for Missing Values
To identify missing values use is.na() which returns a logical vector with TRUE
in the element locations that contain missing values represented by NA. is.na()
will work on vectors, lists, matrices, and data frames.
# vector with missing data
x <- c(1:4, NA, 6:7, NA)
x
## [1] 1 2 3 4 NA 6 7 NA
is.na(x)
## [1] FALSE FALSE FALSE FALSE TRUE FALSE FALSE TRUE
# data frame with missing data

df <- data.frame(col1 = c(1:3, NA),
col2 = c("this", NA,"is", "text"),
col3 = c(TRUE, FALSE, TRUE, TRUE),
col4 = c(2.5, 4.2, 3.2, NA),
# identify NAs in full data frame

is.na(df)
## [1,] FALSE FALSE FALSE FALSE
## [2,] FALSE TRUE FALSE FALSE
## [3,] FALSE FALSE FALSE FALSE
## [4,] TRUE FALSE FALSE TRUE

114 14 Dealing with Missing Values
# identify NAs in specific data frame column

is.na(df$col4)
## [1] FALSE FALSE FALSE TRUE
To identify the location or the number of NAs we can leverage the which() and
sum() functions:
# identify location of NAs in vector
which(is.na(x))
## [1] 5 8
# identify count of NAs in data frame

sum(is.na(df))
## [1] 3
14.2 Recoding Missing Values
To recode missing values; or recode specic indicators that represent missing values,
we can use normal subsetting and assignment operations. For example, we can recode
missing values in vector x with the mean values in x by rst subsetting the vector to
identify NAs and then assign these elements a value. Similarly, if missing values are
represented by another value (i.e. 99) we can simply subset the data for the elements
that contain that value and then assign a desired value to those elements.
# recode missing values with the mean
x[is.na(x)] <- mean(x, na.rm = TRUE)
round(x, 2)
## [1] 1.00 2.00 3.00 4.00 3.83 6.00 7.00 3.83
# data frame that codes missing values as 99

df <- data.frame(col1 = c(1:3, 99), col2 = c(2.5, 4.2, 99, 3.2))
# change 99 s to NAs
df[df == 99] <- NA
df
## col1 col2
## 1 1 2.5
## 2 2 4.2
## 3 3 NA
## 4 NA 3.2
14.3 Excluding Missing Values
We can exclude missing values in a couple different ways. First, if we want to

exclude missing values from mathematical operations use the na.rm = TRUE
argument. If you do not exclude these values most functions will return an NA.
14.3 Excluding Missing Values 115
# A vector with missing values

x <- c(1:4, NA, 6:7, NA)
# including NA values will produce an NA output

mean(x)
## [1] NA
# excluding NA values will calculate the mathematical

# operation for all non-missing values
mean(x, na.rm = TRUE)
## [1] 3.833333
We may also desire to subset our data to obtain complete observations, those
observations (rows) in our data that contain no missing data. We can do this a few
different ways.
# data frame with missing values

df <- data.frame(col1 = c(1:3, NA),
col2 = c("this", NA,"is", "text"),
col3 = c(TRUE, FALSE, TRUE, TRUE),
col4 = c(2.5, 4.2, 3.2, NA),
df
## 2 2 <NA> FALSE 4.2
## 3 3 is TRUE 3.2
## 4 NA text TRUE NA
First, to nd complete cases we can leverage the complete.cases() function

which returns a logical vector identifying rows which are complete cases. So in the fol-
lowing case rows 1 and 3 are complete cases. We can use this information to subset our
data frame which will return the rows which complete.cases() found to be TRUE.
complete.cases(df)
## [1] TRUE FALSE TRUE FALSE
# subset with complete.cases to get complete cases

df[complete.cases(df), ]
## 3 3 is TRUE 3.2
# or subset with `!` operator to get incomplete cases

df[!complete.cases(df), ]
## 2 2 <NA> FALSE 4.2
## 4 NA text TRUE NA
116 14 Dealing with Missing Values
A shorthand alternative is to simply use na.omit() to omit all rows containing

missing values.
# or use na.omit() to get same as above
na.omit(df)
## 3 3 is TRUE 3.2
Part IV
Importing, Scraping, and Exporting Data
with R
What we have is a data glut.
Vernon Vinge
Data are being generated by everything around us at all times. Every digital process
and social media exchange produces it. Systems, sensors and mobile devices trans-
mit it. Countless databases collect it. Data are arriving from multiple sources at an
alarming rate and analysts and organizations are seeking ways to leverage these new
sources of information. Consequently, analysts need to understand how to get data
from these data sources. Furthermore, since analysis is often a collaborative effort
analysts also need to know how to share their data.
This section covers the process of importing, scraping, and exporting data. First,
I cover the basics of importing tabular and spreadsheet data. Second, since modern
day data wrangling often includes scraping data from the flood of web-based data
becoming available to organizations and analysts, I cover the fundamentals of web-
scraping with R. This includes importing spreadsheet data files stored online, scrap-
ing HTML text and data tables, and leveraging APIs. Third, although getting data
into R is essential, I also cover the equally important process of getting data out of
R. Consequently, this section will give you a strong foundation for the different
ways to get your data into and out of R.
Chapter 15
Importing Data
The rst step to any data analysis process is to get the data. Data can come from
many sources but two of the most common include text and Excel les. This chapter
covers how to import data into R by reading data from common text les and Excel
spreadsheets. In addition, I cover how to load data from saved R object les for
holding or transferring data that has been processed in R. In addition to the com-
monly used base R functions to perform data importing, I will also cover functions
from the popular readr, xlsx, and readxl packages.
15.1 Reading Data from Text Files
Text les are a popular way to hold and exchange tabular data as almost any data
application supports exporting data to the CSV (or other text le) formats. Text le
formats use delimiters to separate the different elements in a line, and each line of
data is in its own line in the text le. Therefore, importing different kinds of text
les can follow a fairly consistent process once youve identied the delimiter.
There are two main groups of functions that we can use to read in text les:
Base R functions
readr package functions
15.1.1 Base R Functions
read.table() is a multipurpose work-horse function in base R for importing

data. The functions read.csv() and read.delim() are special cases of
read.table() in which the defaults have been adjusted for efciency.

120 15 Importing Data
To illustrate these functions lets work with a CSV le that is saved in our working
directory which looks like:
variable 1,variable 2,variable 3
10,beer,TRUE
25,wine,TRUE
8,cheese,FALSE
To read in the CSV le we can use read.csv(). Note that when we assess the
structure of the data set that we read in, variable.2 is automatically coerced to
a factor variable and variable.3 is automatically coerced to a logical variable.
Furthermore, any whitespace in the column names are replaced with a ..
mydata = read.csv("mydata.csv")
mydata
## variable.1 variable.2 variable.3
## 1 10 beer TRUE
## 2 25 wine TRUE
## 3 8 cheese FALSE
str(mydata)
## $ variable.1: int 10 25 8
## $ variable.2: Factor w/ 3 levels "beer","cheese",..: 1 3 2
## $ variable.3: logi TRUE TRUE FALSE
However, we may want to read in variable.2 as a character variable rather

then a factor. We can take care of this by changing the stringsAsFactors argu-
ment. The default has stringsAsFactors = TRUE; however, setting it equal
to FALSE will read in the variable as a character variable.
mydata_2 = read.csv("mydata.csv", stringsAsFactors = FALSE)
mydata_2
## 1 10 beer TRUE
## 2 25 wine TRUE
## 3 8 cheese FALSE
str(mydata_2)
## $ variable.1: int 10 25 8
## $ variable.2: chr "beer" "wine" "cheese"
As previously stated read.csv is just a wrapper for read.table but with

adjusted default arguments. Therefore, we can use read.table to read in this same
data. The two arguments we need to be aware of are the eld separator (sep) and the
argument indicating whether the le contains the names of the variables as its rst
line (header). In read.table the defaults are sep = "" and header =
FALSE whereas in read.csv the defaults are sep = "," and header = TRUE.
15.1 Reading Data from Text Files 121
There are multiple other arguments we can use for certain situations which we
illustrate below:
# provides same results as read.csv above
read.table("mydata.csv", sep=",", header = TRUE, stringsAsFactors = FALSE)
## 1 10 beer TRUE
## 2 25 wine TRUE
## 3 8 cheese FALSE
# set column and row names

read.table("mydata.csv", sep=",", header = TRUE, stringsAsFactors
= FALSE,
col.names = c("Var 1", "Var 2", "Var 3"),
row.names = c("Row 1", "Row 2", "Row 3"))
## Var.1 Var.2 Var.3
## Row 1 10 beer TRUE
## Row 2 25 wine TRUE
## Row 3 8 cheese FALSE
# manually set the classes of the columns

set_classes <- read.table("mydata.csv", sep=",", header = TRUE,
colClasses = c("numeric", "character",
"character"))
str(set_classes)
## $ variable.1: num 10 25 8
## $ variable.3: chr "TRUE" "TRUE" "FALSE"
# limit the number of rows to read in

read.table("mydata.csv", sep=",", header = TRUE, nrows = 2)
## 1 10 beer TRUE
## 2 25 wine TRUE
In addition to CSV les, there are other text les that read.table works with.
The primary difference is what separates the elements. For example, tab delimited
text les typically end with the .txt extension. You can also use the read.
delim() function as, similiar to read.csv(), read.delim() is a wrapper of
read.table() with defaults set specically for tab delimited les.
# reading in tab delimited text files
read.delim("mydata.txt")
## 1 10 beer TRUE
## 2 25 wine TRUE
## 3 8 cheese FALSE
# provides same results as read.delim

read.table("mydata.txt", sep="\t", header = TRUE)
## 1 10 beer TRUE
## 2 25 wine TRUE
## 3 8 cheese FALSE
15.1.2 readr Package
Compared to the equivalent base functions, readr functions are around 10 faster.
They bring consistency to importing functions, they produce data frames in a
data.table format which are easier to view for large data sets, the default set-
tings removes the hassels of stringsAsFactors, and they have a more ex-
ible column specication.
To illustrate, we can use read_csv() which is equivalent to base Rs read.
csv() function. However, note that read_csv() maintains the full variable
name (whereas read.csv eliminates any spaces in variable names and lls it with
.). Also, read_csv() automatically sets stringsAsFactors = FALSE,
which can be a controversial topic.1
library(readr)
mydata_3 = read_csv("mydata.csv")
mydata_3
## variable 1 variable 2 variable 3
## 1 10 beer TRUE
## 2 25 wine TRUE
## 3 8 cheese FALSE
str(mydata_3)
## Classes 'tbl_df', 'tbl' and 'data.frame': 3 obs. of 3 variables:
## $ variable 1: int 10 25 8
## $ variable 2: chr "beer" "wine" "cheese"
## $ variable 3: logi TRUE TRUE FALSE
read_csv also offers many additional arguments for making adjustments to

your data as you read it in:
# specify the column class using col_types
read_csv("mydata.csv", col_types = list(col_double(),
col_character(),
col_character()))
## 1 10 beer TRUE
## 2 25 wine TRUE
## 3 8 cheese FALSE
# we can also specify column classes with a string

# in this example d = double, _ skips column, c = character
read_csv("mydata.csv", col_types = "d_c")
## variable 1 variable 3
## 1 10 TRUE
## 2 25 TRUE
## 3 8 FALSE
1
An interesting biography of the stringsAsFactors argument can be found at http://simplystatistics.
org/2015/07/24/stringsasfactors-an-unauthorized-biography/
15.2 Reading Data from Excel Files 123
# set column names

read_csv("mydata.csv", col_names = c("Var 1", "Var 2", "Var 3"), skip = 1)
## Var 1 Var 2 Var 3
## 1 10 beer TRUE
## 2 25 wine TRUE
## 3 8 cheese FALSE
# set the maximum number of lines to read in

read_csv("mydata.csv", n_max = 2)
## 1 10 beer TRUE
## 2 25 wine TRUE
Similar to base R, readr also offers functions to import .txt les (read_
delim()), xed-width les (read_fwf()), general text les (read_table()),
and more.
These examples provide the basics for reading in text les. However, sometimes
even text les can offer unanticipated difculties with their formatting. Both the
base R and readr functions offer many arguments to deal with different formatting
issues and I suggest you take time to look at the help les for these functions to learn
more (i.e. ?read.table). Also, you will nd more resources at the end of this
chapter for importing les.
15.2 Reading Data from Excel Files
With Excel still being the spreadsheet software of choice its important to be able to
efciently import and export data from these les. Often, R users will simply resort
to exporting the Excel le as a CSV le and then import into R using read.csv;
however, this is far from efcient. This section will teach you how to eliminate the
CSV step and to import data directly from Excel using two different packages:
xlsx package
readxl package
Note that there are several packages available to connect R with Excel (i.e.
gdata, RODBC, XLConnect, RExcel, etc.); however, I am only going to cover
the two main packages that I use which provide all the fundamental requirements
Ive needed for dealing with Excel.
15.2.1 xlsx Package
The xlsx package provides tools necessary to interact with Excel 2007 (and older)
les from R. Many of the benets of the xlsx come from being able to export and
format Excel les from R. Some of these capabilities will be covered in the
Exporting Data chapter; however, in this section we will simply cover importing
data from Excel with the xlsx package.
To illustrate, well use similar data from the previous section; however, saved as
an .xlsx le in our working director. To import the Excel data we simply use the
read.xlsx() function:
library(xlsx)
# read in first worksheet using a sheet index or name

read.xlsx("mydata.xlsx", sheetName = "Sheet1")
## 1 10 beer TRUE
## 2 25 wine TRUE
## 3 8 cheese FALSE
read.xlsx("mydata.xlsx", sheetIndex = 1)
## 1 10 beer TRUE
## 2 25 wine TRUE
## 3 8 cheese FALSE
# read in second worksheet

## variable.4 variable.5
## 1 Dayton johnny
## 2 Columbus amber
## 3 Cleveland tony
## 4 Cincinnati alice
Since Excel is such a exible spreadsheet software, people often make notes,
comments, headers, etc. at the beginning or end of the les which we may not want
to include. If we want to read in data that starts further down in the Excel worksheet
we can include the startRow argument. If we have a specic range of rows (or
columns) to include we can use the rowIndex (or colIndex) argument.
# a worksheet with comments in the first two lines
## HEADER..COMPANY.A NA.
## 1 What if we want to disregard header text in Excel file? <NA>
## 2 variable 6 variable 7
## 3 200 Male
## 4 225 Female
## 5 400 Female
## 6 310 Male
# read in all data below the second line

read.xlsx("mydata.xlsx", sheetName = "Sheet3", startRow = 3)
## 1 200 Male
## 2 225 Female
## 3 400 Female
## 4 310 Male
# read in a range of rows

read.xlsx("mydata.xlsx", sheetName = "Sheet3", rowIndex = 3:5)
## 1 200 Male
## 2 225 Female
15.2 Reading Data from Excel Files 125
We can also change the class type of the columns when we read them in:
# read in data without changing class type
mydata_sheet1.1 <- read.xlsx("mydata.xlsx", sheetName = "Sheet1")
str(mydata_sheet1.1)
## $ variable.1: num 10 25 8
## $ variable.2: Factor w/ 3 levels "beer","cheese",..: 1 3 2
# read in data and change class type

mydata_sheet1.2 <- read.xlsx("mydata.xlsx", sheetName = "Sheet1",
stringsAsFactors = FALSE,
colClasses = c("double", "character",
"logical"))
str(mydata_sheet1.2)
## $ variable.1: num 10 25 8
Another useful argument is keepFormulas which allows you to see the text
of any formulas in the Excel spreadsheet:
# by default keepFormula is set to FALSE so only
# the formula output will be read in
## Future.Value Rate Periods Present.Value
## 1 500 0.065 10 266.3630
## 2 600 0.085 6 367.7671
## 3 750 0.080 11 321.6621
## 4 1000 0.070 16 338.7346
# changing the keepFormula to TRUE will display the equations

read.xlsx("mydata.xlsx", sheetName = "Sheet4", keepFormulas = TRUE)
## Future.Value Rate Periods Present.Value
## 1 500 0.065 10 A2/(1+B2)^C2
## 2 600 0.085 6 A3/(1+B3)^C3
## 3 750 0.080 11 A4/(1+B4)^C4
## 4 1000 0.070 16 A5/(1+B5)^C5
15.2.2 readxl Package
readxl is one of the newest packages for accessing Excel data with R and was devel-
oped by Hadley Wickham and the RStudio team who also developed the readr
package. This package works with both legacy .xls formats and the modern xml-
based .xlsx format. Similar to readr the readxl functions are based on a C++
library so they are extremely fast. Unlike most other packages that deal with Excel,
readxl has no external dependencies, so you can use it to read Excel data on just
about any platform. Additional benets readxl provides includes the ability to
load dates and times as POSIXct formatted dates, automatically drops blank col-
umns, and returns outputs as data.table formatted which provides easier viewing for
large data sets.
To read in Excel data with readxl you use the read_excel() function
which has very similar operations and arguments as xlsx. A few important differ-
ences you will see below include: readxl will automatically convert date and
date-time variables to POSIXct formatted variables, character variables will not be
coerced to factors, and logical variables will be read in as integers.
library(readxl)
mydata <- read_excel("mydata.xlsx", sheet = "Sheet5")

mydata
## variable 1 variable 2 variable 3 variable 4 variable 5
## 1 10 beer 1 2015-11-20 2015-11-20 13:30:00
## 2 25 wine 1 <NA> 2015-11-21 16:30:00
## 3 8 <NA> 0 2015-11-22 2015-11-22 14:45:00
str(mydata)
## Classes 'tbl_df', 'tbl' and 'data.frame': 3 obs. of 5 variables:
## $ variable 1: num 10 25 8
## $ variable 2: chr "beer" "wine" NA
## $ variable 3: num 1 1 0
## $ variable 4: POSIXct, format: "2015-11-20" NA
## $ variable 5: POSIXct, format: "2015-11-20 13:30:00" "2015-11-21 16:30:00"
The available arguments allow you to change the data as you import it. Some
examples are provided:
# change variable names by skipping the first row
# and using col_names to set the new names
read_excel("mydata.xlsx", sheet = "Sheet5", skip = 1,
col_names = paste("Var", 1:5))
## Var 1 Var 2 Var 3 Var 4 Var 5
## 1 10 beer 1 42328 2015-11-20 13:30:00
## 2 25 wine 1 NA 2015-11-21 16:30:00
## 3 8 <NA> 0 42330 2015-11-22 14:45:00
# sometimes missing values are set as a sentinel value

# rather than just left blank - (i.e. "999")
read_excel("mydata.xlsx", sheet = "Sheet6")
## variable 1 variable 2 variable 3 variable 4
## 1 10 beer 1 42328
## 2 25 wine 1 999
## 3 8 999 0 42330
# we can change these to missing values with na argument

read_excel("mydata.xlsx", sheet = "Sheet6", na = "999")
## variable 1 variable 2 variable 3 variable 4
## 1 10 beer 1 42328
## 2 25 wine 1 NA
## 3 8 <NA> 0 42330
15.4 Additional Resources 127
One unique difference between readxl and xlsx is how to deal with column
types. Whereas read.xlsx() allows you to change the column types to integer,
double, numeric, character, or logical; read_excel() restricts you to changing
column types to blank, numeric, date, or text. The blank option allows you to skip
columns; however, to change variable 3 to a logical TRUE/FALSE variable requires
a second step.
mydata_ex <- read_excel("mydata.xlsx", sheet = "Sheet5",
col_types = c("numeric", "blank", "numeric",
"date", "blank"))
mydata_ex
## 1 10 1 2015-11-20
## 2 25 1 <NA>
## 3 8 0 2015-11-22
# change variable 3 to a logical variable

mydata_ex$`variable 3` <- as.logical(mydata_ex$`variable 3`)
mydata_ex
## 1 10 TRUE 2015-11-20
## 2 25 TRUE <NA>
## 3 8 FALSE 2015-11-22
15.3 Load Data from Saved R Object File
Sometimes you may need to save data or other R objects outside of your workspace.
You may want to share R data/objects with co-workers, transfer between projects or
computers, or simply archive them. There are three primary ways that people tend
to save R data/objects: as .RData, .rda, or as .rds les. The differences behind when
you use each will be covered in the Saving data as an R object le section. This sec-
tion simply shows how to load these data/object forms.
load("mydata.RData")
load(file = "mydata.rda")
name <- readRDS("mydata.rds")
In addition to text and Excel les, there are multiple other ways that data are stored
and exchanged. Commercial statistical software such as SPSS, SAS, Stata, and
Minitab often have the option to store data in a specic format for that software.
In addition, analysts commonly use databases to store large quantities of data. R has
good support to work with these additional options which we did not cover here.
The following provides a list of additional resources to learn about data importing
for these specic cases:
R data import/export manual: https://cran.r-project.org/doc/manuals/R-data.html
Working with databases
MySQL: https://cran.r-project.org/web/packages/RMySQL/index.html
Oracle: https://cran.r-project.org/web/packages/ROracle/index.html
PostgreSQL: https://cran.r-project.org/web/packages/RPostgreSQL/index.html
SQLite: https://cran.r-project.org/web/packages/RSQLite/index.html
Open Database Connectivity databases: https://cran.rstudio.com/web/packages/
RODBC/
Importing data from commercial software2
The foreign package provides functions that help you load data les from
other programs such as SPSS, SAS, Stata, and others into R.
2
https://cran.r-project.org/doc/manuals/R-data.html#Importing-from-other-statistical-systems
Chapter 16
Scraping Data
Rapid growth of the World Wide Web has signicantly changed the way we share,
collect, and publish data. Vast amount of information is being stored online, both in
structured and unstructured forms. Regarding certain questions or research topics,
this has resulted in a new problemno longer is the concern of data scarcity and
inaccessibility but, rather, one of overcoming the tangled masses of online data.
Collecting data from the web is not an easy process as there are many technolo-
gies used to distribute web content (i.e. HTML, XML, JSON). Therefore, dealing
with more advanced web scraping requires familiarity in accessing data stored in
these technologies via R. Through this chapter I will provide an introduction to
some of the fundamental tools required to perform basic web scraping. This includes
importing spreadsheet data les stored online, scraping HTML text, scraping HTML
table data, and leveraging APIs to scrape data.
My purpose in the following sections is to discuss these topics at a level meant to
get you started in web scraping; however, this area is vast and complex and this
chapter will far from provide you expertise level insight. To advance your knowl-
edge I highly recommend getting copies of XML and Web Technologies for Data
Sciences with R (Nolan and Lang, 2014) and Automated Data Collection with R
(Munzert et al., 2014).
16.1 Importing Tabular and Excel Files Stored Online
The most basic form of getting data from online is to import tabular (i.e. .txt, .csv) or
Excel les that are being hosted online. This is often not considered web scraping1;
however, I think its a good place to start introducing the user to interacting with the
web for obtaining data. Importing tabular data is especially common for the many
1
In Automated Data Collection with R Munzert et al. state that [t]he rst way to get data from the
web is almost too banal to be considered here and actually not a case of web scraping in the nar-
rower sense.

130 16 Scraping Data
types of government data available online. A quick perusal of Data.gov illustrates

nearly 188,510 examples. In fact, we can provide our rst example of importing
online tabular data by downloading the Data.gov CSV le that lists all the federal
agencies that supply data to Data.gov.
# the url for the online CSV
url <- "https://www.data.gov/media/federal-agency-participation.csv"
# use read.csv to import

data_gov <- read.csv(url, stringsAsFactors = FALSE)
# for brevity I only display first 6 rows

data_gov[1:6, c(1,3:4)]
## Agency.Name Datasets Last.Entry
## 1 Commodity Futures Trading Commission 3 01/12/2014
## 2 Consumer Financial Protection Bureau 2 09/26/2015
## 3 Consumer Financial Protection Bureau 2 09/26/2015
## 4 Corporation for National and Community Service 3 01/12/2014
## 5 Court Services and Offender Supervision Agency 1 01/12/2014
## 6 Department of Agriculture 698 12/01/2015
Downloading Excel spreadsheets hosted online can be performed just as easily.

Recall that there is not a base R function for importing Excel data; however, several
packages exist to handle this capability. One package that works smoothly with pull-
ing Excel data from URLs is gdata. With gdata we can use read.xls() to
download this Fair Market Rents for Section 8 Housing Excel le from the given url.
library(gdata)
# the url for the online Excel file

url <- "http://www.huduser.org/portal/datasets/fmr/fmr2015f/FY2015F_4050_Final.xls"
# use read.xls to import

rents <- read.xls(url)
rents[1:6, 1:10]
## fips2000 fips2010 fmr2 fmr0 fmr1 fmr3 fmr4 county State CouSub
## 1 100199999 100199999 788 628 663 1084 1288 1 1 99999
## 2 100399999 100399999 762 494 643 1123 1318 3 1 99999
## 3 100599999 100599999 670 492 495 834 895 5 1 99999
## 4 100799999 100799999 773 545 652 1015 1142 7 1 99999
## 5 100999999 100999999 773 545 652 1015 1142 9 1 99999
## 6 101199999 101199999 599 481 505 791 1061 11 1 99999
Note that many of the arguments covered in the Importing Data chapter (i.e.
specifying sheets to read from, skipping lines) also apply to read.xls(). In addi-
tion, gdata provides some useful functions (sheetCount() and sheet-
Names()) for identifying if multiple sheets exist prior to downloading.
Another common form of le storage is using zip les. For instance, the Bureau
of Labor Statistics (BLS) stores their public-use microdata for the Consumer
Expenditure Survey in .zip les.2 We can use download.file() to download the
le to your working directory and then work with this data as desired.
2
http://www.bls.gov/cex/pumd_data.htm#csv
16.1 Importing Tabular and Excel Files Stored Online 131
url <- "http://www.bls.gov/cex/pumd/data/comma/diary14.zip"
# download .zip file and unzip contents

download.file(url, dest="dataset.zip", mode="wb")
unzip ("dataset.zip", exdir = "./")
# assess the files contained in the .zip file which

# unzips as a folder named "diary14"
list.files("diary14")
## [1] "dtbd141.csv" "dtbd142.csv" "dtbd143.csv" "dtbd144.csv" "dtid141.csv"
## [6] "dtid142.csv" "dtid143.csv" "dtid144.csv" "expd141.csv" "expd142.csv"
## [11] "expd143.csv" "expd144.csv" "fmld141.csv" "fmld142.csv" "fmld143.csv"
## [16] "fmld144.csv" "memd141.csv" "memd142.csv" "memd143.csv" "memd144.csv"
# alternatively, if we know the file we want prior to unzipping

# we can extract the file without unzipping using unz():
zip_data <- read.csv(unz("dataset.zip", "diary14/expd141.csv"))
zip_data[1:5, 1:10]
## NEWID ALLOC COST GIFT PUB_FLAG UCC EXPNSQDY EXPN_QDY EXPNWKDY EXPN_KDY
## 1 2825371 0 6.26 2 2 190112 1 D 3 D
## 2 2825371 0 1.20 2 2 190322 1 D 3 D
## 3 2825381 0 0.98 2 2 20510 3 D 2 D
## 4 2825381 0 0.98 2 2 20510 3 D 2 D
## 5 2825381 0 2.50 2 2 20510 3 D 2 D
The .zip archive le format is meant to compress les and are typically used on
les of signicant size. For instance, the Consumer Expenditure Survey data we
downloaded in the previous example is over 10 MB. Obviously there may be times
in which we want to get specic data in the .zip le to analyze but not always per-
manently store the entire .zip le contents. In these instances we can use the follow-
ing process proposed by Dirk Eddelbuettel to temporarily download the .zip le,
extract the desired data, and then discard the .zip le.
# Create a temp. file name
temp <- tempfile()
# Use download.file() to fetch the file into the temp. file

download.file("http://www.bls.gov/cex/pumd/data/comma/diary14.zip",temp)
# Use unz() to extract the target file from temp. file

zip_data2 <- read.csv(unz(temp, "diary14/expd141.csv"))
# Remove the temp file via unlink()

unlink(temp)
zip_data2[1:5, 1:10]
## NEWID ALLOC COST GIFT PUB_FLAG UCC EXPNSQDY EXPN_QDY EXPNWKDY EXPN_KDY
## 1 2825371 0 6.26 2 2 190112 1 D 3 D
## 2 2825371 0 1.20 2 2 190322 1 D 3 D
## 3 2825381 0 0.98 2 2 20510 3 D 2 D
## 4 2825381 0 0.98 2 2 20510 3 D 2 D
## 5 2825381 0 2.50 2 2 20510 3 D 2 D
One last common scenario Ill cover when importing spreadsheet data from
online is when we identify multiple data sets that wed like to download but are not
centrally stored in a .zip format or the like. As a simple example lets look at the
average consumer price data from the BLS.3 The BLS holds multiple data sets for
different types of commodities within one url; however, there are separate links for
each individual data set.4 More complicated cases of this will have the links to tabu-
lar data sets scattered throughout a webpage.5 The XML package provides the use-
ful getHTMLLinks() function to identify these links.
library(XML)
# url hosting multiple links to data sets

url <- "http://download.bls.gov/pub/time.series/ap/"
# identify the links available

links <- getHTMLLinks(url)
links
## [1] "/pub/time.series/"
## [2] "/pub/time.series/ap/ap.area"
## [3] "/pub/time.series/ap/ap.contacts"
## [4] "/pub/time.series/ap/ap.data.0.Current"
## [5] "/pub/time.series/ap/ap.data.1.HouseholdFuels"
## [6] "/pub/time.series/ap/ap.data.2.Gasoline"
## [7] "/pub/time.series/ap/ap.data.3.Food"
## [8] "/pub/time.series/ap/ap.footnote"
## [9] "/pub/time.series/ap/ap.item"
## [10] "/pub/time.series/ap/ap.period"
## [11] "/pub/time.series/ap/ap.series"
## [12] "/pub/time.series/ap/ap.txt"
This allows us to assess which les exist that may be of interest. In this case the
links that we are primarily interested in are the ones that contain data in their
name (links 47 listed above). We can use the stringr package to extract these
desired links which we will use to download the data.
library(stringr)
# extract names for desired links and paste to url

links_data <- links[str_detect(links, "data")]
# paste url to data links to have full url for data sets
# use str_sub and regexpr to paste links at appropriate
# starting point
filenames <- paste0(url, str_sub(links_data,
start = regexpr("ap.data", links_data)))
3
http://www.bls.gov/data/#prices
4
http://download.bls.gov/pub/time.series/ap/
5
An example is provided in Automated Data Collection with R in which they use a similar
approach to extract desired CSV les scattered throughout the Maryland State Board of Elections
websiteMaryland State Board of Elections website.
16.1 Importing Tabular and Excel Files Stored Online 133
filenames
## [1] "http://download.bls.gov/pub/time.series/ap/ap.data.0.Current"
##[2]"http://download.bls.gov/pub/time.series/ap/ap.data.1.HouseholdFuels"
## [3] "http://download.bls.gov/pub/time.series/ap/ap.data.2.Gasoline"
## [4] "http://download.bls.gov/pub/time.series/ap/ap.data.3.Food"
We can now proceed to develop a simple for loop function (which you will learn
about in the loop control statements chapter) to download each data set. We store the
results in a list which contains 4 items, one item for each data set. Each list item
contains the url in which the data was extracted from and the dataframe containing
the downloaded data. Were now ready to analyze these data sets as necessary.
# create empty list to dump data into

data_ls <- list()
for(i in 1:length(filenames)){
url <- filenames[i]
data <- read.delim(url)
data_ls[[length(data_ls) + 1]] <- list(url = filenames[i], data = data)
}
str(data_ls)
## List of 4
## $ :List of 2
## ..$ url : chr "http://download.bls.gov/pub/time.series/ap/ap.data.0.Current"
## ..$ data:'data.frame': 144712 obs. of 5 variables:
## .. ..$ series_id : Factor w/ 878 levels "APU0000701111 ",..: 1 1
## .. ..$ year : int [1:144712] 1995 1995 1995 1995 1995 1995
## .. ..$ period : Factor w/ 12 levels "M01","M02","M03",..: 1 2 3 4
## .. ..$ value : num [1:144712] 0.238 0.242 0.242 0.236 0.244
## .. ..$ footnote_codes: logi [1:144712] NA NA NA NA NA NA
## $ :List of 2
## ..$ url : chr "http://download.bls.gov/pub/time.series/ap/ap.data.1.Hou"
## .. ..$ year : int [1:90339] 1978 1978 1979 1979 1979 1979 1979
## .. ..$ period : Factor w/ 12 levels "M01","M02","M03",..: 11 12
## .. ..$ value : num [1:90339] 0.533 0.545 0.555 0.577 0.605 0.627
## $ :List of 2
## ..$ url : chr "http://download.bls.gov/pub/time.series/ap/ap.data.2.Gas"
## .. ..$ year : int [1:69357] 1973 1973 1973 1974 1974 1974 1974
## .. ..$ period : Factor w/ 12 levels "M01","M02","M03",..: 10 11
## .. ..$ value : num [1:69357] 0.402 0.418 0.437 0.465 0.491 0.528
## $ :List of 2
## ..$ url : chr "http://download.bls.gov/pub/time.series/ap/ap.data.3.Food"
## .. ..$ year : int [1:122302] 1980 1980 1980 1980 1980 1980 1980
## .. ..$ period : Factor w/ 12 levels "M01","M02","M03",..: 1 2 3 4
## .. ..$ value : num [1:122302] 0.203 0.205 0.211 0.206 0.207 0.21
These examples provide the basics required for downloading most tabular and
Excel les from online. However, this is just the beginning of importing/scraping
data from the web. Next, well start exploring the more conventional forms of scrap-
ing text and data stored in HTML webpages.
16.2 Scraping HTML Text
Vast amount of information exists across the interminable online webpages. Much
of this information are unstructured text that may be useful in our analyses. This
section covers the basics of scraping these texts from online sources. Throughout
this section I will illustrate how to extract different text components of webpages by
dissecting the Wikipedia page on web scraping. However, its important to rst cover
one of the basic components of HTML elements as we will leverage this informa-
tion to pull desired information. I offer only enough insight required to begin scrap-
ing; I highly recommend XML and Web Technologies for Data Sciences with R and
Automated Data Collection with R to learn more about HTML and XML element
structures.
HTML elements are written with a start tag, an end tag, and with the content in
between: <tagname>content</tagname>. The tags which typically contain
the textual content we wish to scrape, and the tags we will leverage in the next two
sections, include:
<h1>, <h2>,,<h6>: Largest heading, second largest heading, etc.
<p>: Paragraph elements
<ul>: Unordered bulleted list
<ol>: Ordered list
<li>: Individual list item
<div>: Division or section
<table>: Table
For example, text in paragraph form that you see online are wrapped with the
HTML paragraph tag <p> as in:
<p>
This paragraph represents
a typical text paragraph
in HTML form
</p>
It is through these tags that we can start to extract textual components (also
referred to as nodes) of HTML webpages.
16.2 Scraping HTML Text 135
16.2.1 Scraping HTML Nodes
To scrape online text well make use of the relatively newer rvest package. rvest
was created by the RStudio team inspired by libraries such as beautiful soup which
has greatly simplied web scraping. rvest provides multiple functionalities; how-
ever, in this section we will focus only on extracting HTML text with rvest. Its
important to note that rvest makes use of the pipe operator (%>%) developed
through the magrittr package. If you are not familiar with the functionality of %>%
I recommend you jump to the chapter on Simplifying Your Code with %>% so that
you have a better understanding of whats going on with the code.
To extract text from a webpage of interest, we specify what HTML elements we
want to select by using html_nodes(). For instance, if we want to scrape the
primary heading for the Web Scraping Wikipedia webpage we simply identify the
<h1> node as the node we want to select. html_nodes() will identify all <h1>
nodes on the webpage and return the HTML element. In our example we see there
is only one <h1> node on this webpage.
library(rvest)
scraping_wiki <- read_html("https://en.wikipedia.org/wiki/Web_scraping")
scraping_wiki %>%
html_nodes("h1")
## {xml:nodeset (1)}
## [1] <h1 id="firstHeading" class="firstHeading" lang="en">Web scraping</h1>
To extract only the heading text for this <h1> node, and not include all the
HTML syntax we use html_text() which returns the heading text we see at the
top of the Web Scraping Wikipedia page.
scraping_wiki %>%
html_nodes("h1") %>%
html_text()
## [1] "Web scraping"
If we want to identify all the second level headings on the webpage we follow the
same process but instead select the <h2> nodes. In this example we see there are
ten second level headings on the Web Scraping Wikipedia page.
scraping_wiki %>%
html_nodes("h2") %>%
html_text()
## [1] "Contents"
## [2] "Techniques[edit]"
## [3] "Legal issues[edit]"
## [4] "Notable tools[edit]"
## [5] "See also[edit]"
## [6] "Technical measures to stop bots[edit]"
## [7] "Articles[edit]"
## [8] "References[edit]"
## [9] "See also[edit]"
## [10] "Navigation menu"
Next, we can move on to extracting much of the text on this webpage which is in
paragraph form. We can follow the same process illustrated above but instead well
select all <p> nodes. This selects the 17 paragraph elements from the web page;
which we can examine by subsetting the list p_nodes to see the rst line of each
paragraph along with the HTML syntax. Just as before, to extract the text from these
nodes and coerce them to a character string we simply apply html_text().
p_nodes <- scraping_wiki %>%
html_nodes("p")
length(p_nodes)
## [1] 17
p_nodes[1:6]
## [1] <p>Web scraping (web harvesting or web data extract
## [2] <p>Web scraping is closely related to <a href="/wiki/Web_indexing" t
## [3] <p/>
## [4] <p/>
## [5] <p>Web scraping is the process of automatically collecting informati
## [6] <p>Web scraping may be against the <a href="/wiki/Terms_of_use" titl
p_text <- scraping_wiki %>%

html_nodes("p") %>%
html_text()
p_text[1]
## [1] "Web scraping (web harvesting or web data extraction) is a
computer software technique of extracting information from web-
sites. Usually, such software programs simulate human exploration
of the World Wide Web by either implementing low-level Hypertext
Transfer Protocol (HTTP), or embedding a fully-fledged web browser,
such as Mozilla Firefox."
Not too bad; however, we may not have captured all the text that we were hoping
for. Since we extracted text for all <p> nodes, we collected all identied paragraph
text; however, this does not capture the text in the bulleted lists. For example, when
you look at the Web Scraping Wikipedia page you will notice a signicant amount
of text in bulleted list format following the third paragraph under the Techniques
heading. If we look at our data well see that that the text in this list format are not
capture between the two paragraphs:
p_text[5]
## [1] "Web scraping is the process of automatically collecting
information from the World Wide Web. It is a field with active
developments sharing a common goal with the semantic web vision,
an ambitious initiative that still requires breakthroughs in text
processing, semantic understanding, artificial intelligence and
human-computer interactions. Current web scraping solutions range
from the ad-hoc, requiring human effort, to fully automated sys-
tems that are able to convert entire web sites into structured
information, with limitations."
p_text[6]
## [1] "Web scraping may be against the terms of use of some websites.
The enforceability of these terms is unclear.[4] While outright duplica-
tion of original expression will in many cases be illegal, in the United
States the courts ruled in Feist Publications v. Rural Telephone Service
that duplication of facts is allowable. U.S. courts have acknowledged
that users of \"scrapers\" or \"robots\" may be held liable for commit-
ting trespass to chattels,[5][6] which involves a computer system itself
being considered personal property upon which the user of a scraper is
trespassing. The best known of these cases, eBay v. Bidder's Edge,
resulted in an injunction ordering Bidder's Edge to stop accessing, col-
lecting, and indexing auctions from the eBay web site. This case involved
automatic placing of bids, known as auction sniping. However, in order
to succeed on a claim of trespass to chattels, the plaintiff must dem-
onstrate that the defendant intentionally and without authorization
interfered with the plaintiff's possessory interest in the computer sys-
tem and that the defendant's unauthorized use caused damage to the plain-
tiff. Not all cases of web spidering brought before the courts have been
considered trespass to chattels.[7]"
This is because the text in this list format are contained in <ul> nodes. To capture
the text in lists, we can use the same steps as above but we select specic nodes which
represent HTML lists components. We can approach extracting list text two ways.
First, we can pull all list elements (<ul>). When scraping all <ul> text, the
resulting data structure will be a character string vector with each element repre-
senting a single list consisting of all list items in that list. In our running example
there are 21 list elements as shown in the example that follows. You can see the rst
list scraped is the table of contents and the second list scraped is the list in the
Techniques section.
ul_text <- scraping_wiki %>%
html_nodes("ul") %>%
html_text()
length(ul_text)
## [1] 21
ul_text[1]
## [1] "\n1 Techniques\n2 Legal issues\n3 Notable tools\n4 See
also\n5 Technical measures to stop bots\n6 Articles\n7 References\
n8 See also\n"
# read the first 200 characters of the second list

substr(ul_text[2], start = 1, stop = 200)
## [1] "\nHuman copy-and-paste: Sometimes even the best web-
scraping technology cannot replace a humans manual examination
and copy-and-paste, and sometimes this may be the only workable
solution when the web"
An alternative approach is to pull all <li> nodes. This will pull the text con-
tained in each list item for all the lists. In our running example theres 146 list items
that we can extract from this Wikipedia page. The rst eight list items are the list of
contents we see towards the top of the page. List items 917 are the list elements
contained in the Techniques section, list items 1844 are the items listed under the
Notable Tools section, and so on.
li_text <- scraping_wiki %>%
html_nodes("li") %>%
html_text()
length(li_text)
## [1] 147
li_text[1:8]
## [1] "1 Techniques" "2 Legal issues"
## [3] "3 Notable tools" "4 See also"
## [5] "5 Technical measures to stop bots" "6 Articles"
## [7] "7 References" "8 See also"
At this point we may believe we have all the text desired and proceed with join-
ing the paragraph (p_text) and list (ul_text or li_text) character strings
and then perform the desired textual analysis. However, we may now have captured
more text than we were hoping for. For example, by scraping all lists we are also
capturing the listed links in the left margin of the webpage. If we look at the 104
136 list items that we scraped, well see that these texts correspond to the left mar-
gin text.
li_text[104:136]
## [1] "Main page" "Contents" "Featured content"
## [4] "Current events" "Random article" "Donate to Wikipedia"
## [7] "Wikipedia store" "Help" "About Wikipedia"
## [10] "Community portal" "Recent changes" "Contact page"
## [13] "What links here" "Related changes" "Upload file"
## [16] "Special pages" "Permanent link" "Page information"
## [19] "Wikidata item" "Cite this page" "Create a book"
## [22] "Download as PDF" "Printable version" "Catal"
## [25] "Deutsch" "Espaol" "Franais"
## [28] "slenska" "Italiano" "Latvieu"
## [31] "Nederlands" "" "Cpc / srpski"
If we desire to scrape every piece of text on the webpage than this wont be of
concern. In fact, if we want to scrape all the text regardless of the content they rep-
resent there is an easier approach. We can capture all the content to include text in
paragraph (<p>), lists (<ul>, <ol>, and <li>), and even data in tables (<table>)
by using <div>. This is because these other elements are usually a subsidiary of an
HTML division or section so pulling all <div> nodes will extract all text contained
in that division or section regardless if it is also contained in a paragraph or list.
all_text <- scraping_wiki %>%

html_nodes("div") %>%
html_text()
16.2.2 Scraping Specific HTML Nodes
However, if we are concerned only with specic content on the webpage then we
need to make our HTML node selection process a little more focused. To do this we,
we can use our browsers developer tools to examine the webpage we are scraping
and get more details on specic nodes of interest. If you are using Chrome or Firefox
you can open the developer tools by clicking F12 (Cmd + Opt + I for Mac) or for
Safari you would use Command-Option-I. An additional option which is recom-
mended by Hadley Wickham is to use selectorgadget.com, a Chrome extension, to
help identify the web page elements you need.6
Once the developers tools are opened your primary concern is with the element
selector. This is located in the top lefthand corner of the developers tools window.
Developer Tools: Element Selector
Once youve selected the element selector you can now scroll over the elements
of the webpage which will cause each element you scroll over to be highlighted.
Once youve identied the element you want to focus on, select it. This will cause
the element to be identied in the developer tools window. For example, if I am only
interested in the main body of the Web Scraping content on the Wikipedia page then
I would select the element that highlights the entire center component of the web-
page. This highlights the corresponding element <div id="bodyContent"
class="mw-body-content"> in the developer tools window as the following
illustrates.
6
You can learn more about selectors at ukeout.github.io
Selecting Content of Interest
I can now use this information to select and scrape all the text from this specic
<div> node by calling the ID name (#mw-content-text) in html_nodes().7
As you can see below, the text that is scraped begins with the rst line in the main
body of the Web Scraping content and ends with the text in the See Also section
which is the last bit of text directly pertaining to Web Scraping on the webpage.
Explicitly, we have pulled the specic text associated with the web content we desire.
body_text <- scraping_wiki %>%
html_nodes("#mw-content-text") %>%
html_text()
# read the first 207 characters

substr(body_text, start = 1, stop = 207)
## [1] "Web scraping (web harvesting or web data extraction) is a
computer software technique of extracting information from web-
sites. Usually, such software programs simulate human exploration
of the World Wide Web"
# read the last 73 characters

substr(body_text, start = nchar(body_text)-73, stop = nchar(body_text))
## [1] "See also[edit]\n\nData scraping\nData wrangling\nKnowledge
extraction\n\n\n\n\n\n\n\n\n"
7
You can simply assess the name of the ID in the highlighted element or you can right click the
highlighted element in the developer tools window and select Copy selector. You can then paste
directly into `html_nodes() as it will paste the exact ID name that you need for that element.
Using the developer tools approach allows us to be as specic as we desire.

We can identify the class name for a specic HTML element and scrape the text for
only that node rather than all the other elements with similar tags. This allows us to
scrape the main body of content as we just illustrated or we can also identify specic
headings, paragraphs, lists, and list components if we desire to scrape only these
specic pieces of text:
# Scraping a specific heading
scraping_wiki %>%
html_nodes("#Techniques") %>%
html_text()
## [1] "Techniques"
# Scraping a specific paragraph

scraping_wiki %>%
html_nodes("#mw-content-text > p:nth-child(20)") %>%
html_text()
## [1] "In Australia, the Spam Act 2003 outlaws some forms of web harvesting,
although this only applies to email addresses.[20][21]"
# Scraping a specific list

scraping_wiki %>%
html_nodes("#mw-content-text > div:nth-child(22)") %>%
html_text()
## [1] "\n\nApache Camel\nArchive.is\nAutomation Anywhere\nCon-
vertigo\ncURL\nData Toolbar\nDiffbot\nFirebug\nGreasemonkey\nHer-
itrix\nHtmlUnit\nHTTrack\niMacros\nImport.io\nJaxer\nNode.js\
nnokogiri\nPhantomJS\nScraperWiki\nScrapy\nSelenium\nSimpleTest\n
watir\nWget\nWireshark\nWSO2 Mashup Server\nYahoo! Query Language
(YQL)\n\n"
# Scraping a specific reference list item

scraping_wiki %>%
html_nodes("#cite_note-22") %>%
html_text()
## [1] "^ \"Web Scraping: Everything You Wanted to Know (but were afraid to ask)\".
Distil Networks. 2015-07-22. Retrieved 2015-11-04."
16.2.3 Cleaning Up
With any webscraping activity, especially involving text, there is likely to be some
clean up involved. For example, in the previous example we saw that we can spe-
cically pull the list of Notable Tools; however, you can see that in between each
list item rather than a space there contains one or more \n which is used in HTML
to specify a new line. We can clean this up quickly with a little character string
manipulation.
library(magrittr)
scraping_wiki %>%
html_text()
## [1] "\n\nApache Camel\nArchive.is\nAutomation Anywhere\nConvertigo\ncURL\nData
Toolbar\nDiffbot\nFirebug\nGreasemonkey\nHeritrix\nHtmlUnit\nHTTrack\niMacros\nImport.
io\nJaxer\nNode.js\nnokogiri\nPhantomJS\nScraperWiki\nScrapy\nSelenium\nSimpleTest\n
watir\nWget\nWireshark\nWSO2 Mashup Server\nYahoo! Query Language (YQL)\n\n"
scraping_wiki %>%
html_text() %>%
strsplit(split = "\n") %>%
unlist() %>%
.[. != ""]
## [1] "Apache Camel" "Archive.is"
## [3] "Automation Anywhere" "Convertigo"
## [5] "cURL" "Data Toolbar"
## [7] "Diffbot" "Firebug"
## [9] "Greasemonkey" "Heritrix"
## [11] "HtmlUnit" "HTTrack"
## [13] "iMacros" "Import.io"
## [15] "Jaxer" "Node.js"
## [17] "nokogiri" "PhantomJS"
## [19] "ScraperWiki" "Scrapy"
## [21] "Selenium" "SimpleTest"
## [23] "watir" "Wget"
## [25] "Wireshark" "WSO2 Mashup Server"
## [27] "Yahoo! Query Language (YQL)"
Similarly, as we saw in our example above with scraping the main body content
(body_text), there are extra characters (i.e. \n, \, ^) in the text that we may not
want. Using a little regex we can clean this up so that our character string consists
of only text that we see on the screen and no additional HTML code embedded
throughout the text.
library(stringr)
# read the last 700 characters

substr(body_text, start = nchar(body_text)-700, stop = nchar(body_text))
## [1] " 2010). \"Intellectual Property: Website Terms of Use\". Issue 26: June 2010.
LK Shields Solicitors Update. p. 03. Retrieved 2012-04-19.\n^ National Office for
the Information Economy (February 2004). \"Spam Act 2003: An overview for business\"
(PDF). Australian Communications Authority. p. 6. Retrieved 2009-03-09.\n^ National
Office for the Information Economy (February 2004). \"Spam Act 2003: A practical
guide for business\" (PDF). Australian Communications Authority. p. 20. Retrieved
2009-03-09.\n^ \"Web Scraping: Everything You Wanted to Know (but were afraid to
ask)\". Distil Networks. 2015-07-22. Retrieved 2015-11-04.\n\n\nSee
also[edit]\n\nData scraping\nData wrangling\nKnowledge extraction\n\n\n\n\n\n\n\n\n"
16.3 Scraping HTML Table Data 143
# clean up text
body_text %>%
str_replace_all(pattern = "\n", replacement = " ") %>%
str_replace_all(pattern = "[\\^]", replacement = " ") %>%
str_replace_all(pattern = "\"", replacement = " ") %>%
str_replace_all(pattern = "\\s+", replacement = " ") %>%
str_trim(side = "both") %>%
substr(start = nchar(body_text)-700, stop = nchar(body_text))

## [1] "012-04-19. National Office for the Information Economy (February 2004). Spam
Act 2003: An overview for business (PDF). Australian Communications Authority. p. 6.
Retrieved 2009-03-09. National Office for the Information Economy (February 2004).
Spam Act 2003: A practical guide for business (PDF). Australian Communications
Authority. p. 20. Retrieved 2009-03-09. Web Scraping: Everything You Wanted to Know
(but were afraid to ask) . Distil Networks. 2015-07-22. Retrieved 2015-11-04. See
also[edit] Data scraping Data wrangling Knowledge extraction"
So there we have it, text scraping in a nutshell. Although not all encompassing,
this section covered the basics of scraping text from HTML documents. Whether
you want to scrape text from all common text-containing nodes such as <div>,
<p>, <ul> and the like or you want to scrape from a specic node using the specic
ID, this section provides you the basic fundamentals of using rvest to scrape the
text you need. In the next section we move on to scraping data from HTML tables.
16.3 Scraping HTML Table Data
Another common structure of information storage on the Web is in the form of HTML
tables. This section reiterates some of the information from the previous section;
however, we focus solely on scraping data from HTML tables. The simplest approach
to scraping HTML table data directly into R is by using either the rvest package or the
XML package. To illustrate, I will focus on the BLS employment statistics webpage
which contains multiple HTML tables from which we can scrape data.
16.3.1 Scraping HTML Tables with rvest
Recall that HTML elements are written with a start tag, an end tag, and with the
content in between: <tagname>content</tagname>. HTML tables are con-
tained within <table> tags; therefore, to extract the tables from the BLS employ-
ment statistics webpage we rst use the html_nodes() function to select the
<table> nodes. In this case we are interested in all table nodes that exist on the
webpage. In this example, html_nodes captures 15 HTML tables. This includes
data from the 10 data tables seen on the webpage but also includes data from a few
additional tables used to format parts of the page (i.e. table of contents, table of
gures, advertisements).
library(rvest)
webpage <- read_html("http://www.bls.gov/web/empsit/cesbmart.htm")
tbls <- html_nodes(webpage, "table")
head(tbls)
## [1] <table id="main-content-table"> \n\t<tr> \n\t\t<td id="secon
## [2] <table id="Table1" class="regular" cellspacing="0" cellpadding="0" x
## [6] <table id="Exhibit1" class="regular" cellspacing="0" cellpadding="0"
Remember that html_nodes() does not parse the data; rather, it acts as a CSS
selector. To parse the HTML table data we use html_table(), which would cre-
ate a list containing 15 data frames. However, rarely do we need to scrape every
HTML table from a page, especially since some HTML tables dont catch any infor-
mation we are likely interested in (i.e. table of contents, table of gures, footers).
More often than not we want to parse specic tables. Lets assume we want to
parse the second and third tables on the webpage:
Table 2. Nonfarm employment benchmarks by industry, March 2014 (in thou-
sands) and
Table 3. Net birth/death estimates by industry supersector, AprilDecember 2014
(in thousands)
This can be accomplished two ways. First, we can assess the previous tbls list
and try to identify the table(s) of interest. In this example it appears that tbls list
items 3 and 4 correspond with Table 2 and Table 3, respectively. We can then subset
the list of table nodes prior to parsing the data with html_table(). This results
in a list of two data frames containing the data of interest.
# subset list of table nodes for items 3 & 4
tbls_ls <- webpage %>%
html_nodes("table") %>%
.[3:4] %>%
html_table(fill = TRUE)
str(tbls_ls)
## List of 2
## $ :'data.frame': 147 obs. of 6 variables:
## ..$ CES Industry Code : chr [1:147] "Amount" "00-000000" "05-000000"
## ..$ CES Industry Title: chr [1:147] "Percent" "Total nonfarm"
## ..$ Benchmark : chr [1:147] NA "137,214" "114,989" "18,675"
## ..$ Estimate : chr [1:147] NA "137,147" "114,884" "18,558"
## ..$ Differences : num [1:147] NA 67 105 117 -50 -12 -16 -2.8
## ..$ NA : chr [1:147] NA "(1)" "0.1" "0.6"
## $ :'data.frame': 11 obs. of 12 variables:
## ..$ CES Industry Code : chr [1:11] "10-000000" "20-000000" "30-000000"
## ..$ CES Industry Title: chr [1:11] "Mining and logging" "Construction"
## ..$ Apr : int [1:11] 2 35 0 21 0 8 81 22 82 12
## ..$ May : int [1:11] 2 37 6 24 5 8 22 13 81 6
## ..$ Jun : int [1:11] 2 24 4 12 0 4 5 -14 86 6
## ..$ Jul : int [1:11] 2 12 -3 7 -1 3 35 7 62 -2
## ..$ Aug : int [1:11] 1 12 4 14 3 4 19 21 23 3
## ..$ Sep : int [1:11] 1 7 1 9 -1 -1 -12 12 -33 -2
## ..$ Oct : int [1:11] 1 12 3 28 6 16 76 35 -17 4
## ..$ Nov : int [1:11] 1 -10 2 10 3 3 14 14 -22 1
## ..$ Dec : int [1:11] 0 -21 0 4 0 10 -10 -3 4 1
## ..$ CumulativeTotal : int [1:11] 12 108 17 129 15 55 230 107 266 29
An alternative approach, which is more explicit, is to use the element selector

process described in the previous section to call the table ID name.
# empty list to add table data to

tbls2_ls <- list()
# scrape Table 2. Nonfarm employment

tbls2_ls$Table1 <- webpage %>%
html_nodes("#Table2") %>%
html_table(fill = TRUE) %>%
.[[1]]
# Table 3. Net birth/death

tbls2_ls$Table2 <- webpage %>%
html_nodes("#Table3") %>%
html_table() %>%
.[[1]]
str(tbls2_ls)
## List of 2
## $ Table1:'data.frame': 147 obs. of 6 variables:
## ..$ CES Industry Code : chr [1:147] "Amount" "00-000000" "05-000000"
## ..$ CES Industry Title: chr [1:147] "Percent" "Total nonfarm"
## ..$ Benchmark : chr [1:147] NA "137,214" "114,989" "18,675"
## ..$ Estimate : chr [1:147] NA "137,147" "114,884" "18,558"
## ..$ Differences : num [1:147] NA 67 105 117 -50 -12 -16 -2.8
## ..$ NA : chr [1:147] NA "(1)" "0.1" "0.6"
## ..$ CES Industry Code : chr [1:11] "10-000000" "20-000000" "30-000000"
## ..$ CES Industry Title: chr [1:11] "Mining and logging" "Construction"
## ..$ Apr : int [1:11] 2 35 0 21 0 8 81 22 82 12
## ..$ May : int [1:11] 2 37 6 24 5 8 22 13 81 6
## ..$ Jun : int [1:11] 2 24 4 12 0 4 5 -14 86 6
## ..$ Jul : int [1:11] 2 12 -3 7 -1 3 35 7 62 -2
## ..$ Aug : int [1:11] 1 12 4 14 3 4 19 21 23 3
## ..$ Sep : int [1:11] 1 7 1 9 -1 -1 -12 12 -33 -2
## ..$ Oct : int [1:11] 1 12 3 28 6 16 76 35 -17 4
## ..$ Nov : int [1:11] 1 -10 2 10 3 3 14 14 -22 1
## ..$ Dec : int [1:11] 0 -21 0 4 0 10 -10 -3 4 1
## ..$ CumulativeTotal : int [1:11] 12 108 17 129 15 55 230 107 266 29
One issue to note is when using rvests html_table() to read a table with
split column headings as in Table 2. Nonfarm employment. html_table will
cause split headings to be included and can cause the rst row to include parts of the
headings. We can see this with Table 2. This requires a little clean up.
head(tbls2_ls[[1]], 4)
## CES Industry Code CES Industry Title Benchmark Estimate Differences NA
## 1 Amount Percent <NA> <NA> NA <NA>
## 2 00-000000 Total nonfarm 137,214 137,147 67 (1)
## 3 05-000000 Total private 114,989 114,884 105 0.1
## 4 06-000000 Goods-producing 18,675 18,558 117 0.6
# remove row 1 that includes part of the headings

tbls2_ls[[1]] <- tbls2_ls[[1]][-1,]
# rename table headings

colnames(tbls2_ls[[1]]) <- c("CES_Code", "Ind_Title", "Benchmark",
"Estimate", "Amt_Diff", "Pct_Diff")
head(tbls2_ls[[1]], 4)
## CES_Code Ind_Title Benchmark Estimate Amt_Diff Pct_Diff
## 2 00-000000 Total nonfarm 137,214 137,147 67 (1)
## 3 05-000000 Total private 114,989 114,884 105 0.1
## 4 06-000000 Goods-producing 18,675 18,558 117 0.6
## 5 07-000000 Service-providing 118,539 118,589 -50 (1)
16.3.2 Scraping HTML Tables with XML
An alternative to rvest for table scraping is to use the XML package. The XML
package provides a convenient readHTMLTable() function to extract data from
HTML tables in HTML documents. By passing the URL to readHTMLTable(),
the data in each table is read and stored as a data frame. In a situation like our run-
ning example where multiple tables exists, the data frames will be stored in a list
similar to rvests html_table.
library(XML)
url <- "http://www.bls.gov/web/empsit/cesbmart.htm"
# read in HTML data

tbls_xml <- readHTMLTable(url)
typeof(tbls_xml)
## [1] "list"
length(tbls_xml)
## [1] 15
You can see that tbls_xml captures the same 15 <table> nodes that html_
nodes captured. To capture the same tables of interest we previously discussed
(Table 2. Nonfarm employment and Table 3. Net birth/death) we can use a cou-
ple approaches. First, we can assess str(tbls_xml) to identify the tables of
interest and perform normal list subsetting. In our example list items 3 and 4 cor-
respond with our tables of interest.
head(tbls_xml[[3]])
## V1 V2 V3 V4 V5 V6
## 1 00-000000 Total nonfarm 137,214 137,147 67 (1)
## 2 05-000000 Total private 114,989 114,884 105 0.1
## 3 06-000000 Goods-producing 18,675 18,558 117 0.6
## 5 08-000000 Private service-providing 96,314 96,326 -12 (1)
## 6 10-000000 Mining and logging 868 884 -16 -1.8
head(tbls_xml[[4]], 3)
## CES Industry Code CES Industry Title Apr May Jun Jul Aug Sep Oct Nov Dec
## 1 10-000000 Mining and logging 2 2 2 2 1 1 1 1 0
## 2 20-000000 Construction 35 37 24 12 12 7 12 -10 -21
## 3 30-000000 Manufacturing 0 6 4 -3 4 1 3 2 0
## CumulativeTotal
## 1 12
## 2 108
## 3 17
Second, we can use the which argument in readHTMLTable() which

restricts the data importing to only those tables specied numerically.
# only parse the 3rd and 4th tables
emp_ls <- readHTMLTable(url, which = c(3, 4))
str(emp_ls)
## List of 2
## ..$ V1: Factor w/ 145 levels "00-000000","05-000000",..: 1 2 3 4 5 6 7 8
## ..$ V2: Factor w/ 143 levels "Accommodation",..: 130 131 52 116 102 74
## ..$ V3: Factor w/ 145 levels "1,010.3","1,048.3",..: 40 35 48 37 145 140
## ..$ V4: Factor w/ 145 levels "1,008.4","1,052.3",..: 41 34 48 36 144 142
## ..$ V5: Factor w/ 123 levels "-0.3","-0.4",..: 113 68 71 48 9 19 29 11
## ..$ V6: Factor w/ 56 levels "-0.1","-0.2",..: 30 31 36 30 30 16 28 14 29
## ..$ CES Industry Code : Factor w/ 11 levels "10-000000","20-000000",..:1
## ..$ CES Industry Title: Factor w/ 11 levels "263","Construction",..: 8 2
## ..$ Apr : Factor w/ 10 levels "0","12","2","204",..: 3 7 1
## ..$ May : Factor w/ 10 levels "129","13","2",..: 3 6 8 5 7
## ..$ Jun : Factor w/ 10 levels "-14","0","12",..: 5 6 7 3 2
## ..$ Jul : Factor w/ 10 levels "-1","-2","-3",..: 6 5 3 10
## ..$ Aug : Factor w/ 9 levels "-19","1","12",..: 2 3 9 4 8
## ..$ Sep : Factor w/ 9 levels "-1","-12","-2",..: 5 8 5 9 1
## ..$ Oct : Factor w/ 10 levels "-17","1","12",..: 2 3 6 5 9
## ..$ Nov : Factor w/ 8 levels "-10","-15","-22",..: 4 1 7 5
## ..$ Dec : Factor w/ 8 levels "-10","-21","-3",..: 4 2 4 7
## ..$ CumulativeTotal : Factor w/ 10 levels "107","108","12",..: 3 2 6 4
The third option involves explicitly naming the tables to parse. This process uses
the element selector process described in the previous section to call the table by
name.8 We use getNodeSet() to select the specied tables of interest. However,
a key difference here is rather than copying the table ID names you want to copy the
XPath. You can do this with the following: After youve highlighted the table ele-
ment of interest with the element selector, right click the highlighted element in the
developer tools window and select Copy XPath. From here we just use readHT-
MLTable() to convert to data frames and we have our desired tables.
library(RCurl)
# parse url
url_parsed <- htmlParse(getURL(url), asText = TRUE)
# select table nodes of interest

tableNodes <- getNodeSet(url_parsed, c('//*[@id="Table2"]', '//*[@
id="Table3"]'))
# convert HTML tables to data frames

bls_table2 <- readHTMLTable(tableNodes[[1]])
bls_table3 <- readHTMLTable(tableNodes[[2]])
head(bls_table2)
## V1 V2 V3 V4 V5 V6
## 1 00-000000 Total nonfarm 137,214 137,147 67 (1)
## 2 05-000000 Total private 114,989 114,884 105 0.1
## 3 06-000000 Goods-producing 18,675 18,558 117 0.6
head(bls_table3, 3)
## CES Industry Code CES Industry Title Apr May Jun Jul Aug Sep Oct Nov Dec
## 1 10-000000 Mining and logging 2 2 2 2 1 1 1 1 0
## 2 20-000000 Construction 35 37 24 12 12 7 12 -10 -21
## 3 30-000000 Manufacturing 0 6 4 -3 4 1 3 2 0
## CumulativeTotal
## 1 12
## 2 108
## 3 17
A few benets of XMLs readHTMLTable that are routinely handy include:

We can specify names for the column headings
We can specify the classes for each column
We can specify rows to skip
For instance, if you look at bls_table2 above notice that because of the split
column headings on Table 2. Nonfarm employment readHTMLTable stripped
and replaced the headings with generic names because R does not know which variable
names should align with each column. We can correct for this with the following:
8
See Sect. 16.2.2 Scraping Specic HTML Nodes for details regarding the element selector
process.
bls_table2 <- readHTMLTable(tableNodes[[1]],

header = c("CES_Code", "Ind_Title", "Benchmark",
"Estimate", "Amt_Diff", "Pct_Diff"))
head(bls_table2)
## CES_Code Ind_Title Benchmark Estimate Amt_Diff Pct_Diff
## 1 00-000000 Total nonfarm 137,214 137,147 67 (1)
## 2 05-000000 Total private 114,989 114,884 105 0.1
## 3 06-000000 Goods-producing 18,675 18,558 117 0.6
Also, for bls_table3 note that the net birth/death values parsed have been
converted to factor levels. We can use the colClasses argument to correct this.
str(bls_table3)
## $ CES Industry Code : Factor w/ 11 levels "10-000000","20-000000",..: 1 2
## $ CES Industry Title: Factor w/ 11 levels "263","Construction",..: 8 2 7
## $ Apr : Factor w/ 10 levels "0","12","2","204",..: 3 7 1 5
## $ May : Factor w/ 10 levels "129","13","2",..: 3 6 8 5 7 9
## $ Jun : Factor w/ 10 levels "-14","0","12",..: 5 6 7 3 2 7
## $ Jul : Factor w/ 10 levels "-1","-2","-3",..: 6 5 3 10 1 7
## $ Aug : Factor w/ 9 levels "-19","1","12",..: 2 3 9 4 8 9 5
## $ Sep : Factor w/ 9 levels "-1","-12","-2",..: 5 8 5 9 1 1
## $ Oct : Factor w/ 10 levels "-17","1","12",..: 2 3 6 5 9 4
## $ Nov : Factor w/ 8 levels "-10","-15","-22",..: 4 1 7 5 8
## $ Dec : Factor w/ 8 levels "-10","-21","-3",..: 4 2 4 7 4 6
## $ CumulativeTotal : Factor w/ 10 levels "107","108","12",..: 3 2 6 4 5
bls_table3 <- readHTMLTable(tableNodes[[2]],

colClasses = c("character","character",
rep("integer", 10)))
str(bls_table3)
## $ CES Industry Code : Factor w/ 11 levels "10-000000","20-
000000",..: 1 2
## $ CES Industry Title: Factor w/ 11 levels "263","Construction",..:
8 2 7
## $ Apr : int 2 35 0 21 0 8 81 22 82 12
## $ May : int 2 37 6 24 5 8 22 13 81 6
## $ Jun : int 2 24 4 12 0 4 5 -14 86 6
## $ Jul : int 2 12 -3 7 -1 3 35 7 62 -2
## $ Aug : int 1 12 4 14 3 4 19 21 23 3
## $ Sep : int 1 7 1 9 -1 -1 -12 12 -33 -2
## $ Oct : int 1 12 3 28 6 16 76 35 -17 4
## $ Nov : int 1 -10 2 10 3 3 14 14 -22 1
## $ Dec : int 0 -21 0 4 0 10 -10 -3 4 1
## $ CumulativeTotal : int 12 108 17 129 15 55 230 107 266 29
Between rvest and XML, scraping HTML tables is relatively easy once you get u-
ent with the syntax and the available options. This section covers just the basics of both
these packages to get you moving forward with scraping tables. In the next section we
move on to working with application program interfaces (APIs) to get data from the web.
16.4 Working with APIs
An application-programming interface (API) in a nutshell is a method of communi-

cation between software programs. APIs allow programs to interact and use each
others functions by acting as a middle man. Why is this useful? Lets say you want
to pull weather data from the NOAA. You have a few options:
You could query the data and download the spreadsheet or manually cut-n-paste
the desired data and then import into R. Doesnt get you any coolness points.
You could use some webscraping techniques previously covered to parse the
desired data. Golf clap. The downfall of this strategy is if NOAA changes their
website structure down the road your code will need to be adjusted.
Or, you can use the rnoaa package which allows you to send specic instructions
to the NOAA API via R, the API will then perform the action requested and
return the desired information. The benet of this strategy is if the NOAA
changes its website structure it wont impact the API data retreival structure
which means no impact to your code. Standing ovation!
Consequently, APIs provide consistency in data retrieval processes which can be
essential for recurring analyses. Luckily, the use of APIs by organizations that col-
lect data are growing exponentially. This is great for you and I as more and more
data continues to be at our nger tips. So what do you need to get started?
16.4.1 Prerequisites?
Each API is unique; however, there are a few fundamental pieces of information
youll need to work with an API. First, the reason youre using an API is to request
specic types of data from a specic data set from a specic organization. You at
least need to know a little something about each one of these:
1. The URL for the organization and data you are pulling. Most pre-built API pack-
ages already have this connection established but when using httr youll need
to specify.
2. The data set you are trying to pull from. Most organizations have numerous data
sets to peruse so you need to make yourself familiar with the names of the avail-
able data sets.
3. The data content. Youll need to specify the specic data variables you want the
API to retrieve so youll need to be familiar with, or have access to, the data library.
In addition to these key components you will also, typically, need to provide a
form of identication and/or authorization. This is done via:
1. API key (aka token). A key is used to identify the user along with track and con-
trol how the API is being used (guard against malicious use). A key is often
obtained by supplying basic information (i.e. name, email) to the organization
and in return they give you a multi-digit key.
16.4 Working with APIs 151
2. OAuth. OAuth is an authorization framework that provides credentials as proof

for access to certain information.9 Multiple forms of credentials exist and OAuth
can actually be a fairly confusing topic; however, the httr package has simpli-
ed this greatly which we demonstrate at the end of this section.
Rather than dwell on these components, theyll likely become clearer as we
progress through examples. So, lets move on to the fun stuff.
16.4.2 Existing API Packages
Like everything else you do in R, when looking to work with an API your rst ques-
tion should be Is there a package for that? R has an extensive list of packages in
which API data feeds have been hooked into R. You can nd a slew of them scat-
tered throughout the CRAN Task View: Web Technologies and Services web page,10
on the rOpenSci web page,11 and elsewhere.12
To give you a taste for how these packages typically work, Ill quickly cover
three packages:
blsAPI for pulling U.S. Bureau of Labor Statistics data
rnoaa for pulling NOAA climate data
rtimes for pulling data from multiple APIs offered by the New York Times
16.4.2.1 blsAPI
The blsAPI allows users to request data for one or multiple series through the
U.S. Bureau of Labor Statistics API. To use the blsAPI app you only need knowledge
on the data; no key or OAuth are required. I lllustrate by pulling Mass Layoff Statistics
data but you will nd all the available data sets and their series code information at
http://www.bls.gov/help/hlpforma.htm.
The key information you will be concerned about is contained in the series iden-
tier. For the Mass Layoff data the series ID code is MLUMS00NN0001003. Each
component of this series code has meaning and can be adjusted to get specic Mass
Layoff data. The BLS provides this breakdown for what each component means
along with the available list of codes for this data set.13 For instance, the S00
(MLUMS00NN0001003) component represents the division/state. S00 will pull for
all states but I could change to D30 to pull data for the Midwest or S39 to pull for
Ohio. The N0001 (MLUMS00NN0001003) component represents the industry/
demographics. N0001 pulls data for all industries but I could change to N0008 to
pull data for the food industry or C00A2 for all persons age 3044.
9
Read more about OAuth athttps://oauth.net/
10
https://cran.r-project.org/web/views/WebTechnologies.html
11
https://ropensci.org/packages/
12
http://stats.stackexchange.com/questions/12670/data-apis-feeds-available-as-packages-in-r
13
http://www.bls.gov/help/hlpforma.htm#ML
I simply call the series identier in the blsAPI() function which pulls the
JSON data object. We can then use the fromJSON() function from the rjson
package to convert to an R data object (a list in this case). You can see that the raw
data pull provides a list of 4 items. The rst three provide some metadata info (sta-
tus, response time, and message if applicable). The data we are concerned about is
in the 4th (Results$series$data) list item which contains 31 observations.
library(rjson)
library(blsAPI)
# supply series identifier to pull data (initial pull is in JSON data)

layoffs_json <- blsAPI('MLUMS00NN0001003')
# convert from JSON into R object

layoffs <- fromJSON(layoffs_json)
List of 4
$ status : chr "REQUEST_SUCCEEDED"
$ responseTime: num 38
$ message : list()
$ Results :List of 1
..$ series:List of 1
.. ..$ :List of 2
.. .. ..$ seriesID: chr "MLUMS00NN0001003"
.. .. ..$ data :List of 31
.. .. .. ..$ :List of 5
.. .. .. .. ..$ year : chr "2013"
.. .. .. .. ..$ period : chr "M05"
.. .. .. .. ..$ periodName: chr "May"
.. .. .. .. ..$ value : chr "1383"
One of the inconveniences of an API is we do not get to specify how the data we
receive is formatted. This is a minor price to pay considering all the other benets
APIs provide. Once we understand the received data format we can typically re-
format using a little list subsetting which we previously covered and looping which
well cover in a future chapter.
# create empty data frame to fill
layoff_df <- data.frame(NULL)
# extract data of interest from each nested year-month list

for(i in seq_along(layoffs$Results$series[[1]]$data)) {
df <- data.frame(layoffs$Results$series[[1]]$data[i][[1]][1:4])
layoff_df <- rbind(layoff_df, df)
}
head(layoff_df)
## year period periodName value
## 1 2013 M05 May 1383
## 2 2013 M04 April 1174
## 3 2013 M03 March 1132
## 4 2013 M02 February 960
## 5 2013 M01 January 1528
## 6 2012 M13 Annual 17080
blsAPI also allows you to pull multiple data series and has optional arguments
(i.e. start year, end year, etc.). You can see other options at help(package =
blsAPI).
16.4.2.2 rnoaa
The rnoaa package allows users to request climate data from multiple data sets
through the National Climatic Data Center API.14 Unlike blsAPI, the rnoaa app
requires you to have an API key. To request a key go to http://www.ncdc.noaa.gov/
cdo-web/token and provide your email; a key will immediately be emailed to you.
key <- "vXTdwNoAVx" # truncated
With the key in hand, we can begin pulling data. The NOAA provides a compre-
hensive metadata library to familiarize yourself with the data available. Lets start
by pulling all the available NOAA climate stations near my residence. I live in
Montgomery county Ohio so we can nd all the stations in this county by inserting
the FIPS code. Furthermore, Im interested in stations that provide data for the
GHCND data set which contains records on numerous daily variables such as
maximum and minimum temperature, total daily precipitation, snowfall, and snow
depth; however, about two thirds of the stations report precipitation only. See
?ncdc_stations for other data sets available via rnoaa.
library(rnoaa)
stations <- ncdc_stations(datasetid ='GHCND',

locationid='FIPS:39113',
token = key)
stations$data
## Source: local data frame [23 x 9]
##
## elevation mindate maxdate latitude
## (dbl) (chr) (chr) (dbl)
## 1 294.1 2009-02-09 2014-06-25 39.6314
## 2 251.8 2009-03-01 2016-01-16 39.6807
## 3 295.7 2009-03-25 2012-09-08 39.6252
## 4 298.1 2009-08-24 2012-07-20 39.8070
## 5 304.5 2010-04-02 2016-01-12 39.6949
## 6 283.5 2012-07-01 2016-01-16 39.7373
## 7 301.4 2012-07-29 2016-01-16 39.8795
## 8 317.3 2012-09-08 2016-01-12 39.8329
## 9 298.1 2012-09-07 2016-01-15 39.6247
## 10 250.5 2012-09-11 2016-01-08 39.7180
## ..
## Variables not shown: name (chr), datacoverage (dbl), id (chr),
## elevationUnit (chr), longitude (dbl)
14
http://www.ncdc.noaa.gov/cdo-web/webservices/v2
So we see that several stations are available from which to pull data. To actually
pull data from one of these stations we need the station ID. The station I want to pull
data from is the Dayton International Airport station. We can see that this station
provides data from 1948-present and I can get the station ID as illustrated. Note that
I use some dplyr for data manipulation here; we will cover dplyr in a later chap-
ter but this just illustrates the fact that we received the data via the API.
library(dplyr)
stations$data %>%
filter(name == "DAYTON INTERNATIONAL AIRPORT, OH US") %>%
select(mindate, maxdate, id)
##
## mindate maxdate id
## (chr) (chr) (chr)
## 1 1948-01-01 2016-01-15 GHCND:USW00093815
To pull all available GHCND data from this station well use ncdc(). We sim-
ply supply the data to pull, the start and end dates (ncdc restricts you to a 1 year
limit), station ID, and your key. We can see that this station provides a full range of
data types.
climate <- ncdc(datasetid='GHCND',

startdate = '2015-01-01',
enddate = '2016-01-01',
stationid='GHCND:USW00093815',
token = key)
climate$data
##
## date datatype station value fl_m fl_q
## (chr) (chr) (chr) (int) (chr) (chr)
## 1 2015-01-01T00:00:00 AWND GHCND:USW00093815 72
## 2 2015-01-01T00:00:00 PRCP GHCND:USW00093815 0
## 3 2015-01-01T00:00:00 SNOW GHCND:USW00093815 0
## 4 2015-01-01T00:00:00 SNWD GHCND:USW00093815 0
## 5 2015-01-01T00:00:00 TAVG GHCND:USW00093815 -38 H
## 6 2015-01-01T00:00:00 TMAX GHCND:USW00093815 28
## 7 2015-01-01T00:00:00 TMIN GHCND:USW00093815 -71
## 8 2015-01-01T00:00:00 WDF2 GHCND:USW00093815 240
## 9 2015-01-01T00:00:00 WDF5 GHCND:USW00093815 240
## 10 2015-01-01T00:00:00 WSF2 GHCND:USW00093815 130
## ..
## Variables not shown: fl_so (chr), fl_t (chr)
Since we recently had some snow here lets pull data on snow fall for 2015. We
adjust the limit argument (by default ncdc limits results to 25) and identify the data
type we want. By sorting we see what days experienced the greatest snowfall (dont
worry, the results are reported in mm!).
snow <- ncdc(datasetid='GHCND',

startdate = '2015-01-01',
enddate = '2015-12-31',
limit = 365,
stationid='GHCND:USW00093815',
datatypeid = 'SNOW',
token = key)
snow$data %>%
arrange(desc(value))
##
## date datatype station value fl_m fl_q
## (chr) (chr) (chr) (int) (chr) (chr)
## 1 2015-03-01T00:00:00 SNOW GHCND:USW00093815 114
## 2 2015-02-21T00:00:00 SNOW GHCND:USW00093815 109
## 3 2015-01-25T00:00:00 SNOW GHCND:USW00093815 71
## 4 2015-01-06T00:00:00 SNOW GHCND:USW00093815 66
## 5 2015-02-16T00:00:00 SNOW GHCND:USW00093815 30
## 6 2015-02-18T00:00:00 SNOW GHCND:USW00093815 25
## 7 2015-02-14T00:00:00 SNOW GHCND:USW00093815 23
## 8 2015-01-26T00:00:00 SNOW GHCND:USW00093815 20
## 9 2015-02-04T00:00:00 SNOW GHCND:USW00093815 20
## 10 2015-02-12T00:00:00 SNOW GHCND:USW00093815 20
## ..
## Variables not shown: fl_so (chr), fl_t (chr)
This is just an intro to rnoaa as the package offers a slew of data sets to pull
from and functions to apply. It even offers built in plotting functions. Use
help(package = "rnoaa") to see all that rnoaa has to offer.
16.4.2.3 rtimes
The rtimes package provides an interface to Congress, Campaign Finance, Article

Search, and Geographic APIs offered by the New York Times. The data libraries
and documentation for the several APIs available can be found at http://developer.
nytimes.com/. To use the Times API youll need to get an API key, which can also
be found at the URL just provided.
article_key <- "4f23572d8" # truncated

cfinance_key <- "ee0b7cef" # truncated
congress_key <- "57b3e8a3" # truncated
Lets start by searching NY Times articles. With the presidential elections upon
us, we can illustrate by searching the least controversial candidateDonald Trump.
We can see that there are 4566 article hits for the term Trump. We can get more
information on a particular article by subsetting.
library(rtimes)
# article search for the term 'Trump'

articles <- as_search(q = "Trump",
begin_date = "20150101",
end_date = '20160101',
key = article_key)
# summary
articles$meta
## hits time offset
## 1 4565 28 0
# pull info on 3rd article

articles$data[3]
## [[1]]
## <NYTimes article>Donald Trumps Strongest Supporters: A Certain Kind of Democrat
## Type: News
## Published: 2015-12-31T00:00:00Z
## Word count: 1469
## URL: http://www.nytimes.com/2015/12/31/upshot/donald-trumps-strongest-
supporters-a-certain-kind-of-democrat.html
## Snippet: In a survey, he also excels among low-turnout voters and among the less
affluent and the less educated, so the question is: Will they show up to vote?
We can use the campaign nance API and functions to gain some insight into
Trumps compaign income and expenditures. The only special data you need is the
FEC ID for the candidate of interest.
trump <- cf_candidate_details(campaign_cycle = 2016,

fec_id = 'P80001571',
key = cfinance_key)
# pull summary data

trump$meta
## id name party
## 1 P80001571 TRUMP, DONALD J REP
## fec_uri
## 1 http://docquery.fec.gov/cgi-bin/fecimg/?P80001571
## committee mailing_address mailing_city
## 1 /committees/C00580100.json 725 FIFTH AVENUE NEW YORK
## mailing_state mailing_zip status total_receipts
## 1 NY 10022 O 1902410.45
## total_from_individuals total_from_pacs total_contributions
## 1 92249.33 0 96298.97
## candidate_loans total_disbursements begin_cash end_cash
## 1 1804747.23 1414674.29 0 487736.16
## total_refunds debts_owed date_coverage_from date_coverage_to
## 1 0 1804747.23 2015-04-02 2015-06-30
## independent_expenditures coordinated_expenditures
## 1 1644396.8 0
rtimes also allows us to gain some insight into what our locally elected of-
cials are up to with the Congress API. First, I can get some informaton on my
Senator and then use that information to see if hes supporting my interest. For
instance, I can pull the most recent bills that he is co-sponsoring.
# pull info on OH senator
senator <- cg_memberbystatedistrict(chamber = "senate",
state = "OH",
key = congress_key)
senator$meta
## id name role gender party
## 1 B000944 Sherrod Brown Senator, 1st Class M D
## times_topics_url twitter_id youtube_id seniority
## 1 SenSherrodBrown SherrodBrownOhio 9
## next_election
## 1 2018
## api_url
## 1 http://api.nytimes.com/svc/politics/v3/us/legislative/congress/members/B000944.json
# use member ID to pull recent bill sponsorship

bills <- cg_billscosponsor(memberid = "B000944",
type = "cosponsored",
key = congress_key)
head(bills$data)
##
## congress number
## (chr) (chr)
## 1 114 S.2098
## 2 114 S.2096
## 3 114 S.2100
## 4 114 S.2090
## 5 114 S.RES.267
## 6 114 S.RES.269
## Variables not shown: bill_uri (chr), title (chr), cosponsored_date
## (chr), sponsor_id (chr), introduced_date (chr), cosponsors (chr),
## committees (chr), latest_major_action_date (chr),
## latest_major_action (chr)
It looks like the most recent bill Sherrod is co-sponsoring is S.2098Student

Right to Know Before You Go Act. Maybe Ill do a NY Times article search with
as_search() to nd out more about this billan exercise for another time.
So this gives you some avor of how these API packages work. You typically
need to know the data sets and variables requested along with an API key. But once
you get these basics its pretty straight forward on requesting the data. Your next
question may be, what if the API that I want to get data from does not yet have an R
package developed for it?
16.4.3 httr for All Things Else
Although numerous R API packages are available, and cover a wide range of data,
you may eventually run into a situation where you want to leverage an organiza-
tions API but an R package does not exist. Enter httr. httr was developed by
Hadley Wickham to easily work with web APIs. It offers multiple functions (i.e.
HEAD(), POST(), PATCH(), PUT() and DELETE()); however, the function we
are most concerned with today is Get(). We use the Get() function to access an
API, provide it some request parameters, and receive an output.
To give you a taste for how the httr package works, Ill quickly cover how to
use it for a basic key-only API and an OAuth-required API:
Key-only API is illustrated by pulling U.S. Department of Education data avail-
able on data.gov
OAuth-required API is illustrated by pulling tweets from my personal Twitter feed
16.4.3.1 Key-Only API
To demonstrate how to use the httr package for accessing a key-only API, Ill
illustrate with the College Scorecard API15 provided by the Department of Education.
First, youll need to request your API key, which can be done at https://api.data.gov/
signup/.
# truncated key
edu_key <- "fd783wmS3Z"
We can now proceed to use httr to request data from the API with the GET()
function. I went to North Dakota State University (NDSU) for my undergrad so Im
interested in pulling some data for this school. I can use the provided data library
and query explanation to determine the parameters required. In this example, the
URL includes the primary path (https://api.data.gov/ed/collegescorecard/), the
API version (v1), and the endpoint (schools). The question mark (?) at the
end of the URL is included to begin the list of query parameters, which only includes
my API key and the school of interest.
library(httr)
URL <- "https://api.data.gov/ed/collegescorecard/v1/schools?"
# import all available data for NDSU

ndsu_req <- GET(URL, query = list(api_key = edu_key,
school.name = "North Dakota State University"))
This request provides me with every piece of information collected by the

U.S. Department of Education for NDSU. To retrieve the contents of this request I
use the content() function which will output the data as an R object (a list in this
15
https://api.data.gov/docs/ed/
case). The data is segmented into two main components: metadata and results. Im
primarily interested in the results.
The results branch of this list provides information on lat-long location, school
identier codes, some basic info on the school (city, number of branches, school
website, accreditor, etc.), and then student data for the years 1997-2013.
ndsu_data <- content(ndsu_req)
names(ndsu_data)
## [1] "metadata" "results"
names(ndsu_data$results[[1]])
## [1] "2008" "2009" "2006" "ope6_id" "2007" "2004"
## [7] "2013" "2005" "location" "2002" "2003" "id"
## [13] "1996" "1997" "school" "1998" "2012" "2011"
## [19] "2010" "ope8_id" "1999" "2001" "2000"
To see what kind of student data categories are offered we can assess a single
year. You can see that available data includes earnings, academics, student info/
demographics, admissions, costs, etc. With such a large data set, which includes
many embedded lists, sometimes the easiest way to learn the data structure is to
peruse names at different levels.
# student data categories available by year
names(ndsu_data$results[[1]]$`2013`)
## [1] "earnings" "academics" "student" "admissions"
"repayment"
## [6] "aid" "cost" "completion"
# cost categories available by year

names(ndsu_data$results[[1]]$`2013`$cost)
## [1] "title_iv" "avg_net_price" "attendance" "tuition"
## [5] "net_price"
# Avg net price cost categories available by year

names(ndsu_data$results[[1]]$`2013`$cost$avg_net_price)
## [1] "other_academic_year" "overall" "program_year"
## [4] "public" "private"
So if Im interested in comparing the rise in cost versus the rise in student debt I
can simply subset for this data once Ive identied its location and naming structure.
Note that for this subsetting we use the magrittr package and the sapply function;
both we cover in later chapters but this is just meant to illustrate the types of data
available through this API.
library(magrittr)
# subset list for annual student data only

ndsu_yr <- ndsu_data$results[[1]][c(as.character(1996:2013))]
# extract median debt data for each year

ndsu_yr %>%
sapply(function(x) x$aid$median_debt$completers$overall) %>%
unlist()
## 1997 1998 1999 2000 2001 2002 2003 2004

## 13388.0 13856.0 14500.0 15125.0 15507.0 15639.0 16251.0 16642.5
## 2005 2006 2007 2008 2009 2010 2011 2012
## 17125.0 17125.0 17125.0 17250.0 19125.0 21500.0 23000.0 24954.5
## 2013
## 25050.0
# extract net price for each year

ndsu_yr %>%
sapply(function(x) x$cost$avg_net_price$overall) %>%
unlist()
## 2009 2010 2011 2012 2013
## 13474 12989 13808 15113 14404
Quite simple isnt itat least once youve learned how the query requests are
formatted for a particular API.
16.4.3.2 OAuth-Required API
At the outset I mentioned how OAuth is an authorization framework that provides

credentials as proof for access. Many APIs are open to the public and only require
an API key; however, some APIs require authorization to account data (think per-
sonal Facebook & Twitter accounts). To access these accounts we must provide
proper credentials and OAuth authentication allows us to do this. This section is not
meant to explain the details of OAuth but, rather, how to use httr in times when
OAuth is required.
Ill demonstrate by accessing the Twitter API using my Twitter account. The rst
thing we need to do is identify the OAuth endpoints used to request access and
authorization. To do this we can use oauth_endpoint() which typically
requires a request URL, authorization URL, and access URL. httr also included
some baked-in endpoints to include LinkedIn, Twitter, Vimeo, Google, Facebook,
and GitHub. We can see the Twitter endpoints using the following:
twitter_endpts <- oauth_endpoints("twitter")
twitter_endpts
## <oauth_endpoint>
## request: https://api.twitter.com/oauth/request_token
## authorize: https://api.twitter.com/oauth/authenticate
## access: https://api.twitter.com/oauth/access_token
Next, I register my application at https://apps.twitter.com/. One thing to note is

during the registration process, it will ask you for the callback url; be sure to use
http://127.0.0.1:1410. Once registered, Twitter will provide you with keys and
access tokens. The two we are concerned about are the API key and API Secret.
twitter_key <- "BZgukbCol" # truncated
twitter_secret <- "YpB8Xy" # truncated
We can then bundle the consumer key and secret into one object with oauth_app().
The rst argument, appname is simply used as a local identier; it does not need to
match the name you gave the Twitter app you developed at https://apps.twitter.com/.
We are now ready to ask for access credentials. Since Twitter uses OAuth 1.0 we
use oauth1.0_token() function and incorporate the endpoints identied and
the oauth_app object we previously named twitter_app.
twitter_token <- oauth1.0_token(endpoint = twitter_endpts, twitter_app)
Waiting for authentication in browser

Press Esc/Ctrl + C to abort
Authentication complete.
Once authentication is complete we can now use the API. I can pull all the tweets
that show up on my personal timeline using the GET() function and the access
cridentials I stored in twitter_token. I then use content() to convert to a
list and I can start to analyze the data.
In this case each tweet is saved as an individual list item and a full range of data
are provided for each tweet (i.e. id, text, user, geo location, favorite count, etc). For
instance, we can see that the rst tweet was by FiveThirtyEight concerning American
politics and, at the time of this analysis, has been favorited by 3 people.
# request Twitter data

req <- GET("https://api.twitter.com/1.1/statuses/home_timeline.json",
config(token = twitter_token))
# convert to R object
tweets <- content(req)
# available data for first tweet on my timeline

names(tweets[[1]])
[1] "created_at" "id"
[3] "id_str" "text"
[5] "source" "truncated"
[7] "in_reply_to_status_id" "in_reply_to_status_id_str"
[9] "in_reply_to_user_id" "in_reply_to_user_id_str"
[11] "in_reply_to_screen_name" "user"
[13] "geo" "coordinates"
[15] "place" "contributors"
[17] "is_quote_status" "retweet_count"
[19] "favorite_count" "entities"
[21] "extended_entities" "favorited"
[23] "retweeted" "possibly_sensitive"
[25] "possibly_sensitive_appealable" "lang"
# further analysis of first tweet on my timeline

tweets[[1]]$user$name
[1] "FiveThirtyEight"
tweets[[1]]$text
[1] "\U0001f3a7 A History Of Data In American Politics (Part 1): William Jennings
Bryan to Barack Obama https://t.co/oCKzrXuRHf https://t.co/6CvKKToxoH"
tweets[[1]]$favorite_count
[1] 3
This provides a fairly simple example of incorporating OAuth authorization. The

httr provides several examples of accessing common social network APIs that
require OAuth. I recommend you go through several of these examples to get famil-
iar with using OAuth authorization; see them at demo(package = "httr").
The most difcult aspect of creating your own connections with APIs is gaining an
understanding of the API and the arguments they leverage. This obviously requires
time and energy devoted to digging into the API documentation and data library.
Next its just a matter of trial and error (likely more the latter than the former) to
learn how to translate these arguments into httr function calls to pull the data of
interest.Also, note that httr provides several other useful functions not covered
here for communicating with APIs (i.e. POST(), BROWSE()). For more on these
other httr capabilities see the httr quickstart vignette.16
As I stated in the outset, this chapter is meant to provide an introduction to basic

web scraping capabilities in R. This area is vast and complex and this chapter will
far from provide you expertise level insight. To advance your knowledge in web-
scraping with R Automated Data Collection with R and XML and Web Technologies
for Data Sciences with R offer the most detailed resources available. But this chap-
ter should be enough to get your curiosity piqued and to start pulling data from the
tangled masses of online data.
Bibliography
Munzert, S., Rubba, C., Meiner, P., & Nyhuis, D. (2014). Automated data collection with R: A
practical guide to web scraping and text mining. John Wiley & Sons.
Nolan, D., & Lang, D. T. (2014). XML and Web Technologies for Data Sciences with R. Springer.
16
https://cran.r-project.org/web/packages/httr/vignettes/quickstart.html
Chapter 17
Exporting Data
Although getting data into R is essential, getting data out of R can be just as important.
Whether you need to export data or analytic results simply to store, share, or feed into
another system it is generally a straight forward process. This section will cover how to
export data to text les, Excel les (along with some additional formatting capabilities),
and save to R data objects. In addition to the commonly used base R functions to perform
data importing, I will also cover functions from the popular readr and xlsx packages
along with a lesser known but useful r2excel package for Excel formatting.
17.1 Writing Data to Text Files
As mentioned in the importing data section, text les are a popular way to hold and
exchange tabular data as almost any data application supports exporting data to the
CSV (or other text le) formats. Consequently, exporting data to a text le is a
pretty standard operation. Plus, since youve already learned how to import text les
you pretty much have the basics required to write to text les, we just use a slightly
different naming convention.
Similar to the examples provided in the importing text les section, the two main
groups of functions that I will demonstrate to write to text les include base R func-
tions and readr package functions.
17.1.1 Base R Functions
write.table() is the multipurpose work-horse function in base R for exporting

data. The functions write.csv() and write.delim() are special cases of
write.table() in which the defaults have been adjusted for efciency. To illus-
trate these functions lets work with a data frame that we wish to export to a CSV
le in our working directory.

164 17 Exporting Data
df <- data.frame(var1 = c(10, 25, 8),

var2 = c("beer", "wine", "cheese"),
var3 = c(TRUE, TRUE, FALSE),
row.names = c("billy", "bob", "thornton"))
df
## var1 var2 var3
## billy 10 beer TRUE
## bob 25 wine TRUE
## thornton 8 cheese FALSE
To export df to a CSV le we can use write.csv(). Additional arguments

allow you to exclude row and column names, specify what to use for missing values,
add or remove quotations around character strings, etc.
# write to a csv file
write.csv(df, file = "export_csv")
# write to a csv and save in a different directory

write.csv(df, file = "/folder/subfolder/subsubfolder/export_csv")
# write to a csv file with added arguments

write.csv(df, file = "export_csv", row.names = FALSE, na = "MISSING!")
In addition to CSV les, we can also write to other text les using write.
table and write.delim().
# write to a tab delimited text files
write.delim(df, file = "export_txt")
# provides same results as read.delim

write.table(df, file = "export_txt", sep="\t")
17.1.2 readr Package
The readr package uses write functions similar to base R. However, readr write
functions are about twice as fast and they do not write row names. One thing to note,
where base R write functions use the file = argument, readr write functions use
path =.
library(readr)
# write to a csv file

write_csv(df, path = "export_csv2")
# write to a csv and save in a different directory

write_csv(df, path = "/folder/subfolder/subsubfolder/export_csv2")
# write to a csv file without column names

write_csv(df, path = "export_csv2", col_names = FALSE)
17.2 Writing Data to Excel Files 165
# write to a txt file without column names

write_delim(df, path = "export_txt2", col_names = FALSE)
17.2 Writing Data to Excel Files
As previously mentioned, many organizations still rely on Excel to hold and share
data so exporting to Excel is a useful bit of knowledge. And rather than saving to a
.csv le to send to a co-worker who wants to work in Excel, its more efcient to just
save R outputs directly to an Excel workbook. Since I covered importing data with
the xlsx package, Ill also cover exporting data with this package. However, the
readxl package which I demonstrated in the importing data section does not have
a function to export to Excel. But there is a lesser known package called r2excel
that provides exporting and formatting functions for Excel which I will cover.
17.2.1 xlsx Package
Saving a data frame to a .xlsx le is as easy as saving to a .csv le:

library(xlsx)
# write to a .xlsx file

write.xlsx(df, file = "output_example.xlsx")
# write to a .xlsx file without row names

write.xlsx(df, file = "output_example.xlsx", row.names = FALSE)
In some cases you may wish to create a .xlsx le that contains multiple data
frames. In this you can just create an empty workbook and save the data frames on
separate worksheets within the same workbook:
# create empty workbook
multiple_df <- createWorkbook()
# create worksheets within workbook

car_df <- createSheet(wb = multiple_df, sheetName = "Cars")
iris_df <- createSheet(wb = multiple_df, sheetName = "Iris")
# add data frames to worksheets; for this example I use the

# built in mtcars and iris data frames
addDataFrame(x = mtcars, sheet = car_df)
addDataFrame(x = iris, sheet = iris_df)
# save as a .xlsx file

saveWorkbook(multiple_df, file = "output_example_2.xlsx")
By default this saves the row and column names but this can be adjusted by add-
ing col.names = FALSE and/or row.names = FALSE to the add-
DataFrame() function. There is also the ability to do some formatting with the
xlsx package. The following provides several examples of how you can edit titles,
subtitles, borders, column width, etc.1 Although at rst glance this can appear
tedious for simple Excel editing, the real benets present themselves when you
integrate this editing into automated analyses.
# create new workbook
wb <- createWorkbook()
#--------------------
# DEFINE CELL STYLES
#--------------------
# title and subtitle styles
title_style <- CellStyle(wb) +
Font(wb, heightInPoints = 16,
color = "blue",
isBold = TRUE,
underline = 1)
subtitle_style <- CellStyle(wb) +

Font(wb, heightInPoints = 14,
isItalic = TRUE,
isBold = FALSE)
# data table styles

rowname_style <- CellStyle(wb) +
Font(wb, isBold = TRUE)
colname_style <- CellStyle(wb) +

Font(wb, isBold = TRUE) +
Alignment(wrapText = TRUE, horizontal = "ALIGN_CENTER") +
Border(color = "black",
position = c("TOP", "BOTTOM"),
pen = c("BORDER_THIN", "BORDER_THICK"))
#-------------------------
# CREATE & EDIT WORKSHEET
#-------------------------
# create worksheet
Cars <- createSheet(wb, sheetName = "Cars")
# helper function to add titles

xlsx.addTitle <- function(sheet, rowIndex, title, titleStyle) {
rows <- createRow(sheet, rowIndex = rowIndex)
sheetTitle <- createCell(rows, colIndex = 1)
setCellValue(sheetTitle[[1,1]], title)
setCellStyle(sheetTitle[[1,1]], titleStyle)
}
1
This example was derived from http://www.sthda.com/english/ Additional options, such as add-
ing plot outputs can be found at STHDA and also in the XML and Web Technologies for Data
Sciences with R book.
17.2 Writing Data to Excel Files 167
# add title and sub title to worksheet

xlsx.addTitle(sheet = Cars, rowIndex = 1,
title = "1974 Motor Trend Car Data",
titleStyle = title_style)
xlsx.addTitle(sheet = Cars, rowIndex = 2,

title = "Performance and design attributes of 32 automobiles",
titleStyle = subtitle_style)
# add data frame to worksheet

addDataFrame(mtcars, sheet = Cars, startRow = 3, startColumn = 1,
colnamesStyle = colname_style,
rownamesStyle = rowname_style)
# change row name column width

setColumnWidth(sheet = Cars, colIndex = 1, colWidth = 18)
# save workbook
saveWorkbook(wb, file = "output_example_3.xlsx")
Formatted Excel Output Example 1
17.2.2 r2excel Package
Although Formatting Excel les using the xlsx package is possible, the last sec-
tion illustrated that it is a bit cumbersome. For this reason, A. Kassambara2 created
the r2excel package which depends on the xlsx package but provides easy to
use functions for Excel formatting. The following provides a simple example but
you can nd many additional formatting functions at http://www.sthda.com/.
# install.packages("devtools")
devtools::install_github("kassambara/r2excel")
library(r2excel)
# create new workbook

wb <- createWorkbook()
2
https://github.com/kassambara
# create worksheet
Casualties <- createSheet(wb, sheetName = "Casualties")
# add title
xlsx.addHeader(wb, sheet = Casualties,
value = "Road Casualties",
level = 1,
color = "red",
underline = 1)
# add subtitle
xlsx.addHeader(wb, sheet = Casualties,
value = "Great Britain 1969-84",
level = 2,
color = "black")
# add author information

author = paste("Author: Bradley C. Boehmke \n",
"Date: January 15, 2016 \n",
"Contact: xxxxx@gmail.com", sep = "")
xlsx.addParagraph(wb, sheet = Casualties,

value = author,
isItalic = TRUE,
colSpan = 2,
rowSpan = 4,
fontColor = "darkgray",
fontSize = 14)
# add hyperlink
xlsx.addHyperlink(wb, sheet = Casualties,
address = "http://bradleyboehmke.github.io/",
friendlyName = "Vist my website", fontSize = 12)
xlsx.addLineBreak(sheet = Casualties, 1)
# add data frame to worksheet, I'm using the built in

# Seatbelt data which you can view at data(Seatbelt)
xlsx.addTable(wb, sheet = Casualties, data = Seatbelts, startCol = 2)
# save the workbook to an Excel file

saveWorkbook(wb, file = "output_example_4.xlsx")
Formatted Excel Output Example 2

17.3 Saving Data as an R Object File
Sometimes you may need to save data or other R objects outside of your workspace.
You may want to share R data/objects with co-workers, transfer between projects or
computers, or simply archive them. There are three primary ways that people tend to
save R data/objects: as .RData, .rda, or as .rds les. .rda is just short for .RData, there-
fore, these le extensions represent the same underlying object type. You use the .rda
or .RData le types when you want to save several, or all, objects and functions that
exist in your global environment. On the other hand, if you only want to save a single
R object such as a data frame, function, or statistical model results its best to use .rds
le type. You can use .rda or .RData to save a single object but the benet of .rds is it
only saves a representation of the object and not the name whereas .rda and .RData
save both the object and its name. As a result, with .rds the saved object can be loaded
into a named object within R that is different from the name it had when originally
saved. The following illustrates how you save R objects with each type.
# save() can be used to save multiple objects in you global environment,
# in this case I save two objects to a .RData file
x <- stats::runif(20)
y <- list(a = 1, b = TRUE, c = "oops")
save(x, y, file = "xy.RData")
# save.image() s just a short-cut for saving your current

# workspace, i.e. all objects in your global environment
save.image()
# save a single object to file

saveRDS(x, "x.rds")
# restore it under a different name

x2 <- readRDS("x.rds")
identical(x, x2)
[1] TRUE
The following provides additional resources for exporting data:

R data import/export manual, which can be found at https://cran.r-project.org/
doc/manuals/R-data.html
WriteXLS package
XLConnect package
Part V
Creating Efficient and Readable Code in R
To iterate is human, to recurse divine.
L. Peter Deutsch
Dont repeat yourself (DRY) is a software development principle aimed at reducing

repetition. Formulated by Andy Hunt and Dave Thomas in their book The Pragmatic
Programmer (Hunt and Thomas, 2000), the DRY principle states that every piece
of knowledge must have a single, unambiguous, authoritative representation within
a system. This principle has been widely adopted to imply that you should not
duplicate code. Although the principle was meant to be far grander than that,1 theres
plenty of merit behind this slight misinterpretation.
Removing duplication is an important part of writing efcient code and reducing
potential errors. First, reduced duplication of code can improve computing time and
reduces the amount of code writing required. Second, less duplication results in less
creation and saving of unnecessary objects. Inefcient code invariably creates cop-
ies of objects you have little interest in other than to feed into some future line of
code; this wrecks havoc on properly managing your objects as it basically results in
a global environment charlie foxtrot! Less duplication also results in less editing.
When changes to code are required, duplicated code becomes tedious to edit and
invariably mistakes or fat-ngering occur in the cut-and-paste editing process which
just lengthens the editing that much more.
Furthermore, its important to have readable code. Clarity in your code creates
clarity in your data analysis process. This is important as data analysis is a collab-
orative process so your code will likely need to be read and interpreted by others.
Plus, invariably there will come a time where you will need to go back to an old
analysis so your code also needs to be clear to your future-self.
This section covers the process of creating efcient and readable code. First, I
cover the basics of writing your own functions so that you can reduce code duplica-
tion and automate generalized tasks to be applied recursively. I then cover loop
1
According to Dave Thomas DRY says that every piece of system knowledge should have one
authoritative, unambiguous representation. Every piece of knowledge in the development of some-
thing should have a single representation. A systems knowledge is far broader than just its code.
It refers to database schemas, test plans, the build system, even documentation.
172 Part V Creating Efcient and Readable Code in R
control statements which allow you to perform repetitive code processes with
different intentions and allow these automated expressions to naturally respond to
features of your data. Lastly, I demonstrate how you can simplify your code to make
it more readable and clear. Combined, these tools will move you forward in writing
efcient, simple, and readable code.
Bibliography
Hunt, A., & Thomas, D. (2000). The pragmatic programmer: from journeyman to master. Addison-
Wesley Professional.
Chapter 18
Functions
R is a functional programming language, meaning that everything you do is basically

built on functions. However, moving beyond simply using pre-built functions to
writing your own functions is when your capabilities really start to take off and your
code development/writing takes on a new level of efficiency. Functions allow you to
reduce code duplication by automating a generalized task to be applied recursively.
Whenever you catch yourself repeating a function or copy-and-pasting code there is
a good change that you should write a function to eliminate the redundancies.
Unfortunately, due to their abstractness, grasping the idea of writing functions
(let alone writing them well) can take some time. However, in this chapter I will
provide you with the basic knowledge of how functions operate in R to get you
started on the right path. To do this, I cover the general components of functions,
specifying function arguments, scoping and evaluation rules, managing function
outputs, handling invalid parameters, and saving and sourcing functions for reuse.
This will provide you with the required knowledge to start building your own func-
tions. Lastly, I offer some additional resources that will help you learn more about
functions in R.
18.1 Function Components
With the exception of primitive functions all R functions have three parts:
body(): the code inside the function
formals(): the list of arguments used to call the function
environment(): the mapping of the location(s) of the functions variables
For example, lets build a function that calculates the present value (PV) of a
single future sum. The equation for a single sum PV is:
PV = FV / (1 + r )n

B.C. Boehmke, Data Wrangling with R, Use R!, DOI10.1007/978-3-319-45599-0_18
174 18Functions
where FV is future value, r is the interest rate, and n is the number of periods. In the
function that follows the body of the function includes the equation
FV / (1+ r )n
and then rounding the output to two decimals. The formals (or arguments)
required for the function include FV, r, and n. And the environment shows that
function operates in the global environment.
PV <- function(FV, r, n) {
PV <- FV / (1 + r)^n
round(PV, 2)
}
body(PV)
## {
## PV <- FV / (1 + r)^n
## round(PV, 2)
## }
formals(PV)
## $FV
##
##
## $r
##
##
## $n
environment(PV)
## <environment: R_GlobalEnv>
18.2 Arguments
To perform the PV() function we can call the arguments in different ways.
# using argument names
PV(FV = 1000, r = .08, n = 5)
## [1] 680.58
# same as above but without using names (aka "positional matching")

PV(1000, .08, 5)
## [1] 680.58
# if using names you can change the order

PV(r = .08, FV = 1000, n = 5)
## [1] 680.58
# if not using names you must insert arguments in proper order

# in this e.g. the function assumes FV = .08, r = 1000, and n = 5
PV(.08, 1000, 5)
## [1] 0
18.3Scoping Rules 175
Note that when building a function you can also set default values for arguments.
In our original PV() we did not provide any default values so if we do not supply
all the argument parameters an error will be returned. However, if we set default
values then the function will use the stated default if any parameters are missing:
# missing the n argument
PV(1000, .08)
## Error in PV(1000, 0.08): argument "n" is missing, with no default
# creating default argument values

PV <- function(FV = 1000, r = .08, n = 5) {
PV <- FV / (1 + r)^n
round(PV, 2)
}
# function will use default n value

PV(1000, .08)
## [1] 680.58
# specifying a different n value

PV(1000, .08, 3)
## [1] 793.83
18.3 Scoping Rules
Scoping refers to the set of rules a programming language uses to lookup the value
for variables and/or symbols. The following illustrates the basic concept behind the
lexical scoping rules that R follows.
A function will first look inside the function to identify all the variables being
called. If all variables exist then their is no additional search required to identify
variables.
PV1 <- function() {
FV <- 1000
r <- .08
n <- 5
FV / (1 + r)^n
}
PV1()
## [1] 680.5832
However, if a variable does not exist within the function, R will look one level
up to see if the variable exists.
# the FV variable is outside the function environment
FV <- 1000
176 18Functions
PV2 <- function() {

r <- .08
n <- 5
FV / (1 + r)^n
}
PV2()
## [1] 680.5832
This same concept applies if you have functions embedded within functions:
FV <- 1000
PV3 <- function() {

r <- .08
n <- 5
denominator <- function() {
(1 + r)^n
}
FV/denominator()
}
PV3()
## [1] 680.5832
This also applies for functions in which some arguments are called but not all
variables used in the body are identified as arguments:
# n is specified within the function
PV4 <- function(FV, r) {
n <- 5
FV / (1 + r)^n
}
PV4(1000, .08)
## [1] 680.5832
# n is specified within the function and

# r is specified outside the function
r <- 0.08
PV5 <- function(FV) {

n <- 5
FV / (1 + r)^n
}
PV5(1000)
## [1] 680.5832
18.5Returning Multiple Outputs fromaFunction 177
18.4 Lazy Evaluation
R functions perform lazy evaluation in which arguments are only evaluated if

required in the body of the function.
# the y argument is not used so not including it causes
# no harm
lazy <- function(x, y){
x * 2
}
lazy(4)
## [1] 8
# however, if both arguments are required in the body

# an error will result if an argument is missing
lazy2 <- function(x, y){
(x + y) * 2
}
lazy2(4)
## Error in lazy2(4): argument "y" is missing, with no default
18.5 Returning Multiple Outputs fromaFunction
If a function performs multiple tasks and therefore has multiple results to report
then we have to include the c() function inside the function to display all the
results. If you do not include the c() function then the function output will only
return the last expression:
bad <- function(x, y) {
2 * x + y
x + 2 * y
2 * x + 2 * y
x / y
}
bad(1, 2)
## [1] 0.5
good <- function(x, y) {

output1 <- 2 * x + y
output2 <- x + 2 * y
output3 <- 2 * x + 2 * y
output4 <- x / y
c(output1, output2, output3, output4)
}
good(1, 2)
## [1] 4.0 5.0 6.0 0.5
178 18Functions
Furthermore, when we have a function which performs multiple tasks (i.e. com-
putes multiple computations) then it is often useful to save the results in a list.
good_list <- function(x, y) {
output1 <- 2 * x + y
output2 <- x + 2 * y
output3 <- 2 * x + 2 * y
output4 <- x / y
c(list(Output1 = output1, Output2 = output2,
Output3 = output3, Output4 = output4))
}
good_list(1, 2)
## $Output1
## [1] 4
##
## $Output2
## [1] 5
##
## $Output3
## [1] 6
##
## $Output4
## [1] 0.5
18.6 Dealing withInvalid Parameters
For functions that will be used again, and especially for those used by someone
other than the creator of the function, it is good to check the validity of arguments
within the function. One way to do this is to use the stop() function. The follow-
ing uses an if() statement to check if the class of each argument is numeric. If one
or more arguments are not numeric then the stop() function will be triggered to
provide a meaningful message to the user.
PV <- function(FV, r, n) {
if(!is.numeric(FV) | !is.numeric(r) | !is.numeric(n)){
stop('This function only works for numeric inputs!\n',
'You have provided objects of the following classes:\n',
'FV: ', class(FV), '\n',
'r: ', class(r), '\n',
'n: ', class(n))
}
PV <- FV / (1 + r)^n
round(PV, 2)
}
18.7Saving andSourcing Functions 179
PV("1000", 0.08, "5")

## Error in PV("1000", 0.08, "5"): This function only works for numeric inputs!
## You have provided objects of the following classes:
## FV: character
## r: numeric
## n: character
Another concern is dealing with missing or NA values. Lets say you wanted to
perform the PV() function on a vector of potential future values. The function as is
will output NA in place of any missing values in the FV input vector. If you want to
remove the missing values then you can incorporate the na.rm parameter in the
function arguments along with an if statement to remove missing values if na.rm
= TRUE.
# vector of future value inputs
fv <- c(800, 900, NA, 1100, NA)
# original PV() function will return NAs

PV(fv, .08, 5)
## [1] 544.47 612.52 NA 748.64 NA
# add na.rm argument

PV <- function(FV, r, n, na.rm = FALSE) {
if(!is.numeric(FV) | !is.numeric(r) | !is.numeric(n)){
stop('This function only works for numeric inputs!\n',
'You have provided objects of the following classes:\n',
'FV: ', class(FV), '\n',
'r: ', class(r), '\n',
'n: ', class(n))
}
if(na.rm == TRUE) {
FV <- FV[!is.na(FV)]
}
PV <- FV / (1 + r)^n
round(PV, 2)
}
# setting na.rm = TRUE argument eliminates NA outputs

PV(fv, 0.08, 5, na.rm = TRUE)
## [1] 544.47 612.52 748.64
18.7 Saving andSourcing Functions
If you want to save a function to be used at other times and within other scripts there
are two main ways to do this. One way is to build a package which I do not cover in
this book but is discussed in more details in Hadley Wickhams R Packages book,
which is openly available at http://r-pkgs.had.co.nz/. Another option, and the one
discussed here, is to save the function in a script. For example, we can save a script
that contains the PV() function and save this script as PV.R.
180 18Functions
Now, if we are working in a fresh script youll see that we have no objects and
functions in our working environment:
If we want to use the PV function in this new script we can simply read in the
function by sourcing the script using source("PV.R"). Now, youll notice that
we have the PV() function in our global environment and can use it as normal. Note
that if you are working in a different directory then where the PV.R file is located
youll need to include the proper path to access the relevant directory.
18.8Additional Resources 181
Functions are a fundamental building block of R and writing functions is a core

activity of an R programmer. It represents the key step of the transition from a mere
user to a developer who creates new functionality for R.As a result, its important
to turn your existing, informal knowledge of functions into a rigorous understand-
ing of what functions are and how they work. A few additional resources that can
help you get to the next step of understanding functions include:
Hadley Wickhams Advanced R book
Roger Pengs R Programming for Data Science book
DataCamps Intermediate R course
Courseras R Programming course
Chapter 19
Loop Control Statements
Looping is similiar to creating functions in that they are merely a means to automate
a certain multi-step process by organizing sequences of R expressions. R consists of
several loop control statements which allow you to perform repetitive code pro-
cesses with different intentions and allow these automated expressions to naturally
respond to features of your data. Consequently, learning these loop control state-
ments will go a long ways in reducing code redundancy and becoming a more ef-
cient data wrangler.
This chapter starts by covering the basic control statements in R, which includes
if, else, along with the for, while, and repeat loop control structures. In
addition, I cover break and next which allow you to further control ow within
the aforementioned control statements. Next I cover a set of vectorized functions
known as the apply family of functions which minimize your need to explicitly cre-
ate loops. I then provide some additional loop-like functions that are helpful in
everyday data analysis followed by a list of additional resources to learn more about
control structures in R.
19.1 Basic Control Statements (i.e. if, for, while, etc.)
19.1.1 if Statement
The conditional if statement is used to test an expression. If the test_expres-

sion is TRUE, the statement gets executed. But if its FALSE, nothing
happens.
# syntax of if statement
if (test_expression) {
statement
}

184 19 Loop Control Statements
The following is an example that tests if any values in a vector are negative.
Notice there are two ways to write this if statement; since the body of the state-
ment is only one line you can write it with or without curly braces. I recommend
getting in the habit of using curly braces, that way if you build onto if statements
with additional functions in the body or add an else statement later you will not
run into issues with unexpected code procedures.
x <- c(8, 3, -2, 5)
# without curly braces

if(any(x < 0)) print("x contains negative numbers")
## [1] "x contains negative numbers"
# with curly braces produces same result

if(any(x < 0)){
print("x contains negative numbers")
}
# an if statement in which the test expression is FALSE

# does not produce any output
y <- c(8, 3, 2, 5)
if(any(y < 0)){

print("y contains negative numbers")
}
19.1.2 ifelse Statement
The conditional ifelse statement is used to test an expression similar to the if

statement. However, rather than nothing happening if the test_expression is
FALSE, the else part of the function will be evaluated.
# syntax of ifelse statement
if (test_expression) {
statement 1
} else {
statement 2
}
The following extends the previous example illustrated for the if statement in
which the if statement tests if any values in a vector are negative; if TRUE it pro-
duces one output and if FALSE it produces the else output.
# this test results in statement 1 being executed
x <- c(8, 3, -2, 5)
19.1 Basic Control Statements (i.e. if, for, while, etc.) 185
if(any(x < 0)){

print("x contains negative numbers")
} else{
print("x contains all positive numbers")
}
# this test results in statement 2 (or the else statement) being executed
y <- c(8, 3, 2, 5)
if(any(y < 0)){

print("y contains negative numbers")
} else{
print("y contains all positive numbers")
}
## [1] "y contains all positive numbers"
Simple ifelse statements, as above, in which only one line of code is being
executed in the statements can be written in a simplied alternative manner. These
alternatives are only recommended for very short ifelse code because they can
become difcult to read as the character length increases.
x <- c(8, 3, 2, 5)
# alternative 1
if(any(x < 0)) print("x contains negative numbers") else print("x contains all
positive numbers")
## [1] "x contains all positive numbers"
# alternative 2 using the ifelse function

ifelse(any(x < 0), "x contains negative numbers", "x contains all positive numbers")
## [1] "x contains all positive numbers"
We can also nest as many ifelse statements as required (or desired). For
example:
# this test results in statement 1 being executed
x <- 7
if(x >= 10){

print("x exceeds acceptable tolerance levels")
} else if(x >= 0 & x < 10){
print("x is within acceptable tolerance levels")
} else {
print("x is negative")
}
## [1] "x is within acceptable tolerance levels"
19.1.3 for Loop
The for loop is used to execute repetitive code statements for a particular number
of times. The general syntax is provided below where i is the counter and as i
assumes each sequential value dened (1 through 100 in this example) the code in
the body will be performed for that ith value.
# syntax of for loop
for(i in 1:100) {
<do stuff here with i>
}
For example, the following for loop iterates through each value (2010, 2011, ,
2016) and performs the paste and print functions inside the curly brackets.
for(i in 2010:2016) {
output <- paste("The year is", i)
print(output)
}
## [1] "The year is 2010"
## [1] "The year is 2011"
## [1] "The year is 2012"
## [1] "The year is 2013"
## [1] "The year is 2014"
## [1] "The year is 2015"
## [1] "The year is 2016"
If you want to perform the for loop but have the outputs combined into a vector
or other data structure than you can initiate the output data structure prior to the for
loop. For instance, if we want to have the previous outputs combined into a single
vector x we can initiate x rst and then append the for loop output to x.
x <- NULL
for(i in 2010:2016) {
x <- append(x, output)
}
x
## [1] "The year is 2010" "The year is 2011" "The year is 2012"
"The year is 2013"
However, an important lesson to learn is that R is not efcient at growing data

objects. As a result, it is more efcient to create an empty data object and fill it with the
for loop outputs. In the previous example we grew x by appending new values to it.
A more efcient practice is to initiate a vector (or other data structure) of the right size
and ll the elements. In the example that follows, we create the vector x of the right
size and then ll in each element within the for loop. Although this inefciency is not
noticed in this small example, when you perform larger repetitions it will become
noticeable so you might as well get in the habit of filling rather than growing.
x <- vector(mode = numeric, length = 7)
counter <- 1
for(i in 2010:2016) {
x[counter] <- output
counter <- counter + 1
}
x
## [1] "The year is 2010" "The year is 2011" "The year is 2012" "The year is 2013"
Another example in which we create an empty matrix with 5 rows and 5 col-
umns. The for loop then iterates over each column (note how i takes on the values
1 through the number of columns in the my.mat matrix) and takes a random draw
of 5 values from a Poisson distribution with mean i in column i:
my.mat <- matrix(NA, nrow = 5, ncol = 5)
for(i in 1:ncol(my.mat)){
my.mat[, i] <- rpois(5, lambda = i)
}
my.mat
## [,1] [,2] [,3] [,4] [,5]
## [1,] 0 2 1 7 1
## [2,] 1 2 2 3 9
## [3,] 2 1 5 6 6
## [4,] 2 1 5 2 10
## [5,] 0 2 2 2 4
19.1.4 while Loop
While loops begin by testing a condition. If it is true, then they execute the state-
ment. Once the statement is executed, the condition is tested again, and so forth,
until the condition is false, after which the loop exits. Its considered a best practice
to include a counter object to keep track of total iterations
# syntax of while loop
counter <- 1
while(test_expression) {
statement
}
while loops can potentially result in innite loops if not written properly; there-
fore, you must use them with care. To provide a simple example to illustrate how
similiar for and while loops are:
counter <- 1
while(counter <= 10) {

print(counter)
}
# this for loop provides the same output

counter <- vector(mode = "numeric", length = 10)
for(i in 1:length(counter)) {
print(i)
}
The primary difference between a for loop and a while loop is: a for loop is
used when the number of iterations a code should be run is known where a while
loop is used when the number of iterations is not known. For instance, the following
takes value x and adds or subtracts 1 from the value randomly until x exceeds the
values in the test expression. The output illustrates that the code runs 14 times until
x exceeded the threshold with the value 9.
counter <- 1
x <- 5
set.seed(3)
while(x >= 3 && x <= 8 ) {

coin <- rbinom(1, 1, 0.5)
if(coin == 1) { ## random walk
x <- x + 1
} else {
x <- x - 1
}
cat("On iteration", counter, ", x =", x, '\n')
}
## On iteration 1 , x = 4
19.1.5 repeat Loop
A repeat loop is used to iterate over a block of code multiple number of times.
There is not a test expression in a repeat loop to end or exit the loop. Rather, we
must put a condition statement explicitly inside the body of the loop and use the
break function to exit the loop. Failing to do so will result into an innite loop.
# syntax of repeat loop
counter <- 1
repeat {
statement
if(test_expression){
break
}
}
For example, say we want to randomly draw values from a uniform distribution
between 1 and 25. Furthermore, we want to continue to draw values randomly until
our sample contains at least each integer value between 1 and 25; however, we do
not care if weve drawn a particular value multiple times. The following code repeats
the random draws of values between 1 and 25 (which we round). We then include
an if statement to check if all values between 1 and 25 are present in our sample.
If so, we use the break statement to exit the loop. If not, we add to our counter and
let the loop repeat until the conditional if statement is found to be true. We can
then check the counter object to assess how many iterations were required to
reach our conditional requirement.
counter <- 1
x <- NULL
repeat {
x <- c(x, round(runif(1, min = 1, max = 25)))
if(all(1:25 %in% x)){
break
}
}
counter
## [1] 75
19.1.6 break Function to Exit a Loop
The break function is used to exit a loop immediately, regardless of what iteration
the loop may be on. break functions are typically embedded in an if statement in
which a condition is assessed, if TRUE break out of the loop, if FALSE continue
on with the loop. In a nested looping situation, where there is a loop inside another
loop, this statement exits from the innermost loop that is being evaluated.
x <- 1:5
for (i in x) {
if (i == 3){
break
}
print(i)
}
## [1] 1
## [1] 2
19.1.7 next Function to Skip an Iteration in a Loop
The next statement is useful when we want to skip the current iteration of a loop
without terminating it. On encountering next, the R parser skips further evaluation
and starts next iteration of the loop.
x <- 1:5
for (i in x) {
if (i == 3){
next
}
print(i)
}
## [1] 1
## [1] 2
## [1] 4
## [1] 5
19.2 Apply Family
The apply family consists of vectorized functions which minimize your need to
explicitly create loops. These functions will apply a specied function to a data
object and there primary difference is in the object class in which the function is
applied to (list vs. matrix, etc) and the object class that will be returned from the
function. The following presents the most common forms of apply functions that I
use for data analysis but realize that additional functions exist (mapply, rapply,
and vapply) which are not covered here.
19.2 Apply Family 191
19.2.1 apply() for Matrices and Data Frames
The apply() function is most often used to apply a function to the rows or col-
umns (margins) of matrices or data frames. However, it can be used with general
arrays, for example, to take the average of an array of matrices. Using apply() is
not faster than using a loop function, but it is highly compact and can be written in
one line.
The syntax for apply() is as follows where
x is the matrix, dataframe or array
MARGIN is a vector giving the subscripts which the function will be applied over.
E.g., for a matrix 1 indicates rows, 2 indicates columns, c(1, 2) indicates rows
and columns.
FUN is the function to be applied
is for any other arguments to be passed to the function
# syntax of apply function

apply(x, MARGIN, FUN, )
To provide examples lets use the mtcars data set provided in R:

# show first few rows of mtcars
head(mtcars)
## mpg cyl disp hp drat wt qsec vs am gear carb
## Mazda RX4 21.0 6 160 110 3.90 2.620 16.46 0 1 4 4
## Mazda RX4 Wag 21.0 6 160 110 3.90 2.875 17.02 0 1 4 4
## Datsun 710 22.8 4 108 93 3.85 2.320 18.61 1 1 4 1
## Hornet 4 Drive 21.4 6 258 110 3.08 3.215 19.44 1 0 3 1
## Hornet Sportabout 18.7 8 360 175 3.15 3.440 17.02 0 0 3 2
## Valiant 18.1 6 225 105 2.76 3.460 20.22 1 0 3 1
# get the mean of each column

apply(mtcars, 2, mean)
## mpg cyl disp hp drat wt
## 20.090625 6.187500 230.721875 146.687500 3.596563 3.217250
## qsec vs am gear carb
## 17.848750 0.437500 0.406250 3.687500 2.812500
# get the sum of each row (not really relevant for this data
# but it illustrates the capability)
apply(mtcars, 1, sum)
## Mazda RX4 Mazda RX4 Wag Datsun 710
## 328.980 329.795 259.580
## Hornet 4 Drive Hornet Sportabout Valiant
## 426.135 590.310 385.540
## Duster 360 Merc 240D Merc 230
## 656.920 270.980 299.570
## Merc 280 Merc 280C Merc 450SE
## 350.460 349.660 510.740
## Merc 450SL Merc 450SLC Cadillac Fleetwood
## 511.500 509.850 728.560
## Lincoln Continental Chrysler Imperial Fiat 128
## 726.644 725.695 213.850

## Honda Civic Toyota Corolla Toyota Corona
## 195.165 206.955 273.775
## Dodge Challenger AMC Javelin Camaro Z28
## 519.650 506.085 646.280
## Pontiac Firebird Fiat X1-9 Porsche 914-2
## 631.175 208.215 272.570
## Lotus Europa Ford Pantera L Ferrari Dino
## 273.683 670.690 379.590
## Maserati Bora Volvo 142E
## 694.710 288.890
# get column quantiles (notice the quantile percents as row names)

apply(mtcars, 2, quantile, probs = c(0.10, 0.25, 0.50, 0.75, 0.90))
## 10% 14.340 4 80.610 66.0 3.007 1.95550 15.5340 0 0 3 1
## 25% 15.425 4 120.825 96.5 3.080 2.58125 16.8925 0 0 3 2
## 50% 19.200 6 196.300 123.0 3.695 3.32500 17.7100 0 0 4 2
## 75% 22.800 8 326.000 180.0 3.920 3.61000 18.9000 1 1 4 4
## 90% 30.090 8 396.000 243.5 4.209 4.04750 19.9900 1 1 5 4
19.2.2 lapply() for ListsOutput as a List
The lapply() function does the following simple series of operations:

1. it loops over a list, iterating over each element in that list
2. it applies a function to each element of the list (a function that you specify)
3. and returns a list (the l is for list).
The syntax for lapply() is as follows where
x is the list
FUN is the function to be applied
is for any other arguments to be passed to the function
# syntax of lapply function
lapply(x, FUN, )
To provide examples well generate a list of four items:

data <- list(item1 = 1:4, item2 = rnorm(10),
item3 = rnorm(20, 1), item4 = rnorm(100, 5))
# get the mean of each list item

lapply(data, mean)
## $item1
## [1] 2.5
##
## $item2
## [1] 0.5529324
##
19.2 Apply Family 193
## $item3
## [1] 1.193884
##
## $item4
## [1] 5.013019
The above provides a simple example where each list item is simply a vector of
numeric values. However, consider the case where you have a list that contains data
frames and you would like to loop through each list item and perform a function to the
data frame. In this case we can embed an apply function within an lapply function.
For example, the following creates a list for Rs built in beaver data sets.
The lapply function loops through each of the two list items and uses apply
to calculate the mean of the columns in both list items. Note that I wrap the apply
function with round to provide an easier to read output.
# list of R's built in beaver data
beaver_data <- list(beaver1 = beaver1, beaver2 = beaver2)
# get the mean of each list item

lapply(beaver_data, function(x) round(apply(x, 2, mean), 2))
## $beaver1
## day time temp activ
## 346.20 1312.02 36.86 0.05
##
## $beaver2
## 307.13 1446.20 37.60 0.62
19.2.3 sapply() for ListsOutput Simplified
The sapply() function behaves similarly to lapply(); the only real difference
is in the return value. sapply() will try to simplify the result of lapply() if
possible. Essentially, sapply() calls lapply() on its input and then applies the
following algorithm:
If the result is a list where every element is length 1, then a vector is returned
If the result is a list where every element is a vector of the same length (> 1), a
matrix is returned.
If neither of the above simplications can be performed then a list is returned
To illustrate the differences we can use the previous example using a list with the
beaver data and compare the sapply and lapply outputs:
# list of R's built in beaver data
beaver_data <- list(beaver1 = beaver1, beaver2 = beaver2)
# get the mean of each list item and return as a list

lapply(beaver_data, function(x) round(apply(x, 2, mean), 2))
## $beaver1
## 346.20 1312.02 36.86 0.05
##
## $beaver2
## 307.13 1446.20 37.60 0.62
# get the mean of each list item and simplify the output
sapply(beaver_data, function(x) round(apply(x, 2, mean), 2))
## beaver1 beaver2
## day 346.20 307.13
## time 1312.02 1446.20
## temp 36.86 37.60
## activ 0.05 0.62
19.2.4 tapply() for Vectors
tapply() is used to apply a function over subsets of a vector. It is primarily used

when we have the following circumstances:
1. A dataset that can be broken up into groups (via categorical variables - aka
factors)
2. We desire to break the dataset up into groups
3. Within each group, we want to apply a function
The arguments to tapply() are as follows:
x is a vector
INDEX is a factor or a list of factors (or else they are coerced to factors)
FUN is a function to be applied
contains other arguments to be passed FUN
simplify, should we simplify the result?
# syntax of tapply function

tapply(x, INDEX, FUN, , simplify = TRUE)
To provide an example well use the built in mtcars dataset and calculate the
mean of the mpg variable grouped by the cyl variable.
# show first few rows of mtcars
head(mtcars)
## Mazda RX4 21.0 6 160 110 3.90 2.620 16.46 0 1 4 4
## Mazda RX4 Wag 21.0 6 160 110 3.90 2.875 17.02 0 1 4 4
## Datsun 710 22.8 4 108 93 3.85 2.320 18.61 1 1 4 1
## Hornet 4 Drive 21.4 6 258 110 3.08 3.215 19.44 1 0 3 1
## Valiant 18.1 6 225 105 2.76 3.460 20.22 1 0 3 1
19.3 Other Useful Loop-Like Functions 195
# get the mean of the mpg column grouped by cylinders

tapply(mtcars$mpg, mtcars$cyl, mean)
## 4 6 8
## 26.66364 19.74286 15.10000
Now lets say you want to calculate the mean for each column in the mtcars
dataset grouped by the cylinder categorical variable. To do this you can embed the
tapply function within the apply function.
# get the mean of all columns grouped by cylinders
apply(mtcars, 2, function(x) tapply(x, mtcars$cyl, mean))
## mpg cyl disp hp drat wt qsec vs
## 4 26.66364 4 105.1364 82.63636 4.070909 2.285727 19.13727 0.9090909
## 6 19.74286 6 183.3143 122.28571 3.585714 3.117143 17.97714 0.5714286
## 8 15.10000 8 353.1000 209.21429 3.229286 3.999214 16.77214 0.0000000
## am gear carb
## 4 0.7272727 4.090909 1.545455
## 6 0.4285714 3.857143 3.428571
## 8 0.1428571 3.285714 3.500000
Note that this type of summarization can also be done using the dplyr package
with clearer syntax. This is covered in the Transforming Your Data with dplyr
section.
19.3 Other Useful Loop-Like Functions
In addition to the apply family which provides vectorized functions that minimize
your need to explicitly create loops, there are also a few commonly applied apply
functions that have been further simplied. These include the calculation of column
and row sums, means, medians, standard deviations, variances, and summary quan-
tiles across the entire data set.
The most common apply functions include calculating the sums and means of
columns and rows. For instance, to calculate the sum of columns across a data frame
or matrix you could do the following:
apply(mtcars, 2, sum)
## 642.900 198.000 7383.100 4694.000 115.090 102.952 571.160 14.000
## am gear carb
## 13.000 118.000 90.000
However, you can perform the same function with the shorter colSums()
function and it performs faster:
colSums(mtcars)
## 642.900 198.000 7383.100 4694.000 115.090 102.952 571.160 14.000
## am gear carb
## 13.000 118.000 90.000
To illustrate the speed difference we can compare the performance of using the
apply() function versus the colSums() function on a matrix with 100 million
values (10K 10K). You can see that the speed of colSums() is signicantly faster.
# develop a 10,000 x 10,000 matrix
mat = matrix(sample(1:10, size=100000000, replace=TRUE), nrow=10000)
system.time(apply(mat, 2, sum))
## user system elapsed
## 1.544 0.329 1.879
system.time(colSums(mat))
## user system elapsed
## 0.126 0.000 0.127
Base R provides the following simplied apply functions:

colSums (x, na.rm = FALSE)
rowSums (x, na.rm = FALSE)
colMeans(x, na.rm = FALSE)
rowMeans(x, na.rm = FALSE)
In addition, the following functions are provided through the specied
packages:
miscTools package (note that these functions will work on data frames)
colMedians()
rowMedians()
matrixStats package (note that these functions only operate on matrices)
colMedians() and rowMedians()
colSds() and rowSds()
colVar() and rowVar()
colRanges() and rowRanges()
colQuantiles() and rowQuantiles()
along with several additional summary statistic functions
In addition, the summary() function will provide relevant summary statistics
over each column of data frames and matrices. Note in the example that follows that
for the rst four columns of the iris data set the summary statistics include min,
med, mean, max, and rst and third quantiles. Whereas the last column (Species)
only provides the total count since this is a factor variable.
summary(iris)
## Sepal.Length Sepal.Width Petal.Length Petal.Width
## Min. :4.300 Min. :2.000 Min. :1.000 Min. :0.100
## 1st Qu.:5.100 1st Qu.:2.800 1st Qu.:1.600 1st Qu.:0.300
## Median :5.800 Median :3.000 Median :4.350 Median :1.300
## Mean :5.843 Mean :3.057 Mean :3.758 Mean :1.199
## 3rd Qu.:6.400 3rd Qu.:3.300 3rd Qu.:5.100 3rd Qu.:1.800
## Max. :7.900 Max. :4.400 Max. :6.900 Max. :2.500

## Species
## setosa :50
## versicolor:50
## virginica :50
##
##
##
This provides an introduction to control statements in R. However, the following

provides additional resources to learn more:
Tutorial on loops by DataCamp
Roger Pengs R Programming for Data Science
Hadley Wickhams Advanced R
Chapter 20
Simplify Your Code with %>%
Removing duplication is an important principle to keep in mind with your code;

however, equally important is to keep your code efcient and readable. Efciency is
often accomplished by leveraging functions and control statements in your code.
However, efciency also includes eliminating the creation and saving of unneces-
sary objects that often result when you are trying to make your code more readable,
clear, and explicit. Consequently, writing code that is simple, readable, and efcient
is often considered contradictory. For this reason, the magrittr package is a pow-
erful tool to have in your data wrangling toolkit.
The magrittr package was created by Stefan Milton Bache and, in Stefans words,
has two primary aims: to decrease development time and to improve readability
and maintainability of code. Hence, it aims to increase efciency and improve
readability; and in the process it greatly simplies your code. The following covers
the basics of the magrittr toolkit.
20.1 Pipe (%>%) Operator
The principal function provided by the magrittr package is %>%, or whats

called the pipe operator. This operator will forward a value, or the result of an
expression, into the next function call/expression. For instance a function to lter
data can be written as:
filter(data, variable == numeric_value)
or
data %>% filter(variable == numeric_value)

200 20 Simplify Your Code with %>%
Both functions complete the same task and the benet of using %>% may not be
immediately evident; however, when you desire to perform multiple functions its
advantage becomes obvious. For instance, if we want to lter some data, group it by
categories, summarize it, and then order the summarized results we could write it
out three different ways. Dont worry, youll learn how to operate these specic
functions in the next section.
20.1.1 Nested Option
library(magrittr)
library(dplyr)
arrange(
summarize(
group_by(
filter(mtcars, carb > 1),
cyl
),
Avg_mpg = mean(mpg)
),
desc(Avg_mpg)
)
##
## cyl Avg_mpg
## (dbl) (dbl)
## 1 4 25.90
## 2 6 19.74
## 3 8 15.10
This rst option is considered a nested option such that functions are nested
within one another. Historically, this has been the traditional way of integrating code;
however, it becomes extremely difcult to read what exactly the code is doing and it
also becomes easier to make mistakes when making updates to your code. Although
not in violation of the DRY principle, it denitely violates the basic principle of read-
ability and clarity, which makes communication of your analysis more difcult. To
make things more readable, people often move to the following approach
20.1.2 Multiple Object Option
a <- filter(mtcars, carb > 1)

b <- group_by(a, cyl)
c <- summarise(b, Avg_mpg = mean(mpg))
d <- arrange(c, desc(Avg_mpg))
print(d)
20.1 Pipe (%>%) Operator 201
##
## cyl Avg_mpg
## (dbl) (dbl)
## 1 4 25.90
## 2 6 19.74
## 3 8 15.10
This second option helps in making the data wrangling steps more explicit and
obvious but denitely violates the DRY principle. By sequencing multiple functions in
this way you are likely saving multiple outputs that are not very informative to you or
others; rather, the only reason you save them is to insert them into the next function to
eventually get the nal output you desire. This inevitably creates unnecessary copies
and wrecks havoc on properly managing your objectsbasically it results in a global
environment charlie foxtrot! To provide the same readability (or even better), we can
use %>% to string these arguments together without unnecessary object creation
20.1.3 %>% Option
mtcars %>%
filter(carb > 1) %>%
group_by(cyl) %>%
summarise(Avg_mpg = mean(mpg)) %>%
arrange(desc(Avg_mpg))
##
## cyl Avg_mpg
## (dbl) (dbl)
## 1 4 25.90
## 2 6 19.74
## 3 8 15.10
This nal option which integrates %>% operators makes for more efcient and
legible code. Its efcient in that it doesnt save unnecessary objects (as in option 2)
and performs as effectively (as both option 1 and 2) but makes your code more read-
able in the process. Its legible in that you can read this as you would read normal
prose (we read the %>% as and then) - take mtcars and then filter and then
group by and then summarize and then arrange.
And since R is a functional programming language, meaning that everything you
do is basically built on functions, you can use the pipe operator to feed into just
about any argument call. For example, we can pipe into a linear regression function
and then get the summary of the regression parameters. Note in this case I insert
data = . into the lm() function. When using the %>% operator the default is
the argument that you are forwarding will go in as the first argument of the function
that follows the %>%. However, in some functions the argument you are forwarding
does not go into the default rst position. In these cases, you place . to signal
which argument you want the forwarded expression to go to.
mtcars %>%
lm(mpg ~ cyl + hp, data = .) %>%
summary()
##
## Call:
## lm(formula = mpg ~ cyl + hp, data = .)
##
## Residuals:
## Min 1Q Median 3Q Max
## -4.6163 -1.4162 -0.1506 1.6181 5.2021
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 35.67647 2.28382 15.621 2.16e-13 ***
## cyl -2.22014 0.52619 -4.219 0.000353 ***
## hp -0.01414 0.01323 -1.069 0.296633
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.689 on 22 degrees of freedom
## Multiple R-squared: 0.7601, Adjusted R-squared: 0.7383
## F-statistic: 34.85 on 2 and 22 DF, p-value: 1.516e-07
You can also use %>% to feed into plots:

library(ggplot2)
mtcars %>%
qplot(x = wt, y = mpg, data = .)
30
25
mpg
20
15
10
2 3 4 5
wt
Piping into a Plot
20.2 Additional Functions 203
You will also nd that the %>% operator is now being built into packages to make
programming much easier. For instance, in the section that follows where I illustrate
how to reshape and transform your data with the dplyr and tidyr packages, you
will see that the %>% operator is already built into these packages. It is also built into the
ggvis and dygraphs packages (visualization packages), the httr package (which
we covered in the data scraping chapter), and a growing number of newer packages.
20.2 Additional Functions
In addition to the %>% operator, magrittr provides several additional functions

which make operations such as addition, multiplication, logical operators, re-
naming, etc. more pleasant when composing chains using the %>% operator. Some
examples follow but you can see the current list of the available aliased functions by
typing ?magrittr::add in your console.
# subset with extract
mtcars %>%
extract(, 1:4) %>%
head
## mpg cyl disp hp
## Mazda RX4 21.0 6 160 110
## Mazda RX4 Wag 21.0 6 160 110
## Datsun 710 22.8 4 108 93
## Hornet 4 Drive 21.4 6 258 110
## Hornet Sportabout 18.7 8 360 175
## Valiant 18.1 6 225 105
# add, subtract, multiply, divide and other operations are available

mtcars %>%
extract(, "mpg") %>%
multiply_by(5)
## [1] 105.0 105.0 114.0 107.0 93.5 90.5 71.5 122.0 114.0 96.0 89.0
## [12] 82.0 86.5 76.0 52.0 52.0 73.5 162.0 152.0 169.5 107.5 77.5
## [23] 76.0 66.5 96.0 136.5 130.0 152.0 79.0 98.5 75.0 107.0
# logical assessments and filters are available

mtcars %>%
extract(, "cyl") %>%
equals(4)
## [1] FALSE FALSE TRUE FALSE FALSE FALSE FALSE TRUE TRUE FALSE FALSE
## [12] FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE FALSE
## [23] FALSE FALSE FALSE TRUE TRUE TRUE FALSE FALSE FALSE TRUE
# renaming columns and rows is available

mtcars %>%
head %>%
set_colnames(paste("Col", 1:11, sep = ""))
## Col1 Col2 Col3 Col4 Col5 Col6 Col7 Col8 Col9 Col10
## Mazda RX4 21.0 6 160 110 3.90 2.620 16.46 0 1 4
## Mazda RX4 Wag 21.0 6 160 110 3.90 2.875 17.02 0 1 4
## Datsun 710 22.8 4 108 93 3.85 2.320 18.61 1 1 4
## Hornet 4 Drive 21.4 6 258 110 3.08 3.215 19.44 1 0 3
## Hornet Sportabout 18.7 8 360 175 3.15 3.440 17.02 0 0 3

## Valiant 18.1 6 225 105 2.76 3.460 20.22 1 0 3
## Col11
## Mazda RX4 4
## Mazda RX4 Wag 4
## Datsun 710 1
## Hornet 4 Drive 1
## Hornet Sportabout 2
## Valiant 1
20.3 Additional Pipe Operators
magrittr also offers some alternative pipe operators. Some functions, such as
plotting functions, will cause the string of piped arguments to terminate. The tee
(%T>%) operator allows you to continue piping functions that normally cause
termination.
# normal piping terminates with the plot() function resulting in
# NULL results for the summary() function
mtcars %>%
extract(, 1:4) %>%
plot() %>%
summary()
4 5 6 7 8 50 150 250
mpg 30
20
10
8
4 5 6 7
cyl
300
disp
100
50 150 250
hp
10 15 20 25 30 100 200 300 400

Regular Pipe Operator Terminates String of Functions at a Plot
20.3 Additional Pipe Operators 205
## Length Class Mode

## 0 NULL NULL
# inserting %T>% allows you to plot and perform the functions that
# follow the plotting function
mtcars %>%
extract(, 1:4) %T>%
plot() %>%
summary()
4 5 6 7 8 50 150 250
30
mpg
20
10
8
4 5 6 7
cyl
300
disp
100
50 150 250
hp
10 15 20 25 30 100 200 300 400

Tee Operator Allows You to Pipe Through a Plot
## mpg cyl disp hp

## Min. :10.40 Min. :4.00 Min. : 75.7 Min. : 52.0
## 1st Qu.:15.20 1st Qu.:6.00 1st Qu.:146.7 1st Qu.:110.0
## Median :17.80 Median :8.00 Median :275.8 Median :175.0
## Mean :18.62 Mean :6.64 Mean :257.7 Mean :163.7
## 3rd Qu.:21.00 3rd Qu.:8.00 3rd Qu.:351.0 3rd Qu.:205.0
## Max. :30.40 Max. :8.00 Max. :472.0 Max. :335.0
The compound assignment %<>% operator is used to update a value by rst pip-
ing it into one or more expressions, and then assigning the result. For instance, lets
say you want to transform the mpg variable in the mtcars data frame to a square
root measurement. Using %<>% will perform the functions to the right of %<>% and
save the changes these functions perform to the variable or data frame called to the
left of %<>%.
# note that mpg is in its typical measurement

head(mtcars)
## Mazda RX4 21.0 6 160 110 3.90 2.620 16.46 0 1 4 4
## Mazda RX4 Wag 21.0 6 160 110 3.90 2.875 17.02 0 1 4 4
## Datsun 710 22.8 4 108 93 3.85 2.320 18.61 1 1 4 1
## Hornet 4 Drive 21.4 6 258 110 3.08 3.215 19.44 1 0 3 1
## Valiant 18.1 6 225 105 2.76 3.460 20.22 1 0 3 1
# we can square root mpg and save this change using %<>%
mtcars$mpg %<>% sqrt
head(mtcars)
## Mazda RX4 4.582576 6 160 110 3.90 2.620 16.46 0 1 4 4
## Mazda RX4 Wag 4.582576 6 160 110 3.90 2.875 17.02 0 1 4 4
## Datsun 710 4.774935 4 108 93 3.85 2.320 18.61 1 1 4 1
## Hornet 4 Drive 4.626013 6 258 110 3.08 3.215 19.44 1 0 3 1
## Valiant 4.254409 6 225 105 2.76 3.460 20.22 1 0 3 1
Some functions (e.g. lm, aggregate, cor) have a data argument, which allows the
direct use of names inside the data as part of the call. The exposition (%$%) operator
is useful when you want to pipe a data frame, which may contain many columns,
into a function that is only applied to some of the columns. For example, the correla-
tion (cor) function only requires an x and y argument so if you pipe the mtcars
data into the cor function using %>% you will get an error because cor doesnt
know how to handle mtcars. However, using %$% allows you to say take this
dataframe and then perform cor() on these specied columns within mtcars.
# regular piping results in an error
mtcars %>%
subset(vs == 0) %>%
cor(mpg, wt)
## Error in pmatch(use, c("all.obs", "complete.obs", "pairwise.complete.obs", :
object 'wt' not found
# using %$% allows you to specify variables of interest

mtcars %>%
subset(vs == 0) %$%
cor(mpg, wt)
## [1] -0.830671
The magrittr package and its pipe operators are a great tool for making your code
simple, efcient, and readable. There are limitations, or at least suggestions, on when
and how you should use the operators. Garrett Grolemund and Hadley Wickham
offer some advice on the proper use of pipe operators in their R for Data Science
book. However, the %>% has greatly transformed our ability to write simplied
code in R. As the pipe gains in popularity you will likely nd it in more future pack-
ages and being familiar will likely result in better communication of your code.
Some additional resources regarding magrittr and the pipe operators you
may nd useful:
The magrittr vignette (vignette("magrittr")) in your console) pro-
vides additional examples of using pipe operators and functions provided by
magrittr.
A blog post by Stefan Milton Bache regarding the past, present and future of
magrittr1
magrittr questions on Stack Overow
The ensurer package, also written by Stefan Milton Bache, provides a useful way
of verifying and validating data outputs in a sequence of pipe operators.
1
https://www.r-bloggers.com/simpler-r-coding-with-pipes-the-present-and-future-of-the-magrittr-
package/
Part VI
Shaping and Transforming
Your Data with R
Up to 80 % of data analysis is spent on the process of cleaning
and preparing data.
cf.Wickham ( 2014) and Dasu and Johnson (2003)
A tremendous amount of time is spent on fundamental preprocessing tasks to get

your data into the right form in order to feed it into the visualization and modeling
stages. This typically requires a large amount of reshaping and transformation of
your data. Although many fundamental data processing functions exist in R, they
have been a bit convoluted to date and have lacked consistent coding and the ability
to easily ow together. The RStudio team has been driving a lot of new packages to
collate data management tasks and better integrate them with other analysis activi-
ties. As a result, a lot of data processing tasks are becoming packaged in more
cohesive and consistent ways which leads to more efcient code and easier to read
syntax. This section covers two of these packages: tidyr and dplyr.
In this section, I start by providing a fundamental understanding of tidy data fol-
lowed by demonstrating how to to use tidyr to turn wide data to long, long data
to wide, splitting and combining variables, along with illustrating some lesser-
known functions. Subsequently, I provide an introduction to the dplyr package by
covering seven primary functions dplyr provides for simplied data transforma-
tion and manipulation. This includes tasks such as ltering, summarizing, ordering,
joining, and much more. Understanding and using these two packages will help to
signicantly reduce the time you spend on the data wrangling process.
Chapter 21
Reshaping Your Data with tidyr
Cannot emphasize enough how much time you save by putting

analysis efforts into tidying data first.
Hilary Parker
Jenny Bryan stated that classroom data are like teddy bears and real data are like a
grizzley bear with salmon blood dripping out its mouth. In essence, she was getting
to the point that often when we learn how to perform a modeling approach in the
classroom, the data used is provided in a format that appropriately feeds into the
modeling tool of choice. In reality, datasets are messy and every messy dataset is
messy in its own way.1 The concept of tidy data was established by Hadley
Wickham and represents standardized way to link the structure of a dataset (its
physical layout) with its semantics (its meaning).2 The objective should always be
to get a dataset into a tidy form which consists of:
1. Each variable forms a column
2. Each observation forms a row
3. Each type of observational unit forms a table
To create tidy data you need to be able to reshape your data; preferably via ef-
cient and simple code. To help with this process Hadley created the tidyr pack-
age. This chapter covers the basics of tidyr to help you reshape your data as
necessary. I demonstrate how to turn wide data to long, long data to wide, splitting
and combining variables, and nally I will cover some lesser known functions in
tidyr that are useful. Note that throughout I use the %>% operator we covered in
the last chapter. Although not required, the tidyr package has the %>% operator
baked in to its functionality, which allows you to sequence multiple tidy functions
together.
1
Wickham, H. (2014). Tidy data. Journal of Statistical Software, 59(10). [document].
2
Ibid.

212 21 Reshaping Your Data with tidyr
21.1 Making Wide Data long
There are times when our data is considered wide or unstacked and a common
attribute/variable of concern is spread out across columns. To reformat the data such
that these common attributes are gathered together as a single variable, the
gather() function will take multiple columns and collapse them into key-value
pairs, duplicating all other columns as needed.
For example, lets say we have the given data frame.
library(dplyr) # I'm using dplyr just to create the data frame with tbl_df()
wide <- tbl_df(read.table(header = TRUE, text = "

Group Year Qtr.1 Qtr.2 Qtr.3 Qtr.4
1 2006 15 16 19 17
1 2007 12 13 27 23
1 2008 22 22 24 20
1 2009 10 14 20 16
2 2006 12 13 25 18
2 2007 16 14 21 19
2 2008 13 11 29 15
2 2009 23 20 26 20
3 2006 11 12 22 16
3 2007 13 11 27 21
3 2008 17 12 23 19
3 2009 14 9 31 24
"))
This data is considered wide since the time variable (represented as quarters) is
structured such that each quarter represents a variable. To re-structure the time compo-
nent as an individual variable, we can gather each quarter within one column variable
and also gather the values associated with each quarter in a second column variable.
library(tidyr)
long <- wide %>% gather(Quarter, Revenue, Qtr.1:Qtr.4)
# note, for brevity, I only show the first 15 observations

head(long, 15)
##
## Group Year Quarter Revenue
## (int) (int) (fctr) (int)
## 1 1 2006 Qtr.1 15
## 2 1 2007 Qtr.1 12
## 3 1 2008 Qtr.1 22
## 4 1 2009 Qtr.1 10
## 5 2 2006 Qtr.1 12
## 6 2 2007 Qtr.1 16
## 7 2 2008 Qtr.1 13
## 8 2 2009 Qtr.1 23
## 9 3 2006 Qtr.1 11
21.3 Splitting a Single Column into Multiple Columns 213
## 10 3 2007 Qtr.1 13
## 11 3 2008 Qtr.1 17
## 12 3 2009 Qtr.1 14
## 13 1 2006 Qtr.2 16
## 14 1 2007 Qtr.2 13
## 15 1 2008 Qtr.2 22
Its important to note that there is exibility in how you specify the columns you
would like to gather. These all produce the same results:
wide %>% gather(Quarter, Revenue, Qtr.1:Qtr.4)
wide %>% gather(Quarter, Revenue, -Group, -Year)
wide %>% gather(Quarter, Revenue, 3:6)
wide %>% gather(Quarter, Revenue, Qtr.1, Qtr.2, Qtr.3, Qtr.4)
21.2 Making Long Data wide
There are also times when we are required to turn long formatted data into wide
formatted data. As a complement to gather(), the spread() function spreads
a key-value pair across multiple columns. So now lets take our long data frame
from above and turn the Quarter variable into column headings and spread the
Revenue values across the quarters they are related to.
back2wide <- long %>% spread(Quarter, Revenue)
back2wide
##
## Group Year Qtr.1 Qtr.2 Qtr.3 Qtr.4
## (int) (int) (int) (int) (int) (int)
## 1 1 2006 15 16 19 17
## 2 1 2007 12 13 27 23
## 3 1 2008 22 22 24 20
## 4 1 2009 10 14 20 16
## 5 2 2006 12 13 25 18
## 6 2 2007 16 14 21 19
## 7 2 2008 13 11 29 15
## 8 2 2009 23 20 26 20
## 9 3 2006 11 12 22 16
## 10 3 2007 13 11 27 21
## 11 3 2008 17 12 23 19
## 12 3 2009 14 9 31 24
21.3 Splitting a Single Column into Multiple Columns
Many times a single column variable will capture multiple variables, or even parts of
a variable you just dont care about. This is exemplied in the following messy_df
data frame. Here, the Grp_Ind variable combines an individual variable (a, b, c)
with the group variable (1, 2, 3), the Yr_Mo variable combines a year variable with
a month variable, etc. In each case there may be a purpose for separating parts of
these columns into separate variables.
messy_df
## Grp_Ind Yr_Mo City_State Extra_variable
## 1 1.a 2006_Jan Dayton (OH) XX01person_1
## 2 1.b 2006_Feb Grand Forks (ND) XX02person_2
## 3 1.c 2006_Mar Fargo (ND) XX03person_3
## 4 2.a 2007_Jan Rochester (MN) XX04person_4
This can be accomplished using the separate() function which turns a single
character column into multiple columns. Additional arguments provide some exi-
bility with separating columns.
# separate Grp_Ind column into two variables named "Grp" & "Ind"
messy_df %>% separate(col = Grp_Ind, into = c("Grp", "Ind"))
## Grp Ind Yr_Mo City_State Extra_variable
## 1 1 a 2006_Jan Dayton (OH) XX01person_1
## 2 1 b 2006_Feb Grand Forks (ND) XX02person_2
## 3 1 c 2006_Mar Fargo (ND) XX03person_3
## 4 2 a 2007_Jan Rochester (MN) XX04person_4
# default separater is any non alpha-numeric character but you can

# specify the specific character to separate at
messy_df %>% separate(col = Extra_variable, into = c("X", "Y"), sep = "_")
## Grp_Ind Yr_Mo City_State X Y
## 1 1.a 2006_Jan Dayton (OH) XX01person 1
## 2 1.b 2006_Feb Grand Forks (ND) XX02person 2
## 3 1.c 2006_Mar Fargo (ND) XX03person 3
## 4 2.a 2007_Jan Rochester (MN) XX04person 4
# you can keep the original column that you are separating
messy_df %>% separate(col = Grp_Ind, into = c("Grp", "Ind"), remove = FALSE)
## Grp_Ind Grp Ind Yr_Mo City_State Extra_variable
## 1 1.a 1 a 2006_Jan Dayton (OH) XX01person_1
## 2 1.b 1 b 2006_Feb Grand Forks (ND) XX02person_2
## 3 1.c 1 c 2006_Mar Fargo (ND) XX03person_3
## 4 2.a 2 a 2007_Jan Rochester (MN) XX04person_4
21.4 Combining Multiple Columns into a Single Column
Similarly, there are times when we would like to combine the values of two vari-
ables. As a compliment to separate(), the unite() function is a convenient
function to paste together multiple variable values into one. Consider the following
data frame that has separate date variables. To perform time series analysis or for
visualizations we may desire to have a single date column.
21.5 Additional tidyr Functions 215
expenses <- tbl_df(read.table(header = TRUE, text = "

Year Month Day Expense
2015 01 01 500
2015 02 05 90
2015 02 22 250
2015 03 10 325
"))
To perform time series analysis or for visualizations we may desire to have a

single date column. We can accomplish this by uniting these columns into one vari-
able with unite().
# default separator when uniting is "_"
expenses %>% unite(col = "Date", c(Year, Month, Day))
##
## Date Expense
## (chr) (int)
## 1 2015_1_1 500
## 2 2015_2_5 90
## 3 2015_2_22 250
## 4 2015_3_10 325
# specify sep argument to change separator

expenses %>% unite(col = "Date", c(Year, Month, Day), sep = "-")
##
## Date Expense
## (chr) (int)
## 1 2015-1-1 500
## 2 2015-2-5 90
## 3 2015-2-22 250
## 4 2015-3-10 325
21.5 Additional tidyr Functions
The previous four functions (gather, spread, separate and unite) are the
primary functions you will nd yourself using on a continuous basis; however, there
are some handy functions that are lesser known with the tidyr package.
expenses <- tbl_df(read.table(header = TRUE, text = "
Dept Year Month Day Cost
A 2015 01 01 $500.00
NA NA 02 05 $90.00
NA NA 02 22 $1,250.45
NA NA 03 NA $325.10
B NA 01 02 $260.00
NA NA 02 05 $90.00
", stringsAsFactors = FALSE))
Often Excel reports will not repeat certain variables. When we read these reports
in, the empty cells are typically lled in with NA such as in the Dept and Year
columns of our expense data frame. We can ll these values in with the previous
entry using fill().
expenses %>% fill(Dept, Year)
##
## Dept Year Month Day Cost
## (chr) (int) (int) (int) (chr)
## 1 A 2015 1 1 $500.00
## 2 A 2015 2 5 $90.00
## 3 A 2015 2 22 $1,250.45
## 4 A 2015 3 NA $325.10
## 5 B 2015 1 2 $260.00
## 6 B 2015 2 5 $90.00
Also, sometimes accounting values in Excel spreadsheet get read in as a charac-

ter value, which is the case for the Cost variable. We may wish to extract only the
numeric part of this regular expression, which can be done with extract_
numeric(). Note that extract_numeric works on a single variable so when
you pipe the expense data frame into the function you need to use %$% operator
as discussed in the last chapter.
library(magrittr)
expenses %$% extract_numeric(Cost)

## [1] 500.00 90.00 1250.45 325.10 260.00 90.00
# you can use this to convert and save the Cost column to a
# numeric variable
expenses$Cost <- expenses %$% extract_numeric(Cost)
expenses
##
## (chr) (int) (int) (int) (dbl)
## 1 A 2015 1 1 500.00
## 2 NA NA 2 5 90.00
## 3 NA NA 2 22 1250.45
## 4 NA NA 3 NA 325.10
## 5 B NA 1 2 260.00
## 6 NA NA 2 5 90.00
You can also easily replace missing (or NA) values with a specied value:
library(magrittr)
# replace the missing Day value

expenses %>% replace_na(replace = list(Day = "unknown"))
##
21.6 Sequencing Your tidyr Operations 217

## (chr) (int) (int) (chr) (dbl)
## 1 A 2015 1 1 500.00
## 2 NA NA 2 5 90.00
## 3 NA NA 2 22 1250.45
## 4 NA NA 3 unknown 325.10
## 5 B NA 1 2 260.00
## 6 NA NA 2 5 90.00
# replace both the missing Day and Year values

expenses %>% replace_na(replace = list(Year = 2015, Day = "unknown"))
##
## (chr) (dbl) (int) (chr) (dbl)
## 1 A 2015 1 1 500.00
## 2 NA 2015 2 5 90.00
## 3 NA 2015 2 22 1250.45
## 4 NA 2015 3 unknown 325.10
## 5 B 2015 1 2 260.00
## 6 NA 2015 2 5 90.00
21.6 Sequencing Your tidyr Operations
Since the %>% operator is embedded in tidyr, we can string multiple operations
together to efciently tidy data and make the process easy to read and follow. To
illustrate, lets use the following data, which has multiple messy attributes.
a_mess <- tbl_df(read.table(header = TRUE, text = "
Dep_Unt Year Q1 Q2 Q3 Q4
A.1 2006 15 NA 19 17
B.1 NA 12 13 27 23
A.2 NA 22 22 24 20
B.2 NA 12 13 25 18
A.1 2007 16 14 21 19
B.2 NA 13 11 16 15
A.2 NA 23 20 26 20
B.2 NA 11 12 22 16
"))
In this case, a tidy dataset should result in columns of Dept, Unit, Year, Quarter,
and Cost. Furthermore, we want to ll in the year column where NAs currently exist.
And well assume that we know the missing value that exists in the Q2 column, and
wed like to update it.
a_mess %>%
fill(Year) %>%
gather(Quarter, Cost, Q1:Q4) %>%
separate(Dep_Unt, into = c("Dept", "Unit")) %>%
replace_na(replace = list(Cost = 17))
##
## Dept Unit Year Quarter Cost
## (chr) (chr) (int) (fctr) (dbl)
## 1 A 1 2006 Q1 15
## 2 B 1 2006 Q1 12
## 3 A 2 2006 Q1 22
## 4 B 2 2006 Q1 12
## 5 A 1 2007 Q1 16
## 6 B 2 2007 Q1 13
## 7 A 2 2007 Q1 23
## 8 B 2 2007 Q1 11
## 9 A 1 2006 Q2 17
## 10 B 1 2006 Q2 13
## ..
This chapter covers most, but not all, of what tidyr provides. There are several
other resources you can check out to learn more.
Data wrangling presentation I gave at Miami University3
Hadley Wickhams tidy data (Wickham, 2014)
tidyr reference manual4
R Studios Data wrangling with R and RStudio webinar5
R Studios Data wrangling cheat sheet6
Bibliography
Wickham, Hadley (2014). Tidy data. Journal of Statistical Software, 59(10) 123.
3
http://rpubs.com/bradleyboehmke/data_processing
4
https://cran.r-project.org/web/packages/tidyr/tidyr.pdf
5
https://www.rstudio.com/resources/webinars/
6
You can get the RStudio cheatsheets at https://www.rstudio.com/resources/cheatsheets/ or within
a working RStudio session by going to Help > Cheatsheets
Chapter 22
Transforming Your Data with dplyr
Transforming your data is a basic part of data wrangling. This can include filtering,
summarizing, and ordering your data by different means. This also includes com-
bining disparate data sets, creating new variables, and many other manipulation
tasks. Although many fundamental data transformation and manipulation functions
exist in R, historically they have been a bit convoluted and lacked a consistent and
cohesive code structure. Consequently, Hadley Wickham developed the very popu-
lar dplyr package to make these data processing tasks more efficient along with a
syntax that is consistent and easier to remember and read.
dplyrs roots originate in the popular plyr package, also produced by Hadley
Wickham. plyr covers data transformation and manipulation for a range of data
structures (data frames, lists, arrays) whereas dplyr is focused on transformation
and manipulation of data frames. And since the bulk of data analysis leverages data
frames I am going to focus on dplyr. Even so, dplyr offers far more functional-
ity than I can cover in one chapter. Consequently, Im going to cover the seven pri-
mary functions dplyr provides for data transformation and manipulation.
Throughout, I also mention additional, useful functions that can be integrated with
these functions. The full list of capabilities can be found in the dplyr reference
manual; I highly recommend going through it as there are many great functions
provided by dplyr that I will not cover here. Also, similar to tidyr, dplyr has
the %>% operator baked in to its functionality.
For most of these examples well use the following census data which includes
the K-12 public school expenditures by state. This data frame currently is 50 16
and includes expenditure data for 14 unique years (50 states and has data through
year 2011). Here I only show you a subset of the data.
## Division State X1980 X1990 X2000 X2001 X2002 X2003
## 1 6 Alabama 1146713 2275233 4176082 4354794 4444390 4657643
## 2 9 Alaska 377947 828051 1183499 1229036 1284854 1326226
## 3 8 Arizona 949753 2258660 4288739 4846105 5395814 5892227
## 4 7 Arkansas 666949 1404545 2380331 2505179 2822877 2923401

220 22 Transforming Your Data with dplyr
## 5 9 California 9172158 21485782 38129479 42908787 46265544 47983402

## 6 8 Colorado 1243049 2451833 4401010 4758173 5151003 5551506
## X2004 X2005 X2006 X2007 X2008 X2009 X2010 X2011
## 1 4812479 5164406 5699076 6245031 6832439 6683843 6670517 6592925
## 2 1354846 1442269 1529645 1634316 1918375 2007319 2084019 2201270
## 3 6071785 6579957 7130341 7815720 8403221 8726755 8482552 8340211
## 4 3109644 3546999 3808011 3997701 4156368 4240839 4459910 4578136
## 5 49215866 50918654 53436103 57352599 61570555 60080929 58248662 57526835
## 6 5666191 5994440 6368289 6579053 7338766 7187267 7429302 7409462
22.1 Selecting Variables of Interest
When working with a sizable data frame, often we desire to only assess specific vari-
ables. The select() function allows you to select and/or rename variables. Lets
say our goal is to only assess the five most recent years worth of expenditure data.
Applying the select() function we can select only the variables of concern.
sub_exp <- expenditures %>% select(Division, State, X2007:X2011)
# for brevity only display first 6 rows

head(sub_exp)
## Division State X2007 X2008 X2009 X2010 X2011
## 1 6 Alabama 6245031 6832439 6683843 6670517 6592925
## 2 9 Alaska 1634316 1918375 2007319 2084019 2201270
## 3 8 Arizona 7815720 8403221 8726755 8482552 8340211
## 4 7 Arkansas 3997701 4156368 4240839 4459910 4578136
## 5 9 California 57352599 61570555 60080929 58248662 57526835
## 6 8 Colorado 6579053 7338766 7187267 7429302 7409462
We can also apply some of the special functions within select(). For instance
we can select all variables that start with X (?select to see the available
functions):
expenditures %>%
select(starts_with("X")) %>%
head()
## X1980 X1990 X2000 X2001 X2002 X2003 X2004 X2005
## 1 1146713 2275233 4176082 4354794 4444390 4657643 4812479 5164406
## 2 377947 828051 1183499 1229036 1284854 1326226 1354846 1442269
## 3 949753 2258660 4288739 4846105 5395814 5892227 6071785 6579957
## 4 666949 1404545 2380331 2505179 2822877 2923401 3109644 3546999
## 5 9172158 21485782 38129479 42908787 46265544 47983402 49215866 50918654
## 6 1243049 2451833 4401010 4758173 5151003 5551506 5666191 5994440
## X2006 X2007 X2008 X2009 X2010 X2011
## 1 5699076 6245031 6832439 6683843 6670517 6592925
## 2 1529645 1634316 1918375 2007319 2084019 2201270
## 3 7130341 7815720 8403221 8726755 8482552 8340211
## 4 3808011 3997701 4156368 4240839 4459910 4578136
## 5 53436103 57352599 61570555 60080929 58248662 57526835
## 6 6368289 6579053 7338766 7187267 7429302 7409462
22.2 Filtering Rows 221
You can also de-select variables by using - prior to name or function. The fol-
lowing produces the inverse of functions above:
expenditures %>% select(-X1980:-X2006)
expenditures %>% select(-starts_with("X"))
And for convenience, you can rename selected variables with two options:
# select and rename a single column
expenditures %>% select(Yr_1980 = X1980)
# Select and rename the multiple variables with an "X" prefix:

expenditures %>% select(Yr_ = starts_with("X"))
# keep all variables and rename a single variable

expenditures %>% rename(`2011` = X2011)
22.2 Filtering Rows
Filtering data is a common task to identify/select observations in which a particular

variable matches a specific value/condition. The filter() function provides this
capability. Continuing with our sub_exp data frame which includes only the
recent 5 years worth of expenditures, we can filter by Division:
sub_exp %>% filter(Division == 3)
## 1 3 Illinois 20326591 21874484 23495271 24695773 24554467
## 2 3 Indiana 9497077 9281709 9680895 9921243 9687949
## 3 3 Michigan 17013259 17053521 17217584 17227515 16786444
## 4 3 Ohio 18251361 18892374 19387318 19801670 19988921
## 5 3 Wisconsin 9029660 9366134 9696228 9966244 10333016
We can apply multiple logic rules in the filter() function such as:
< Less than != Not equal to
> Greater than %in% Group membership
== Equal to is.na is NA
<= Less than or equal to !is.na is not NA
>= Greater than or equal to &,|,! Boolean operators
For instance, we can filter for Division 3 and expenditures in 2011 that were
greater than $10B. This results in Indiana being excluded since it falls within divi-
sion 3 and its expenditures were < $10B (FYIthe raw census data are reported in
units of $1000).
# Raw census data are in units of $1000

sub_exp %>% filter(Division == 3, X2011 > 10000000)
## 1 3 Illinois 20326591 21874484 23495271 24695773 24554467
## 2 3 Michigan 17013259 17053521 17217584 17227515 16786444
## 3 3 Ohio 18251361 18892374 19387318 19801670 19988921
## 4 3 Wisconsin 9029660 9366134 9696228 9966244 10333016
There are additional filtering and subsetting functions that are quite useful:
# remove duplicate rows
sub_exp %>% distinct()
# random sample, 50% sample size without replacement

sub_exp %>% sample_frac(size = 0.5, replace = FALSE)
# random sample of 10 rows with replacement

sub_exp %>% sample_n(size = 10, replace = TRUE)
# select rows 3-5

sub_exp %>% slice(3:5)
# select top n entries - in this case ranks variable X2011 and selects
# the rows with the top 5 values
sub_exp %>% top_n(n = 5, wt = X2011)
22.3 Grouping Data by Categorical Variables
Often, observations are nested within groups or categories and our goal is to per-
form statistical analysis both at the observation level and also at the group level. The
group_by() function allows us to create these categorical groupings.
The group_by() function is a silent function in which no observable manipu-
lation of the data is performed as a result of applying the function. Rather, the only
change youll notice is, when you print the data frame you will notice underneath
the Source information and prior to the actual data frame, an indicator of what vari-
able the data is grouped by will be provided. In the example that follows youll
notice that we grouped by Division and there are nine categories for this vari-
able. The real magic of the group_by() function comes when we perform sum-
mary statistics which we will cover shortly.
group.exp <- sub_exp %>% group_by(Division)
group.exp
## Groups: Division [9]
##
## (int) (chr) (int) (int) (int) (int) (int)
22.4 Performing Summary Statistics on Variables 223
## 1 6 Alabama 6245031 6832439 6683843 6670517 6592925

## 2 9 Alaska 1634316 1918375 2007319 2084019 2201270
## 3 8 Arizona 7815720 8403221 8726755 8482552 8340211
## 4 7 Arkansas 3997701 4156368 4240839 4459910 4578136
## 5 9 California 57352599 61570555 60080929 58248662 57526835
## 6 8 Colorado 6579053 7338766 7187267 7429302 7409462
## 7 1 Connecticut 7855459 8336789 8708294 8853337 9094036
## 8 5 Delaware 1437707 1489594 1518786 1549812 1613304
## 9 5 Florida 22887024 24224114 23328028 23349314 23870090
## 10 5 Georgia 14828715 16030039 15976945 15730409 15527907
## ..
# we can ungroup our data with

ungroup(group.exp)
##
## (int) (chr) (int) (int) (int) (int) (int)
## 1 6 Alabama 6245031 6832439 6683843 6670517 6592925
## 2 9 Alaska 1634316 1918375 2007319 2084019 2201270
## 3 8 Arizona 7815720 8403221 8726755 8482552 8340211
## 4 7 Arkansas 3997701 4156368 4240839 4459910 4578136
## 5 9 California 57352599 61570555 60080929 58248662 57526835
## 6 8 Colorado 6579053 7338766 7187267 7429302 7409462
## 7 1 Connecticut 7855459 8336789 8708294 8853337 9094036
## 8 5 Delaware 1437707 1489594 1518786 1549812 1613304
## 9 5 Florida 22887024 24224114 23328028 23349314 23870090
## 10 5 Georgia 14828715 16030039 15976945 15730409 15527907
## ..
22.4 Performing Summary Statistics on Variables
Obviously the goal of all this data wrangling is to be able to perform statistical
analysis on our data. The summarise() function allows us to perform the major-
ity of summary statistics when performing exploratory data analysis.
Lets get the mean expenditure value across all states in 2011:
sub_exp %>% summarise(Mean_2011 = mean(X2011))
## Mean_2011
## 1 10513678
Not too bad, lets get some more summary stats:
sub_exp %>% summarise(Min = min(X2011, na.rm = TRUE),

Median = median(X2011, na.rm = TRUE),
Mean = mean(X2011, na.rm = TRUE),
Var = var(X2011, na.rm = TRUE),
SD = sd(X2011, na.rm = TRUE),
Max = max(X2011, na.rm = TRUE),
N = n())
## Min Median Mean Var SD Max N

## 1 1049772 6527404 10513678 1.48619e+14 12190938 57526835 50
This information is useful, but being able to compare summary statistics at mul-
tiple levels is when you really start to gather some insights. This is where the
group_by() function comes in. First, lets group by Division and see how the
different regions compare across years 2010 and 2011.
sub_exp %>%
group_by(Division)%>%
summarise(Mean_2010 = mean(X2010, na.rm = TRUE),
Mean_2011 = mean(X2011, na.rm = TRUE))
##
## Division Mean_2010 Mean_2011
## (int) (dbl) (dbl)
## 1 1 5121003 5222277
## 2 2 32415457 32877923
## 3 3 16322489 16270159
## 4 4 4672332 4672687
## 5 5 10975194 11023526
## 6 6 6161967 6267490
## 7 7 14916843 15000139
## 8 8 3894003 3882159
## 9 9 15540681 15468173
Now were starting to see some differences pop out. How about we compare
states within a Division? We can start to apply multiple functions weve learned so
far to get the 5 year average for each state within Division 3.
library(tidyr)
sub_exp %>%
gather(Year, Expenditure, X2007:X2011) %>% # turn wide data to long
filter(Division == 3) %>% # only assess Division 3
group_by(State) %>% # summarize data by state
summarise(Mean = mean(Expenditure), # calculate mean & SD
SD = sd(Expenditure))
##
## State Mean SD
## (chr) (dbl) (dbl)
## 1 Illinois 22989317 1867527.7
## 2 Indiana 9613775 238971.6
## 3 Michigan 17059665 180245.0
## 4 Ohio 19264329 705930.2
## 5 Wisconsin 9678256 507461.2
There are several built-in summary functions in dplyr as displayed below. You
can also build in your own functions as well.
22.5 Arranging Variables by Value 225
Built-in Summary Functions
22.5 Arranging Variables by Value
Sometimes we wish to view observations in rank order for a particular variable(s).

The arrange() function allows us to order data by variables in ascending or
descending order. Lets say we want to assess the average expenditures by division.
We could apply the arrange() function at the end to order the divisions from low-
est to highest expenditure for 2011. This makes it easier to see the significant differ-
ences between Divisions 8, 4, 1 and 6 as compared to Divisions 5, 7, 9, 3 and 2.
sub_exp %>%
Mean_2011 = mean(X2011, na.rm = TRUE)) %>%
arrange(Mean_2011)
##
## 1 8 3894003 3882159
## 2 4 4672332 4672687
## 3 1 5121003 5222277
## 4 6 6161967 6267490
## 5 5 10975194 11023526
## 6 7 14916843 15000139
## 7 9 15540681 15468173
## 8 3 16322489 16270159
## 9 2 32415457 32877923
We can also apply a descending argument to rank-order from highest to lowest.

The following shows the same data but in descending order by applying desc()
within the arrange() function.
sub_exp %>%
Mean_2011 = mean(X2011, na.rm = TRUE)) %>%
arrange(desc(Mean_2011))

##
## 1 2 32415457 32877923
## 2 3 16322489 16270159
## 3 9 15540681 15468173
## 4 7 14916843 15000139
## 5 5 10975194 11023526
## 6 6 6161967 6267490
## 7 1 5121003 5222277
## 8 4 4672332 4672687
## 9 8 3894003 3882159
22.6 Joining Data Sets
Often we have separate data frames that can have common and differing variables
for similar observations and we wish to join these data frames together. dplyr
offers multiple joining functions (xxx_join()) that provide alternative ways to
join data frames:
inner_join()
left_join()
right_join()
full_join()
semi_join()
anti_join()
Our public education expenditure data represents then-year dollars. To make any
accurate assessments of longitudinal trends and comparisons we need to adjust for
inflation. I have the following data frame which provides inflation adjustment fac-
tors for base-year 2012 dollars.
## Year Annual Inflation
## 28 2007 207.342 0.9030811
## 29 2008 215.303 0.9377553
## 30 2009 214.537 0.9344190
## 31 2010 218.056 0.9497461
## 32 2011 224.939 0.9797251
## 33 2012 229.594 1.0000000
To join to my expenditure data I obviously need to get my expenditure data in the

proper form that allows me to join these two data frames. I can apply the following
functions to accomplish this:
long_exp <- sub_exp %>%

gather(Year, Expenditure, X2007:X2011) %>%
separate(Year, into=c("x", "Year"), sep = "X") %>%
22.6 Joining Data Sets 227
select(-x) %>%
mutate(Year = as.numeric(Year))
head(long_exp)
## Division State Year Expenditure
## 1 6 Alabama 2007 6245031
## 2 9 Alaska 2007 1634316
## 3 8 Arizona 2007 7815720
## 4 7 Arkansas 2007 3997701
## 5 9 California 2007 57352599
## 6 8 Colorado 2007 6579053
I can now apply the left_join() function to join the inflation data to the
expenditure data. This aligns the data in both data frames by the Year variable and
then joins the remaining inflation data to the expenditure data frame as new
variables.
join_exp <- long_exp %>% left_join(inflation)
head(join_exp)
## Division State Year Expenditure Annual Inflation
## 1 6 Alabama 2007 6245031 207.342 0.9030811
## 2 9 Alaska 2007 1634316 207.342 0.9030811
## 3 8 Arizona 2007 7815720 207.342 0.9030811
## 4 7 Arkansas 2007 3997701 207.342 0.9030811
## 5 9 California 2007 57352599 207.342 0.9030811
## 6 8 Colorado 2007 6579053 207.342 0.9030811
To illustrate the other joining methods we can use the a and b data frames from
the EDAWR package1:
library(EDAWR)
a
## x1 x2
## 1 A 1
## 2 B 2
## 3 C 3
b
## x1 x2
## 1 A TRUE
## 2 B FALSE
## 3 D TRUE
# include all of a, and join matching rows of b

left_join(a, b, by = "x1")
## x1 x2.x x2.y
## 1 A 1 TRUE
## 2 B 2 FALSE
## 3 C 3 NA
1
The EDAWR package contains multiple data sets and can be downloaded by executing
devtools::install_github(rstudio/EDAWR)
# include all of b, and join matching rows of a

right_join(a, b, by = "x1")
## x1 x2.x x2.y
## 1 A 1 TRUE
## 2 B 2 FALSE
## 3 D NA TRUE
# join data, retain only matching rows in both data frames

inner_join(a, b, by = "x1")
## x1 x2.x x2.y
## 1 A 1 TRUE
## 2 B 2 FALSE
# join data, retain all values, all rows

full_join(a, b, by = "x1")
## x1 x2.x x2.y
## 1 A 1 TRUE
## 2 B 2 FALSE
## 3 C 3 NA
## 4 D NA TRUE
# keep all rows in a that have a match in b

semi_join(a, b, by = "x1")
## x1 x2
## 1 A 1
## 2 B 2
# keep all rows in a that do not have a match in b

anti_join(a, b, by = "x1")
## x1 x2
## 1 C 3
There are additional dplyr functions for merging data sets worth exploring:
intersect(y, z) # Rows that appear in both y and z
union(y, z) # Rows that appear in either or both y and z
setdiff(y, z) # Rows that appear in y but not z
bind_rows(y, z) # Append z to y as new rows
bind_cols(y, z) # Append z to y as new columns
22.7 Creating New Variables
Often we want to create a new variable that is a function of the current variables in
our data frame or we may just want to add a new variable that is external to our
existing variables. The mutate() function allows us to add new variables while
preserving the existing variables. If we go back to our previous join_exp
dataframe, remember that we joined inflation rates to our non-inflation adjusted
expenditures for public schools. The dataframe looks like:
22.7 Creating New Variables 229
## Division State Year Expenditure Annual Inflation

## 1 6 Alabama 2007 6245031 207.342 0.9030811
## 2 9 Alaska 2007 1634316 207.342 0.9030811
## 3 8 Arizona 2007 7815720 207.342 0.9030811
## 4 7 Arkansas 2007 3997701 207.342 0.9030811
## 5 9 California 2007 57352599 207.342 0.9030811
## 6 8 Colorado 2007 6579053 207.342 0.9030811
If we wanted to adjust our annual expenditures for inflation we can use mutate()
to create a new inflation adjusted cost variable which well name Adj_Exp:
inflation_adj <- join_exp %>% mutate(Adj_Exp = Expenditure / Inflation)
head(inflation_adj)
## Division State Year Expenditure Annual Inflation Adj_Exp
## 1 6 Alabama 2007 6245031 207.342 0.9030811 6915249
## 9 Alaska 2007 1634316 207.342 0.9030811 1809711
## 3 8 Arizona 2007 7815720 207.342 0.9030811 8654505
## 4 7 Arkansas 2007 3997701 207.342 0.9030811 4426735
## 5 9 California 2007 57352599 207.342 0.9030811 63507696
## 6 8 Colorado 2007 6579053 207.342 0.9030811 7285119
Lets say we wanted to create a variable that rank-orders state-level expenditures

(inflation adjusted) for the year 2010 from the highest level of expenditures to the
lowest.
rank_exp <- inflation_adj %>%
filter(Year == 2010) %>%
arrange(desc(Adj_Exp)) %>%
mutate(Rank = 1:length(Adj_Exp))
head(rank_exp)
## Division State Year Expenditure Annual Inflation Adj_Exp Rank
## 1 9 California 2010 58248662 218.056 0.9497461 61330774 1
## 2 2 New York 2010 50251461 218.056 0.9497461 52910417 2
## 3 7 Texas 2010 42621886 218.056 0.9497461 44877138 3
## 4 3 Illinois 2010 24695773 218.056 0.9497461 26002501 4
## 5 2 New Jersey 2010 24261392 218.056 0.9497461 25545135 5
## 6 5 Florida 2010 23349314 218.056 0.9497461 24584797 6
If you wanted to assess the percent change in cost for a particular state you can
use the lag() function within the mutate() function:
inflation_adj %>%
filter(State == "Ohio") %>%
mutate(Perc_Chg = (Adj_Exp - lag(Adj_Exp)) / lag(Adj_Exp))
## Division State Year Expenditure Annual Inflation Adj_Exp Perc_Chg
## 1 3 Ohio 2007 18251361 207.342 0.9030811 20210102 NA
## 2 3 Ohio 2008 18892374 215.303 0.9377553 20146378 -0.003153057
## 3 3 Ohio 2009 19387318 214.537 0.9344190 20747992 0.029862103
## 4 3 Ohio 2010 19801670 218.056 0.9497461 20849436 0.004889357
## 5 3 Ohio 2011 19988921 224.939 0.9797251 20402582 -0.021432441
You could also look at what percent of all US expenditures each state made up in
2011. In this case we use mutate() to take each states inflation adjusted expen-
diture and divide by the sum of the entire inflation adjusted expenditure column. We
also apply a second function within mutate() that provides the cumulative per-
cent in rank-order. This shows that in 2011, the top 8 states with the highest expen-
ditures represented over 50 % of the total U.S. expenditures in K-12 public schools.
(I remove the non-inflation adjusted Expenditure, Annual & Inflation columns so
that the columns dont wrap on the screen view)
cum_pct <- inflation_adj %>%
mutate(Pct_of_Total = Adj_Exp/sum(Adj_Exp),
Cum_Perc = cumsum(Pct_of_Total)) %>%
select(-Expenditure, -Annual, -Inflation)
head(cum_pct, 8)
## Division State Year Adj_Exp Pct_of_Total Cum_Perc
## 1 9 California 2011 58717324 0.10943237 0.1094324
## 2 2 New York 2011 52575244 0.09798528 0.2074177
## 3 7 Texas 2011 43751346 0.08154005 0.2889577
## 4 3 Illinois 2011 25062609 0.04670957 0.3356673
## 5 5 Florida 2011 24364070 0.04540769 0.3810750
## 6 2 New Jersey 2011 24128484 0.04496862 0.4260436
## 7 2 Pennsylvania 2011 23971218 0.04467552 0.4707191
## 8 3 Ohio 2011 20402582 0.03802460 0.5087437
An alternative to mutate() is transmute() which creates a new variable

and then drops the other variables. In essence, it allows you to create a new data
frame with only the new variables created. We can perform the same string of
functions as above but this time use transmute to only keep the newly created
variables.
inflation_adj %>%
transmute(Pct_of_Total = Adj_Exp/sum(Adj_Exp),
Cum_Perc = cumsum(Pct_of_Total)) %>%
head()
## Pct_of_Total Cum_Perc
## 1 0.10943237 0.1094324
## 2 0.09798528 0.2074177
## 3 0.08154005 0.2889577
## 4 0.04670957 0.3356673
## 5 0.04540769 0.3810750
## 6 0.04496862 0.4260436
Lastly, you can apply the summarise and mutate functions to multiple col-
umns by using summarise_each() and mutate_each() respectively.
22.7 Creating New Variables 231
# calculate the mean for each division with summarise_each

# call the function of interest with the funs() argument
sub_exp %>%
select(-State) %>%
group_by(Division) %>%
summarise_each(funs(mean)) %>%
head()
##
## Division X2007 X2008 X2009 X2010 X2011
## (int) (dbl) (dbl) (dbl) (dbl) (dbl)
## 1 1 4680691 4952992 5173184 5121003 5222277
## 2 2 28844158 30652645 31304697 32415457 32877923
## 3 3 14823590 15293644 15895459 16322489 16270159
## 4 4 4175766 4425739 4658533 4672332 4672687
## 5 5 10230416 10857410 11018102 10975194 11023526
## 6 6 5584277 6023424 6076507 6161967 6267490
# for each division calculate the percent of total

# expenditures for each state across each year
sub_exp %>%
select(-State) %>%
group_by(Division) %>%
mutate_each(funs(. / sum(.))) %>%
head()
## Groups: Division [4]
##
## Division X2007 X2008 X2009 X2010 X2011
## (int) (dbl) (dbl) (dbl) (dbl) (dbl)
## 1 6 0.27958099 0.28357787 0.27498705 0.27063262 0.26298109
## 2 9 0.02184221 0.02387438 0.02515947 0.02682018 0.02846193
## 3 8 0.28093187 0.27793321 0.28144201 0.27229536 0.26854292
## 4 7 0.07854895 0.07565703 0.07402700 0.07474621 0.07630156
## 5 9 0.76650258 0.76625202 0.75304632 0.74962818 0.74380904
## 6 8 0.23648054 0.24272678 0.23179279 0.23848536 0.23857413
Similar to the summary function, dplyr allows you to build in your own func-
tions to be applied within mutate_each() and also has the following built in
functions that can be applied.
Built-in Functions for mutate_each()

This chapter introduced you to dplyrs basic set of tools and demonstrated how to
use them on data frames. Additional resources are available that go into more detail
or provide additional examples of how to use dpyr. In addition, there are other
resources that illustrate how dplyr can perform tasks not mentioned in this chapter
such as connecting to remote databases and translating your R code into SQL code
for data pulls.
Data wrangling presentation I gave at Miami University2
dplyr reference manual3
R Studios Data wrangling with R and RStudio webinar4
R Studios Data wrangling cheat sheet5
Hadley Wickhams dplyr tutorial at useR! 2014, Part 16
Hadley Wickhams dplyr tutorial at useR! 2014, Part 27
2
http://rpubs.com/bradleyboehmke/data_processing
3
https://cran.r-project.org/web/packages/dplyr/dplyr.pdf
4
https://www.rstudio.com/resources/webinars/
5
You can get the RStudio cheatsheets at https://www.rstudio.com/resources/cheatsheets/or within
a working RStudio session by going to Help > Cheatsheets
6
https://www.youtube.com/watch?v=8SGif63VW6E
7
https://www.youtube.com/watch?v=Ue08LVuk790
Index
Function colsums, 195, 196

comment, 94
complete.cases, 115
A content, 158, 161
abbreviate, 47 createSheet, 165, 166, 168
addDataFrame, 165, 167 createWorkbook, 165167
all.equal, 39
anti_join, 226
apply, 191, 196 D
arrange, 225 data.frame, 81, 82, 106, 107
as.character, 42, 61, 159 dbinom, 36
as.data.frame, 106, 107 dexp, 36
as.Date, 72 dgamma, 37
as.double, 32 dhours, 77, 78
as.factor, 68 dimnames, 102
as.integer, 32 dminutes, 77
as.logical, 127 dnorm, 35
as.numeric, 32, 227 download.le, 130, 131
as.POSIXct, 75, 76 dpois, 36
as.vector, 23, 86 droplevels, 69
attr, 94 dseconds, 77, 78
attributes, 82 dyears, 77
B E
blsAPI, 152 environment, 173
body, 173 extract_numeric, 216
break, 183, 189
F
C factor, 67
cat, 43 ll, 216
cbind, 100, 107, 108 lter, 154, 199202, 204, 205, 221
ceiling, 40 oor, 40
chartr, 46, 47 for, 23
class, 67 force_tz, 78
colnames, 110 formals, 173

B.C. Boehmke, Data Wrangling with R, Use R!, DOI 10.1007/978-3-319-45599-0
234 Index
fromJSON, 152 mean, 26, 35, 39

full_join, 226, 228 month, 74
function, 14, 16, 18 mutate, 228230
mutate_each, 230, 231
G
gather, 18, 212, 213 N
GET, 158 na.omit, 116
getHTMLLinks, 132 names, 87, 93, 110
getNodeSet, 148 nchar, 46, 50
getURL, 148 ncol, 106
getwd, 14 new_duration, 77
grep, 5860 next, 183, 190
grepl, 60, 61, 63 noquote, 43
group_by, 222, 224 now, 76, 77
gsub, 56, 57, 62, 65 nrow, 106
H O
help, 16, 17 oauth_endpoints, 160
history, 14 oauth1.0_token, 161
html_nodes, 135, 140, 143, 144 OlsonNames, 76
htmlParse, 148 options, 15, 16, 21
html_text, 135138, 140142
Iidentical, 39, 53
if, 178 P
ifelse, 185 paste, 41, 49
inner_join, 226, 228 paste0, 49
install.packages, 18, 49, 63 pbinom, 35
intersect, 52, 228 pexp, 36
is.character, 42 pgamma, 37
is.element, 54 pnorm, 35
is.na, 113 ppois, 36
ISOdate, 73 print, 23, 43
L Q
lapply, 192193 qbinom, 36
left_join, 226, 227 qexp, 36
length, 24, 45, 49, 50 qgamma, 37
levels, 67 qnorm, 35
library, 18, 19, 49, 63, 71, 72, 7477, 122, qpois, 36
124126, 130, 132, 135, 142, 144, 146,
148, 151154, 156, 158, 159, 162, 164,
165, 167, 200, 202, 212, 216, 224, 227 R
list, 91, 92, 133, 145, 152 rbind, 100, 107, 108
list.les, 131 rbinom, 35, 188
load, 15, 127 read.csv, 105, 119122
ls, 14 read.delim, 119, 121
read.table, 105, 119, 121
read.xls, 127
M read.xlsx, 124
matrix, 99 read_csv, 122
mday, 74 read_excel, 126, 127
mdy, 72 read_fwf, 123
Index 235
read_html, 135, 144 sub, 56

read_table, 123 substr, 47, 50
readRDS, 127, 169 substring, 47, 48
regexpr, 60, 61 sum, 38, 61, 114
rep, 33, 149 summarise, 223
repeat, 183, 189, 216 summarise_each, 230
replace_na, 216, 217 summary, 35, 69, 196, 202,
revalue, 69 204, 205
rexp, 36 Sys.Date, 71, 76
rgamma, 37 Sys.time, 71
right_join, 226, 228 Sys.timezone, 71
rm, 14
rnorm, 35, 37, 192
round, 114, 178, 193 T
rownames, 102 tapply, 194195
rpois, 36, 187 temple, 131
runif, 34 tolower, 46
toString, 42
toupper, 46
S transmute, 230
sample, 34 typeof, 31
sapply, 160, 193194
save, 15, 169
save.image, 15, 169 U
saveRDS, 169 unclass, 68
saveWorkbook, 165, 167, 168 ungroup, 223
search, 16, 19, 157 union, 52, 228
select, 154, 220 unite, 214, 215
semi_join, 226, 228 unlink, 131
separate, 214 unlist, 49, 65, 142, 159, 160
seq, 33, 75 unzip, 131
set.seed, 37 update, 74
setdiff, 52
setequal, 53
setwd, 14 V
sort, 54 vignette, 19
spread, 213
sprint, 43, 44
str, 81 W
str_c, 49 wday, 74
str_detect, 63 which, 38, 114
str_dub, 51 while, 183, 187, 188
str_extract, 64 with_tz, 77
str_extract_all, 64 write.csv, 163, 164
str_length, 50 write.delim, 163, 164
str_locate, 63 write.table, 163
str_locate_all, 64 write.xlsx, 165
str_pad, 52 write_csv, 164
str_replace, 65 write_delim, 165
str_replace_all, 65
str_sub, 49, 50
str_trim, 51 Y
stringsAsFactors, 106, year, 74
120, 122 ymd, 72, 74
strsplit, 4749, 62, 65 ymd_hms, 77
236 Index
Words Double, 7, 31, 56, 58, 85, 88, 90, 95, 127
Dplyr package, 195, 219
Duration, 76, 77
A
Abbreviate, 47
API key, 151, 153, 155, 157, 158, 160 E
Application-programming interface (API), Element equality, 53
129, 150162 Element selector, 139, 148
Apply family, 190 Evaluation, 177, 190
Arguments, 18, 20, 44, 50, 68, 72, 73, 75, Exact equality, 39, 5354
104, 112, 114, 120126, 130, 147, Excel, 105, 119, 123127, 129134, 163,
149, 153, 154, 161, 162, 164, 165169, 216
173179, 191, 192, 194, 201, 204, Exponential distribution, 36
206, 214, 225, 231 Exporting data, 4, 15
Assignment, 19, 20, 114, 205 Extract dates, 7375
Attributes, 8183, 85, 8788, 91, 9395, 99, Extracting patterns, 64
101103, 105, 109111, 212, 217 Extract substrings, 4749
B F
Binomial distribution, 3436 Factors, 6769, 105, 106, 120, 126, 149, 194,
BlsAPI package, 151153 196, 226
Filtering data, 221
Floating point numbers, 31
C for loop, 22, 23
Calculator, 15, 21 Function components, 173175
Case conversion, 46 Functional programming language,
Character replacement, 4647 173, 201
Character strings, 4, 16, 41, 48, 5255, 62, 65,
66, 68, 72, 136138, 141, 142, 164
Comparison operators, 38 G
Console, 11, 1316, 72, 203, 207 Gamma distribution, 37
CRAN, 11, 18 gdata package, 123, 130
Create dates, 73 Getting help, 3
Creating new variables, 219 Grouping data, 222223
CSV, 119121, 123, 130, 163, 164
Current date & time, 71
H
HTML, 134150
D httr package, 150, 151, 158162, 203
Data frame, 13, 14, 82, 105107, 110, 112,
113, 115, 122, 144, 146, 148, 163, 165,
169, 191, 193, 195, 196, 205, 212214, I
216, 219, 226228, 230, 232 if statement, 179, 184, 189
Data structures, 4, 81, 83, 89, 95, 105, 137, ifelse statement, 184
159, 186, 219 Importing data, 119, 123, 163, 165
Data wrangling, 3, 4, 14, 66, 199, 201, 219, Integer, 22, 31, 32, 39, 44, 67, 68, 82, 85, 88,
223 91, 107, 126, 127, 189
Dates, 4, 7178, 126, 127, 154, 214, 215 Invalid parameters, 178179
Date sequences, 71, 7576
Daylight savings, 71, 7678
Detecting patterns, 63 J
Dimensions, 4, 82, 86, 99, 101, 103, 104, 112 Joining data, 226228
Index 237
L R
Lazy evaluation, 177 R, 3, 4, 79, 1117, 1925, 31, 32, 34, 41, 43,
Levels, 8, 39, 6769, 129, 135, 149, 159, 162, 4649, 51, 52, 55, 59, 62, 63, 66, 67,
173, 175, 222, 224, 229 71, 72, 76, 77, 81, 85, 91, 99, 105,
List, 9, 1317, 19, 49, 60, 64, 65, 72, 76, 119123, 125, 127130, 143, 148,
9193, 9597, 105, 106, 110, 111, 113, 150152, 157, 158, 162165, 169,
128, 130, 133, 134, 136138, 141, 144, 173, 175, 177, 181, 183, 186, 190,
146, 151, 152, 158, 159, 161, 173, 178, 191, 193, 196, 197, 201, 207, 218,
183, 190, 192194, 203, 219 219, 232
Locating patterns, 6364 r2excel package, 163, 167
Logical operators, 37, 203 RCurl package, 148
Long data, 213 readr package, 122123, 125, 163165
Loop control statements, 4, 183 readxl package, 125127, 165
Lubridate package, 73, 75, 76 Regular expressions, 4, 55, 60, 216
Repeat loop, 183, 189
Replace substrings, 47, 48
M Replacing patterns, 65
Magrittr package, 199, 203, 204, 207 Reproducible, 31, 37
Manipulate dates, 7375 rjson package, 152
Matrix, 99101, 103, 106, 107, 109, 187, 190, rnoaa package, 150, 153
191, 193, 195, 196 R object, 169
Metacharacters, 56 rOpenSci, 8
Missing values, 4, 113116, 164, 179 Rounding, 3940, 174
RStudio, 3, 8, 1113, 16, 19, 26, 135, 232
rtimes package, 155157
N rvest package, 135, 143146, 150
Naming, 2425, 92, 148, 159, 163, 203
Near equality, 37, 39
Nested list, 86, 92, 97, 190, 200, 222 S
Normal distribution, 34, 35, 37 Scoping rules, 175
Scraping data, 4, 105, 134, 143
Script editor, 13
O Seed, 31, 37
OAuth, 151, 158, 160162 Selecting variables, 220221
Open source, 7, 8, 24 Sequence of non-random numbers, 3233
Order data, 225 Sequence of random numbers, 3337
Organization, 4, 7, 8, 25, 150, 151, 158, 165 sequences, 25, 3237, 41, 5557, 75, 85, 183,
207, 211
Set intersection, 5253
P Set union, 52
Packages, 3, 9, 11, 1519, 71, 119, 123, 125, 130, Simplifying, 8890, 95, 96
150, 151, 157, 158, 163, 196, 203, 207 Sorting, 4, 154
Pattern matching, 55, 60 Sourcing functions, 173
Pattern replacement, 60 String manipulation, 46, 63
Pipe operator, 4, 135, 201, 204, 207 String splitting, 6566
Poisson distribution, 36, 187 stringr package, 41, 46
POSIX, 55, 5859 Style guide, 24, 25, 27
POSIXct, 75, 126 Subsetting, 8890, 93, 9597, 103104,
Preserving, 89, 95, 228 111112, 114, 136, 155, 159, 222
Printing strings, 4346 Summary functions, 224, 231
Summary statistics, 196, 222225
Syntax, 11, 13, 23, 46, 5559, 73, 77, 135,
Q 136, 150, 184, 186, 191, 192, 195,
Quantiers, 55, 59 219
238 Index
T 111, 113115, 137, 179, 184, 186, 191,

tidyr package, 18 193, 194
Time zones, 71, 7678 Vectorization, 3, 11, 22
TXT, 121, 123, 129
W
U while loop, 187, 188
Uniform numbers, 34 whitespace, 46, 52, 120
Wide data, 212213
Workspace environment, 13, 14
V
Vector, 13, 14, 2224, 31, 32, 35, 37, 38, 43,
44, 46, 47, 5254, 6065, 67, 68, 75, X
85, 87, 90, 99, 100, 105, 106, 108, 110, xlsx package, 123125
XML package, 132

Data Wrangling With R

Uploaded by

Copyright:

Available Formats

Data Wrangling With R

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Data Wrangling With R

Uploaded by

Copyright:

Available Formats

What are some of the main data types discussed in R?

What are some of the main data types discussed in R?

What are some common operations that can be performed on vectors in R?

What are some common operations that can be performed on vectors in R?

UseR !

More information about this series at http://www.springer.com/series/6991

Data Wrangling with R

ISSN 2197-5736 ISSN 2197-5744 (electronic)

Library of Congress Control Number: 2016953509

Springer International Publishing Switzerland 2016

Printed on acid-free paper

This Springer imprint is published by Springer Nature

Who This Book Is for

What You Need for This Book

3.7 Styling Guide ................................................................................... 24

Part II Working with Different Types of Data in R

5.4 Set Operatons for Character Strings ................................................ 52

Part III Managing Data Structures in R

Part IV Importing, Scraping, and Exporting Data with R

15.3 Load Data from Saved R Object File ............................................. 127

Part V Creating Efficient and Readable Code in R

19.1.6 break Function to Exit a Loop.......................................... 189

Part VI Shaping and Transforming Your Data with R

Index ................................................................................................................. 233

Get Clean Transform Communicate

Fig. 1 Analytic Process

Water, water, everywhere, nor any a drop to drink

Springer International Publishing Switzerland 2016 3

Fig. 1.1 Data Wrangling

2.1 Open Source

Springer International Publishing Switzerland 2016 7

Although the history of S and R is interesting,2 the principal artifact to observe is

Many researchers in academic institutions are using and developing R code to

Programming is like kicking yourself in the face, sooner or later

A computer language is described by its syntax and semantics; where syntax is

3.1 Installing R and RStudio

Springer International Publishing Switzerland 2016 11

4. Follow the instructions of the installer

Fig. 3.1 RStudio Icon

Fig. 3.2 RStudio Console

3.2 Understanding the Console

3.2.1 Script Editor

3.2.2 Workspace Environment

Fig. 3.3 Four fundamental windows of the RStudio console

# set the working directory to a specified directory

For example, if I call getwd() the le path /Users/bradboehmke/Desktop/

# identify if an R object with a given name is present

# remove defined object from the environment

# you can remove multiple objects by using the c() function

# basically removes everything in the working environment -- use with

# show 100 most recent commands

# show entire saved history

# save specified objects to a .RData file

3.2.4 Misc. Displays

3.2.5 Workspace Options and Shortcuts

# view current option settings

# change a specific option (i.e. number of digits to print on output)

3.3 Getting Help

3.3.1 General Help

To leverage general help resources you can use: