-
Notifications
You must be signed in to change notification settings - Fork 0
/
02-manipulating-data.qmd
331 lines (250 loc) · 14.8 KB
/
02-manipulating-data.qmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
# Reading & Manipulating Data {#sec-chap02}
```{r}
#| label: setup
#| results: hold
#| include: false
base::source(file = "R/helper.R")
ggplot2::theme_set(ggplot2::theme_bw())
options(show.signif.stars = FALSE)
```
## Table of content for chapter 02
::: {#obj-chap02}
::: my-objectives
::: my-objectives-header
Chapter section list
:::
::: my-objectives-container
- Data Input
- Accessing data from a package (@sec-chap02-1-1)
- ~~Entering a data frame directly (empty)~~ (@sec-chap02-1-2)
- ~~Reading data from plain-text files (empty)~~ (@sec-chap02-1-3)
- ~~File and Paths (empty)~~ {@sec-chap02-1-4}
- ~~Exporting or saving a data frame to a file (empty)~~ (@sec-chap02-1-5)
- Reading and writing other file formats (@sec-chap02-1-6)
- Other Approaches to Reading and Managing Data Sets in R (@sec-chap02-2)
- Working with data frames (@sec-chap02-3)
- ~~How the R interpreter finds objects (empty)~~ (@sec-chap02-3-1)
- Missing data (@sec-chap02-3-2)
- Modifying and transforming data (@sec-chap02-3-3)
- ~~Binding rows and columns (empty)~~ (@sec-chap02-3-4)
- ~~Aggregating data frames (empty)~~ (@sec-chap02-3-5)
- ~~Merging data frames (empty)~~ (@sec-chap02-3-6)
- ~~Reshaping data (empty)~~ (@sec-chap02-3-7)
- ~~Working with matrices, arrays, and lists (empty)~~ (@sec-chap02-4)
- ~~Dates and times (empty)~~ (@sec-chap02-5)
- ~~Character data (empty)~~ (@sec-chap02-6)
- ~~Large datasets in R (empty)~~ (@sec-chap02-7)
:::
:::
:::
## Data Input
### Accessing data from a package {#sec-chap02-1-1}
There are two ways to access data from a package:
- Loading the package with `base::library(<packageName>)`
- Accessing with
`utils::data("<dataName>", package = "<packageName>")` without first
loading the package
::: my-r-code
::: my-r-code-header
::: {#cnj-chap02-load-data-display-extract}
: Load data and display a data extract with different methods
:::
:::
::: my-r-code-container
::: {#lst-chap02-load-data-display-extract}
```{r}
#| label: load-data-display-extract
#| results: hold
utils::data("Davis", package = "carData")
glue::glue("---------------------------------------------------------------------")
glue::glue("utils::str: Compactly Display the Structure of an Arbitrary R Object")
glue::glue("---------------------------------------------------------------------")
utils::str(Davis)
glue::glue(" ")
glue::glue("---------------------------------------------------------------------")
glue::glue("car::brief: Print Abbreviated Ouput")
glue::glue("---------------------------------------------------------------------")
car::brief(Davis)
glue::glue(" ")
glue::glue("---------------------------------------------------------------------")
glue::glue("my_glance_data: my own function")
glue::glue("---------------------------------------------------------------------")
my_glance_data(Davis)
glue::glue(" ")
glue::glue("---------------------------------------------------------------------")
glue::glue("dplyr::glimpse: Get a glimpse of your data")
glue::glue("---------------------------------------------------------------------")
dplyr::glimpse(Davis)
```
Accessing data from a package and using different methods for displaying
a data extract
:::
------------------------------------------------------------------------
There is a problem with the data files from the {**carData**} package:
The files have relevant information in the row names. But row names are
not part of the data frame. This can be easily seen by applying
functions like `utils::str()` or `dplyr::glimpse()`.
This is not a great deal here with the Davis data set as it only
contains the index of the row numbers. But it is an annoying "feature"
when it has important information that belong into a column of the data
frame. We have already met such a data frame with `Duncan` in
@lst-chap01-get-duncan-data.
See @sec-chap02-2 for a more detailed treatment and why I need row names
using the {**car**} package.
:::
:::
Some packages require that you call the `utils::data()` function to use
data frames in the package, even when the package is loaded. Other
packages, like {**carData**}, use R’s lazy data mechanism to provide
direct access to data frames when the package is loaded. On a package’s
help page, accessed, for example, by `help(package="carData")`, click on
the link for the package `DESCRIPTION file`. Data frames are
automatically available if the line `LazyData: yes` appears in the
package description.
### Entering a data frame directly (empty) {#sec-chap02-1-2}
### Reading data from plain-text files (empty) {#sec-chap02-1-3}
### File and Paths (empty) {#sec-chap02-1-4}
### Exporting or saving a data frame to a file (empty) {#sec-chap02-1-5}
### Reading and writing other file formats {#sec-chap02-1-6}
The {**car**} package includes a function named `car::Import()` that
calls the `import()` function in {**rio**} but uses different defaults
and arguments, more closely matching the specifications that the authors
expect users of the {**car**} package to prefer. In particular:
1. **Row names**: The `rio::import()` function does not support row
names, while `car::Import()` allows you to convert a column in the
input data file into row names. By default, `car::Import()` assumes
that the first character column with unique values in the input data
set represents row names. If you do not want `car::Import()` to use
row names, set the argument `row.names = FALSE`.
2. **Conversion to factors**: The `rio::import()` function does not
support automatic conversion of character data to factors, while
`car::Import()` by default converts character variables to factors.
To suppress converting character variables to factors, add the
argument `stringsAsFactors = FALSE`.
The `car::Export()` function similarly writes data files in various
formats. `car::Export()` is identical in use to the `rio::export()` but
`car::Export()` has an additional argument, `keep.row.names`: Setting
`keep.row.names = TRUE` adds the row names in a data frame as an
additional initial column of the output data set, with the column name
`id`.
::: {#wrn-prefer-tidyverse .callout-warning}
##### My preferences are {**readr**}, {**haven**} and {**foreign**}
As I am trying to stick with the {**tidyverse**} approach and not to use
different packages for the same task, I will neither use the {**rio**}
nor the {**car**} file import/export commands.
My preferences are {**readr**}, {**haven**} and {**foreign**} (in that
order).
:::
## Other approaches to reading and managing data sets in R {#sec-chap02-2}
Fox & Weisberg are not using the tidyverse file commands for two
reasons:
1. {**tidyverse**} packages are actively antagonistic to the use of row
names.
2. {**tidyverse**} packages does not automatically convert character
variables to factors.
Both arguments for the critics are not convinging for me:
ad 1) **Why using row names for indiviual labelling?**
> Avoiding row names may be a reasonable strategy for huge data sets where the cases don’t have individual identities beyond their row numbers, but automatic labeling of individual cases by their row names can be very useful in regression diagnostics and for other methods that identify unusual or influential cases.
Labelling individual cases are important information and should be incorparated into the data file as a column. The strategy to use these individual names could be fulfilled referencing this columns or applying a default value (for instance the first column in the data set).
As things stand I need to use row names to use the advanced features of the {**car**} package. This excludes the use of tibbles: Whenever you add row names to a tibble it will loose it special features because it will be converted automatically to a data frame.
::: {.callout-important #imp-row-names-tools}
##### Working with row names
There are several tools for working with row names (see: [Tools for working with row names](https://tibble.tidyverse.org/reference/rownames.html)) which will become important when applying the {**car**} package and trying still to be following the tidyverse approach as well as in any way possible:
- `tibble::has_rownames(<dataFrame>)`
- `remove_rownames(<dataFrame>)`
- `rownames_to_column(<dataFrame>, var = "<rowNameColumn>")`
- `rowid_to_column(<dataFrame>, var = "<rowNameId>")`
- `column_to_rownames(.data, var = "<rowNameColumn>")`
:::
ad 2) **Why using `stringAsFactor = TRUE` as default?**
Even R has with version 4.0.0 the default changed to `stringsAsFactors = FALSE`, and "hence by default [R] no longer convert strings to factors in calls to `data.frame()` and `read.table()`" ([R 4.0.0 is released](https://stat.ethz.ch/pipermail/r-announce/2020/000653.html)).
Besides it is easy to change this default. It is therefore no big deal in my opinion that justifies to decline the advantages of the {**tidyverse**} packages.
:::::{.my-remark}
:::{.my-remark-header}
:::::: {#rem-chap02-car-tidyverse}
: My strategy to harmonize {**car**} with {**tidyverse**}
::::::
:::
::::{.my-remark-container}
As far as I understand the {**car**} packages provides many important enhancement for the regression analysis that are not included in base R or other packages including {**tidyverse**}. But I trust the future development of the {**tidyverse**} approache more than the professors Fox and Weisberg, who both are already retired (professor emeritus).
So what I am trying with the book:
1. First of all to learn the regression tools in {**car**} for an advanced regression analysis.
2. To look around if these tools could be incorporated in the {**tidyverse**} approach. Here I am thinking at the moment of two ways:
a) Are there packages following the {**tidyverse**} approach that already implement these advanced regression analysis tools? I am thinking here of the huge list of [ggplot2 extensions](https://exts.ggplot2.tidyverse.org/). Currently (2024-05-19) there are 133 registered extensions available to explore!
b) Is it possible to code the desired feature myself? At the moment I am not thinking of developing a new package. Maybe it is possible to add the feature with 2-3 lines of code into a seuqence of R commands? An example that inspired me is the formula by D. Freedman and Diaconis [-@freedman1981a] that could be added in calls to `ggplot2::geom_histogram()`.
::::
:::::
## Working with data frames {#sec-chap02-3}
### How the R interpreter finds objects (empty) {#sec-chap02-3-1}
### Missing data {#sec-chap02-3-2}
:::::{.my-resource}
:::{.my-resource-header}
:::::: {#lem-missing-data}
: Packages for working with missing data
::::::
:::
::::{.my-resource-container}
- Amelia [@Amelia]
- ggmice [@ggmice]
- mi [@mi]
- mice [@mice]
- naniar [@naniar]
- norm [@norm]
***
- See also the [CRAN task view about Missing Data](https://cran.r-project.org/view=MissingData)Flexible
- [R-miss-tastic](https://rmisstastic.netlify.app/): A resource website on missing values - Methods and references for managing missing data
- [Multiple Imputation of Missing Data](https://www.john-fox.ca/Companion/appendices/Appendix-Multiple-Imputation.pdf): An Appendix to An R Companion to Applied Regression [@fox2018b]
- Flexible Imputation of Missing Data Imputation of Missing Data [@buuren2018]. There is also an [online-version](https://stefvanbuuren.name/fimd/) [@buuren2018a].
- [Welcome to amices](https://amices.org/): A place for people interested in solving missing data problems.
- [Amelias II](https://gking.harvard.edu/amelia): Amelia II: A Program for Missing Data.
- Visualization and Imputation of Missing Values: With Applications in R [@templ2023]
***
```{r}
#| label: download-figures-missing-data
pkgs_dl(c("Amelia", "ggmice", "mi", "mice", "naniar", "norm"))
```
***
The top package of my package selection is {**mice**}. It also has the most extensive documentation.
An interesting choice --- not mentioned in the {**car**} companion --- is {**naniar**}, because it implements NA imputation following the {**tidyverse**} approach.
::::
:::::
Missing data is a profound statistical issue concerning how best to use available information ("imputation of missing data") when missing data are encountered. I will ignore these issues here and postpone for later treatment (learning sessions). Here I will just list general commands to exclude missing data.
::: {#bul-ignore-na-commands}
:::::{.my-bullet-list}
:::{.my-bullet-list-header}
Commands to exclude missing data (NA’s)
:::
::::{.my-bullet-list-container}
- `na.rm`: Many R functions that calculate simple statistical summaries, such as `base::mean()`, `stats::var()`, `stats::sd()`, and `stats::quantile()`, have an `na.rm` argument to compute the summaries without NA's. The default value is always `na.rm = FALSE` so you will get an NA result if you forgot to remove the missing values.
- `na.action`: Most statistical-modeling functions in R have an `na.action` argument, which controls how missing data are handled: `na.action` is set to a function that takes a data frame as an argument and returns a similar data frame composed entirely of valid data.
- `na.omit()`: Default value, which removes all cases with missing data on any variable in the computation. --- You can change the default `na.action` with the `options()` command. For example, `options(na.action="na.exclude")`
- `na.exclude()`: Similar to `na.omit()` but saves information about missing cases that are removed. This is the recommended option!
- `na.fail()`: Produces an error message.
- `stats::complete.cases()`: Returns a logical vector indicating which cases are complete, i.e., have no missing values.
- `tidyr::drop_na()`: Drops rows containing missing values. If no columns are mentioned then the function applies to the whole data set.
***
Don't forget that the proper way to test for missing values is `base::is.na()`. The equal operator `==` will not work!
::::
:::::
Commands to exclude missing data (NA’s)
:::
### Modifying and transforming data {#sec-chap02-3-3}
For modifying data I will rouintely use the {**dpylr**} package. Here I will only list other options to remind me if I will see these functions.
:::::{.my-bullet-list}
:::{.my-bullet-list-header}
Modifying and transforming data without using {**dpylr**}
:::
::::{.my-bullet-list-container}
- `base::transform()`: Create or modify several variables in a data frame at once.
- `base::cut()`: creates bins for numeric variables. (For this function I didn't find an equivalent in the {**tidyverse**} ecosysystem.)
- `car::Recode()`: same as `base::cut()` but mor flexible.
::::
:::::
### Binding rows and columns (empty) {#sec-chap02-3-4}
### Aggregating data frames (empty) {#sec-chap02-3-5}
### Merging data frames (empty) {#sec-chap02-3-6}
### Reshaping data (empty) {#sec-chap02-3-7}
## Working with matrices, arrays, and lists (empty) {#sec-chap02-4}
## Dates and times (empty) {#sec-chap02-5}
## Character data (empty) {#sec-chap02-6}
## Large datasets in R (empty) {#sec-chap02-7}