-
Notifications
You must be signed in to change notification settings - Fork 11
/
pkgdev.Rmd
416 lines (361 loc) · 22.7 KB
/
pkgdev.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
# Guide for Authors {#pkgdev}
The current chapter should be considered an extension of the corresponding
["Guide for Authors"](https://devguide.ropensci.org/authors-guide.html) in
rOpenSci's "Dev Guide". The principles for package development described there
also apply to statistical packages, with this chapter describing additional
processes and practices for packages intended for submission to the statistical
software peer review system.
The major additional process is the documentation of how a package complies
with the general and category-specific standards given in [Chapter
6](#standards). Statistical software packages must document compliance with the
[*General Standards*](#general-standards), as well as at least one set of
[category-specific standards](#standards). Authors need to document within the
software itself every point at which it complies with every general and every
applicable category-specific standard. The process of doing so is facilitated
by the [`srr` package](https://github.com/ropensci-review-tools/srr), as
described in detail in [Sub-section 3.4, below](#pkgdev-srr).
Documenting compliance with our standards will generally entail more work than
for non-statistical software submitted to our general peer review system. The
first of the following sub-sections thus presents some of the benefits
authors may expect from submitting their statistical software for peer-review.
The chapter then proceeds with a description of the scope of statistical
software able to be considered for review, followed by descriptions of two
tools intended to be used through the entire process of package development.
The first tool is the [`pkgcheck`
package](https://github.com/ropensci-review-tools/pkgcheck) which should be used
to confirm whether software is ready for submission or not, and enables authors
to locally run the suite of checks which are automatically run on package
submission. The subsequent sub-section describes our [`autotest`
tool](https://github.com/ropensci-review-tools/autotest), which is intended to
be used through the entire process of package development. The third
sub-section describes how use the [`srr`
package](https://github.com/ropensci-review-tools/srr) to address the major
task of aligning software with our [general and category-specific
standards](#standards) for statistical software, and the final sub-section
describes the final step of specifying which grade of badge authors are aiming
for.
## Benefits of statistical software peer review
- First, feedback from your reviewers and editors will increase the quality and
usability of your software. Both authors and reviewers consistently report
that review improves code and makes them better programmers through the
feedback and interaction with each other.
- Bringing your software into compliance with our standards will make your
software easier to maintain, as many of our standards, such as those for
testing and continuous integration, are designed to make sure software
continues to function well as it is updated.
- As our standards and peer-review process aim to improve the quality and
completeness of documentation, going through peer-review should make your
software more understandable and reduce the support you need to provide to
users.
- Approval by rOpenSci will make it easier to submit your software to
repositories and journals. Packages approved by rOpenSci generally are
CRAN-ready. They are eligible for expedited review at the [*Journal of Open
Source Software*](https://joss.theoj.org/), and the [*Journal of Statistical
Software* (JStatSoft)](https://www.jstatsoft.org/index) already [recommends
that developers refer to our standards](https://www.jstatsoft.org/authors).
In our experience code peer-review is looked on favorably by reviewers of
software papers at many journals.
- In regulated environments and other fields where strong demonstration of
quality and compliance is needed, approval by peer-review provides strong
evidence of software quality rarely available for statistical software. Our
approach to annotating _within code_ how software meets standards makes
demonstration of compliance clear and granular. We are actively working with
the [R Validation Hub](https://pharmar.org) to make approval under our system
of review and standards a component of validation in regulated environments
such as pharmaceutical or clinical research.
## Scope
The first important task prior to submitting a package is to estimate whether
a package is likely to be considered within our scope for statistical software.
As described in the [Overview](#overview-scope), packages are generally
considered in scope if they fit one or more of the [categories listed
there](#overview-categories). Prior to submission, authors must choose one
or more of these categories, and document how their software aligns with the
corresponding standards given in [Chapter 6](#standards), according to the
procedures [described below](#pkgdev-srr). Any software which can be aligned
with one or more sets of category-specific standards will by definition be
considered in scope.
Authors are encouraged to contact us at any point prior to, or during,
development, to ask about whether a package might be in scope, or which
categories it might fit within. Categorisation of packages may not always be
straightforward, and we particularly encourage authors who are unsure about
whether a package belongs in a particular category or not to contact us for
discussion. An initial judgement of whether or not a package belongs in
a particular category may be gained by examining the respective standards. Any
package for which a large number of standards from a particular category may be
considered applicable (regardless of whether or not they would actually be
checked) is likely to fit within that category. Once you have determined that
your package is likely to fit into one or more of our in-scope categories,
you'll need to apply our three primary development tools described in the
following two sub-sections.
## The [`pkgcheck` package](https://github.com/ropensci-review-tools/pkgcheck)
The [`pkgcheck` package](https://github.com/ropensci-review-tools/pkgcheck) can
be used to confirm whether software is ready for submission or not. The checks
implemented within this package are also automatically run upon submission, and
packages are expected to successfully pass all checks prior to initial
submission. Packages may only be submitted once the main [`pkgcheck()`
function](hThe://ropensci-review-tools.github.io/pkgcheck/reference/pkgcheck.html)
indicates such, through clearly stating,
> *This package may be submitted*
This function accepts a single argument of the local path to
the package being checked, and returns a detailed list of checks and associated
results. The return object has a `summary` method which prints a formatted
result to the console indicating whether a package is ready for
submission or not. See the main [package
website](https://github.com/ropensci-review-tools/pkgcheck) for more details.
The [`pkgcheck()`
function](hThe://ropensci-review-tools.github.io/pkgcheck/reference/pkgcheck.html)
is also applied to all packages upon initial submission, in response to which
our `ropensci-review-bot` will print the results in the issue. In the unlikely
circumstances that a package is unable to pass particular checks, explanations
should be given upon submission about why those checks fail, and why review may
proceed in spite of such failures.
An example result of the [`pkgcheck()`
function](hThe://ropensci-review-tools.github.io/pkgcheck/reference/pkgcheck.html)
may be seen by applying it to the skeleton [`srr` (**S**oftware **R**eview
**R**oclets) package](https://github.com/ropensci-review-tools/srr):
```{r pkgcheck-srr}
#| echo: false
#| eval: false
path <- srr::srr_stats_pkg_skeleton ()
checks <- pkgcheck::pkgcheck (path)
summary (checks)
```
```{r pkgcheck-srr-output}
#| echo: false
#| eval: false
#| collapse: true
cli::cli_h1 ("demo 0.0.0.9000")
cli::cli_text ("")
cli::cli_alert_success ("Package uses 'roxygen2'")
cli::cli_alert_danger ("Package does not have a 'contributing.md' file")
cli::cli_alert_danger ("Package does not have a 'CITATION' file")
cli::cli_alert_danger ("Package does not have a 'codemeta.json' file")
cli::cli_alert_success ("All functions have examples")
cli::cli_alert_danger ("Package 'DESCRIPTION' does not have a URL field")
cli::cli_alert_danger ("Package 'DESCRIPTION' does not have a BugReports field")
cli::cli_alert_success ("Package name is available")
cli::cli_alert_danger ("Package has no continuous integration checks")
cli::cli_alert_danger ("Package coverage is 0% (should be at least 75%)")
cli::cli_alert_success ("R CMD check found no errors")
cli::cli_alert_success ("R CMD check found no warnings")
cli::cli_alert_danger ("This package still has TODO standards and can not be submitted")
cli::cli_text ("")
cli::cli_alert_info ("Current status:")
cli::cli_alert_danger ("This package is not ready to be submitted")
```
## The [`autotest` package](https://github.com/ropensci-review-tools/autotest) {#pkgdev-autotest}
The [`autotest` package](https://github.com/ropensci-review-tools/autotest) is
an automated assessment tool which all packages are expected to pass in order
to be accepted for submission. The package implements a form of "mutation
testing," by examining the types of all input parameters, implementing
type-specific mutations, and examining the response of each function in
a package to all such mutations. This kind of mutation testing is a very
effective way to uncover any unexpected behaviour which authors themselves
might not necessarily pre-empt. The purpose of using
[`autotest`](https://github.com/ropensci-review-tools/autotest) to prepare
packages is to avoid as much as possible the common situation of reviewers
discovering bugs when they attempt to use software in ways that differ from
typical uses envisioned by authors themselves. Reviews of software prepared
with the help of
[`autotest`](https://github.com/ropensci-review-tools/autotest) should be less
burdened by discussions of what are often minor technical details, and more
able to focus on "higher level" aspects of software quality.
Full documentation of how to use
[`autotest`](https://github.com/ropensci-review-tools/autotest) in package
development is provided on the [package
website](https://ropensci-review-tools.github.io/autotest/), and we
particularly encourage any authors intending to develop packages for submission
to our peer review system to step through the [main `autotest`
vignette](https://ropensci-review-tools.github.io/autotest/articles/autotest.html),
and to apply [`autotest`](https://ropensci-review-tools.github.io/autotest/)
continuously throughout package development, to ensure that
[`autotest_package()`](https://ropensci-review-tools.github.io/autotest/reference/autotest_package.html)
returns clean (`NULL`) results when the package is first submitted.
## The [`srr` package](https://github.com/ropensci-review-tools/srr) {#pkgdev-srr}
Once a package has been sufficiently developed to begin alignment with our
standards, and once all issues revealed by
[`autotest`](https://ropensci-review-tools.github.io/autotest/) have been
addressed, authors will need to use our third tool, the [`ssr` (**s**oftware
**r**eview **r**oclets) package](https://ropensci-review-tools.github.io/srr/)
to insert both general and category-specific standards into their code, and to
begin the process of documenting within the code itself how and where the code
adheres to the individual standards. The [`srr`
package](https://ropensci-review-tools.github.io/srr/) can be installed locally
by running either one of the following two lines.
```{r srr-install-dev, eval = FALSE, echo = TRUE}
#| echo: true
#| eval: false
remotes::install_github("ropensci-review-tools/srr")
pak::pkg_install("ropensci-review-tools/srr")
```
`srr` procedures are described in detail on the [package
website](https://ropensci-review-tools.github.io/srr/), and in particular in
the [main
vignette](https://ropensci-review-tools.github.io/srr/articles/srr-stats.html).
Authors are first encouraged to obtain a local copy of the [source code for
that
vignette](https://github.com/ropensci-review-tools/srr/blob/main/vignettes/srr-stats.Rmd),
and to step through each line in order to understand how the procedure works.
Having done that, you may then insert standards into your own package by
running the following line from within the local directory of your package,
```{r srr-standards}
#| echo: true
#| eval: false
srr_stats_roxygen (category = c ("<category-1>", "<category-2>"))
```
That will insert a new file into the `R/` directory of your package called (by
default) `srr-stats-standards.R`. All standards initially have
a [`roxygen2`](https://roxygen2.r-lib.org) tag of `@srrstatsTODO`, to indicate
that these standards are yet to be addressed. These tags are processed by the
[`srr` roclet](https://ropensci-review-tools.github.io/srr/) which needs to be
connected with your package by modifying the `Roxygen` line of your
`DESCRIPTION` file to the following form:
```{r srr-DESC}
#| echo: true
#| eval: false
Roxygen: list (markdown = TRUE, roclets = c ("namespace", "rd", "srr::srr_stats_roclet"))
```
You do not need to add the `srr` package anywhere else in your `DESCRIPTION`
file, nor do you need to retain this line when submitting packages to CRAN (or
elsewhere). You should nevertheless retain the line at all other times, and you
can easily disable the roclet output by including `#' @srrVerbose FALSE`
somewhere within your documentation. Note that `srr` documentation lines are
used only to produce on-screen output triggered by running
[`roxygen2::roxygensise()`](https://roxygen2.r-lib.org/reference/roxygenize.html),
or the equivalent function,
[`devtools::document()`](http://devtools.r-lib.org/reference/document.html),
and do not appear in actual package documentation.
The [`srr` roclet](https://ropensci-review-tools.github.io/srr/) recognises and
process three tags:
1. `@srrstatsTODO` to flag standards yet to be addressed;
2. `@srrstats` to flag standards which have been addressed, and followed by
descriptions of how your code addresses those standards; and
3. `@srrstatsNA` to flag standards which you deem not to be applicable to your
code, followed by explanations of why you deem those standards not
applicable.
The file generated by
[`srr_stats_roxygen()`](https://ropensci-review-tools.github.io/srr/reference/srr_stats_roxygen.html)
initially contains two [`roxygen2`](https://roxygen2.r-lib.org) blocks, the
first containing every standard potentially applicable to your package, tagged
with `@srrstatsTODO`, and the second with a title of `NA_standards`, to
document standards deemed not applicable. The first task after having generated
this file is to move standards to approximate locations within your package
where they are likely to be addressed. For example, standards concerning tests
should be moved somewhere within the `tests/` directory, standards concerning
documentation to the main `README.Rmd` file, or within a vignette file. The
[package
skeleton](https://ropensci-review-tools.github.io/srr/reference/srr_stats_pkg_skeleton.html)
includes code demonstrating how to include roclet tags within `.Rmd` files.
Moving different standards to more appropriate locations within your code will
break down an initially large single list of standards into more manageable
groups dispersed throughout your code. As each standard is addressed, it should
be moved to one or more locations in your code as near as possible to relevant
code, the tag changed from `@srrstatsTODO` to `@srrstats`, and a brief
description appended to explain how that standard is addressed. Standards
deemed not to be applicable to your package should all be grouped together
within a single [`roxygen2`](https://roxygen2.r-lib.org) block with a title of
`NA_standards`, each with a tag of `@srrstatsNA`, and a brief description of
why those standards are deemed not to be applicable.
Software to be submitted for review must contain no `@srrstatsTODO` tags --
that is, all standards must have been addressed by modifying every tag to
either `@srrstats` or `@srrstatsNA`, as described above. Two useful functions
to aid package alignment with standards are:
1. The [`srr_stats_pre_submit()`
function](https://ropensci-review-tools.github.io/srr/reference/srr_stats_pre_submit.html),
which confirms that all standards have been addressed prior to submission.
2. The [`srr_report()`
function](https://ropensci-review-tools.github.io/srr/reference/srr_report.html),
which generates a summary report with hyperlinks to locations within your
code at which all standards are placed.
The output of both of these functions are included in the result of the
[`pkgcheck()`
function](hThe://ropensci-review-tools.github.io/pkgcheck/reference/pkgcheck.html),
both when run locally, and as run upon initial package submission. The
[`srr_stats_pre_submit()`
function](https://ropensci-review-tools.github.io/srr/reference/srr_stats_pre_submit.html)
can be used locally to confirm that,
```{r pre-submit-success}
#| echo: false
cli::cli_alert_success ("All applicable standards have been documented in this package")
```
while the result of the [`srr_report()`
function](https://ropensci-review-tools.github.io/srr/reference/srr_report.html)
may be accessed through the link given in the `pkgcheck` output, or it can be
viewed by calling that function directly.
## Gold, Silver, and Bronze Badges {#pkgdev-badges}
All statistical software which is recommended for acceptance by reviewers is
entitled to display an rOpenSci badge. This badge is a modified version of the
badge for the [current peer-review
system](https://devguide.ropensci.org/building.html#readme), with an additional
section on the far right indicating the version of that standards against which
the software was assessed, coloured according to the "grade" of the badge. The
three possible badges look like this:
[![](badges/stats-bronze.png)](https://github.com/ropensci/statistical-software-review-book)
**bronze** for software which is sufficiently or minimally compliant with
standards to pass review.
[![](badges/stats-silver.png)](https://github.com/ropensci/statistical-software-review-book)
**silver** for software for which complies with more than a minimal set of
applicable standards, and which extends beyond bronze in least one notable way,
as explained below.
[![](badges/stats-gold.png)](https://github.com/ropensci/statistical-software-review-book)
**gold** for software which complies with *all* standards which reviewers have
deemed potentially applicable.
The [submission
template](https://github.com/ropensci/software-review/blob/master/.github/ISSUE_TEMPLATE/F-submit-statistical-software-for-review.md)
for statistical software submissions requires authors to identify the grade
they wish to attain from the review process. These standards are not static,
and it is always possible to elevate a badge to a higher grade subsequent to
review. Badge grades may also be downgraded for code which is not continuously
aligned with ongoing developments in standards. The following sub-sections
provide further clarification of each grade.
### Bronze [![](badges/stats-bronze.png)](https://github.com/ropensci/statistical-software-review-book) {#pkgdev-bronze}
Software which is sufficiently or minimally compliant with standards will
receive a bronze badge. One common reason for this badge is software which
authors do not intend to develop further following review. This commonly
arises for software produced from research projects which have been completed,
leaving no funding to further develop the software. Another reason might be
that software has been developed for a particular use case, with authors
unable to align it with additional standards in order to expand its general
utility. A bronze badge need not signify any weakness or inadequacy in
software, rather it will generally signify software which has been developed
for one particular use case, and which will not be subject to significant
further development.
### Silver [![](badges/stats-silver.png)](https://github.com/ropensci/statistical-software-review-book) {#pkgdev-silver}
Silver badges are granted to software which extends beyond the minimal
requirements of bronze in at least one the following four aspects:
- Compliance with a good number of standards beyond those identified as
minimally necessary. This will require reviewers and authors to agree on
identification of both a minimal subset of necessary standards, and a full
set of potentially applicable standards. This aspect may be considered
fulfilled if at least one quarter of the additional potentially applicable
standards have been met, and should definitely be considered fulfilled if
more than one half have been met.
- Demonstrating excellence in compliance with multiple standards from at least
two broad sub-categories. Sub-categories are distinguished in the [Standards
Chapter](#standards) by three numbers, so that the [*General
Standards*](#general-standards) have five sub-categories numbered 6.1.1 to
6.1.5. This aspect would require software to extend notably beyond the
requirements of two or more standards in at least two sub-categories
(regardless of whether general or category-specific standards). For example,
software which might otherwise be assessed at bronze grade, yet which is both
excellently documented, and has an outstanding test suite, may be considered
to fulfil this aspect.
- Have a demonstrated *generality* of usage beyond one single envisioned use
case. Software is frequently developed for one particular use case envisioned
by the authors themselves. Generalising the utility of software so that it
is readily applicable to other use cases, and satisfactorily documenting such
generality of usage, represents another aspect which may be considered
sufficient for software to attain a silver grade.
- Internal aspects of package structure and design. Many aspects of the
internal structure and design of software are too variable to be effectively
addressed by standards. Packages which are judged by reviewers to reflect
notably excellent design choices, especially in the implementation of core
statistical algorithms, may also be considered worthy of a silver grade.
### Gold [![](badges/stats-gold.png)](https://github.com/ropensci/statistical-software-review-book) {#pkgdev-gold}
To attain a gold badge, software must comply with all applicable standards, and
must also fulfil at least three of the four aspects described above for
silver-grade badges. Both the applicability of standards, and fulfilment of
these three aspects, will ultimately determined by reviewers. Moreover,
compliance with all grades is assessed against current standards, meaning that
a gold badge must be actively maintained as standards themselves are revised
and updated.