Pagination

[sboysel]

Had the need recently to use something like this

library(fredr)
library(tidyverse)
fredr_paginate <- function(fredr, ..., sleep = 0L, verbose = FALSE) {
  
  stopifnot(inherits(fredr, "function"))
  stopifnot(length(sleep) == 1, is.numeric(sleep))
  
  args <- list(...)
  stopifnot(!"offset" %in% names(args))
  
  # iteration setup
  done <- FALSE
  offset <- 0L
  if (!"limit" %in% names(args)) {
    limit <- 1000L
  }
  results_list <- list()
  
  # iterate
  while (!done) {
    
    # set offset parameter
    args[["offset"]] <- offset * limit
    
    if (verbose) {
      message(paste("Offset:", args[["offset"]]))
    }
    
    # get page
    results <- do.call(what = fredr, args = args)
    
    # add page to page list
    results_list[[offset + 1]] <- results
    
    # done if results returned are less than limit parameter
    if (nrow(results) < limit) {
      done <- TRUE
    }
    
    # increment offset
    offset <- offset + 1
    
    # pause before iterating again
    Sys.sleep(sleep)
    
  }
  
  # return results
  results_list
  
}

series_list <- fredr_paginate(fredr_series_search_text, search_text = "Mean Commute Time", verbose = TRUE)
#> Offset: 0
#> Offset: 1000
#> Offset: 2000
#> Offset: 3000
series <- dplyr::bind_rows(series_list)
series
#> # A tibble: 3,143 x 16
#>    id    realtime_start realtime_end title observation_sta… observation_end
#>    <chr> <chr>          <chr>        <chr> <chr>            <chr>          
#>  1 B080… 2018-10-03     2018-10-03   Mean… 2009-01-01       2016-01-01     
#>  2 B080… 2018-10-03     2018-10-03   Mean… 2009-01-01       2016-01-01     
#>  3 B080… 2018-10-03     2018-10-03   Mean… 2009-01-01       2016-01-01     
#>  4 B080… 2018-10-03     2018-10-03   Mean… 2009-01-01       2016-01-01     
#>  5 B080… 2018-10-03     2018-10-03   Mean… 2009-01-01       2016-01-01     
#>  6 B080… 2018-10-03     2018-10-03   Mean… 2009-01-01       2016-01-01     
#>  7 B080… 2018-10-03     2018-10-03   Mean… 2009-01-01       2016-01-01     
#>  8 B080… 2018-10-03     2018-10-03   Mean… 2009-01-01       2016-01-01     
#>  9 B080… 2018-10-03     2018-10-03   Mean… 2009-01-01       2016-01-01     
#> 10 B080… 2018-10-03     2018-10-03   Mean… 2009-01-01       2016-01-01     
#> # ... with 3,133 more rows, and 10 more variables: frequency <chr>,
#> #   frequency_short <chr>, units <chr>, units_short <chr>,
#> #   seasonal_adjustment <chr>, seasonal_adjustment_short <chr>,
#> #   last_updated <chr>, popularity <int>, group_popularity <int>,
#> #   notes <chr>

^{Created on 2018-10-03 by the reprex package (v0.2.1)}

I'm quite ignorant of best practices regarding these types of functions so any input would be appreciated.

[sboysel]

As suggested by @DavisVaughan in #103, would be worthwhile to investigate using httr2::req_perform_iterative()