Implemented marker-based pagination in `box_ls` #79

awong234 · 2018-06-20T23:53:29Z

Problem: [Issue #74]

Offset-based pagination has an offset limit of 300,000 which is not stated in the Box API documentation, but will be added following conversation with the Box team. The original function for box_pagination has been renamed as box_paginate_offset to preserve original functionality.

Solution

The team recommended the use of marker-based pagination. Marker-based pagination is, according to the Box API docs, "the preferred method and is most performant", and so this change makes marker-based pagination the default method for GET-ting folder items.

Tests

My tests succeed in comparing and syncing my local set of files to the remote set of my box folder with quantity of files >330,000 where offset-based pagination returns an HTTP 400 error. Microbenchmark of the tests show comparable time to query 100,000 file names

Unit: seconds
   expr      min       lq     mean   median       uq      max neval
 OFFSET 37.52012 40.04692 41.27804 40.93159 41.68023 50.97874   100
 MARKER 40.87141 42.31870 44.24652 42.98368 43.90257 59.99026   100

Merging recent changes from original repo

…ing offset-based paging methods. User can now set which one, but marker is the default.

# Conflicts: # R/boxr_misc.R # man/box_ls.Rd

ijlyttle · 2018-06-21T14:46:23Z

Hi Alec,

I appreciate this PR. Unfortunately it may be a few days before I will have time to look at it properly.

Ian

awong234 · 2018-06-21T14:52:46Z

Certainly, take your time! I will let you know if I encounter any issues within my fork.

ijlyttle

Hi Alec,

Thanks for putting this together - I have a few ideas to make things a bit cleaner.

Unless you have a reason to preserve offset-based pagination, I would redo this by converting box_pagination() to use markers, and dropping support for offsets.

Thanks!

ijlyttle · 2018-06-24T19:01:26Z

DESCRIPTION

 Type: Package
 Title: Interface for the 'Box.com API'
-Version: 0.3.4.99999
+Version: 0.3.4.999999


Let's leave the version number the same, for the time-being.

ijlyttle · 2018-06-24T19:03:17Z

R/boxr_misc.R

+
+  out = list()
+
+  url$query$usemarker = T


please use TRUE instead of T

ijlyttle · 2018-06-24T19:03:42Z

R/boxr_misc.R

+
+  n_so_far = 0
+
+  out = list()


Does this need to be here twice?

ijlyttle · 2018-06-24T19:05:11Z

R/boxr_misc.R

-box_pagination <- function(url, max = 200) {
+box_paginate_marker = function(url, max){
+
+  out = list()


Please follow the convention of the package to use <- for assignment

ijlyttle · 2018-06-24T19:08:53Z

R/boxr_misc.R

+box_ls <- function(dir_id = box_getwd(), limit = 100, max = Inf, fields = NULL, pageMode = 'marker') {

  # maybe some logic here to check that limit <= 1000



While you're here, could you add the logic make sure that limit is no more than 1000? If it is, I think it would be appropriate to set it to 1000 and issue a warning. Thanks!

ijlyttle · 2018-06-24T19:10:47Z

R/boxr_misc.R

+
+    if (req$status_code == 404) {
+      message("box.com indicates that no results were found")
+      return()


I might return NULL here.

This is picky, but it may be better to use if (identical(resp$status_code, 404L))

ijlyttle · 2018-06-24T20:03:47Z

R/boxr_misc.R

+
+  url$query$usemarker = T
+
+  while(!is.null(marker)){


I would use a logical variable here, next_page, like box_paginate_offset().

awong234 · 2018-06-24T20:09:22Z

Hi Ian,

I agree with dropping support for offset-based pagination; there was no benefit that I could see other than preserving legacy function. I will get these changed soon. Thanks!

…and arguments. Pagination function is now once again `box_pagination()`. Warning emitted when limit > 1000 Other minor changes as requested.

…= 1000.

awong234 · 2018-06-29T04:02:40Z

Hi Ian,

Wasn't sure if you were notified of this - I made the changes requested. Thanks for reviewing the code!

ijlyttle · 2018-07-02T14:23:44Z

Hi Alec,

I hope to get to this in the next couple of days, thanks!

awong234 · 2018-07-02T14:34:19Z

Ok great thanks! I've had a few of my users use my fork without errors so far, I will let you know if we encounter anything.

…

On Mon, Jul 2, 2018, 15:23 Ian Lyttle ***@***.***> wrote: Hi Alec, I hope to get to this in the next couple of days, thanks! — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#79 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/Aj2lF3jtp9HcdvmjzA1RJEkZMxYyBYqxks5uCizwgaJpZM4Uwrwf> .

ijlyttle

Hi Alec,

This looks much cleaner - I am not surprised that it is working well for you and your colleagues. This will be a great improvement to the package as a whole.

My only requests are:

formatting, to follow the existing style.
to add an item to NEWS.md - you will see an item there describing your previous improvement (on fields). Please refer to the original issue (#74, I believe)
if you have not already, please add fixes #74 to a commit message.

Thanks!

Ian

ijlyttle · 2018-07-06T20:49:33Z

R/boxr_misc.R

-#'   
-#' @author Brendan Rocks \email{foss@@brendanrocks.com} and Ian Lyttle 
+#'
+#' @author Brendan Rocks \email{foss@@brendanrocks.com} and Ian Lyttle


Could you add yourself to @authors of this function?

ijlyttle · 2018-07-06T21:00:06Z

R/boxr_misc.R

 box_ls <- function(dir_id = box_getwd(), limit = 100, max = Inf, fields = NULL) {

-  # maybe some logic here to check that limit <= 1000
+  if(limit > 1000){


Logic looks good here - could you format to match the spacing and indentation of the code at L41, and use <- for assignment?

Could you do this for the other logic blocks (if, while) and use <- for assignment, generally?

awong234 · 2018-07-12T13:39:52Z

Thanks Ian! I will get to these soon; I'm just returning from travel so I can address the last bits now.

…ed myself to authors.

…package conventions

One of my users attempted to connect to a shared folder that they were not a member of, and the additional information would have been helpful to isolate the problem from my end.

NEWS.md : Added to improvements list for PR r-box#79. Added to bug fixes Issue r-box#74 solution.

awong234 · 2018-07-13T23:16:26Z

Hi Ian,

I made those changes that you requested.

I also took the liberty to make the 404 error a little more explicit (see 12834e4; I had a user try to connect to a shared folder that they were not a member of, and I think the extra information could have made me see the problem sooner.

Thanks for the review!

Alec

ijlyttle

This looks super, thanks!

awong234 added 6 commits June 18, 2018 17:06

Merge pull request #1 from brendan-r/master

f2ca2ca

Merging recent changes from original repo

boxr_misc : Added new functions for marker-based paging, while retain…

119171e

…ing offset-based paging methods. User can now set which one, but marker is the default.

Merge branch 'HEAD' of https://github.com/awong234/boxr.git

ce9398a

# Conflicts: # R/boxr_misc.R # man/box_ls.Rd

added another 9 to the version #

d41a721

boxr_misc : removed bug ; n_so_far was being incremented twice.

e082673

documented changes to boxr_misc

49552f9

ijlyttle requested changes Jun 24, 2018

View reviewed changes

awong234 added 4 commits June 25, 2018 23:45

Edited version number to remain unchanged.

c097ea9

Removed offset-based pagination, and all of the associated variables …

dfadbe9

…and arguments. Pagination function is now once again `box_pagination()`. Warning emitted when limit > 1000 Other minor changes as requested.

documented

7cf9e40

boxr_misc : Adjusted boxr_ls warning to change limit > 1000 to limit …

3f81867

…= 1000.

ijlyttle reviewed Jul 6, 2018

View reviewed changes

awong234 added 5 commits July 13, 2018 18:54

boxr_misc.R : Adjusted assignment operator from = to <-. Also add…

0742bee

…ed myself to authors.

boxr_misc.R : Adjusted formatting of control statements to adhere to …

2c5bd7f

…package conventions

boxr_misc.R : Upon 404 error, I made the message a little more explicit.

12834e4

One of my users attempted to connect to a shared folder that they were not a member of, and the additional information would have been helpful to isolate the problem from my end.

Added Alec Wong to authors, documented.

ffa2bd6

FIXES ISSUE r-box#74

87d9f9c

NEWS.md : Added to improvements list for PR r-box#79. Added to bug fixes Issue r-box#74 solution.

ijlyttle approved these changes Jul 13, 2018

View reviewed changes

ijlyttle merged commit b7c257a into r-box:master Jul 13, 2018

		box_ls <- function(dir_id = box_getwd(), limit = 100, max = Inf, fields = NULL, pageMode = 'marker') {

		# maybe some logic here to check that limit <= 1000

Uh oh!

Implemented marker-based pagination in box_ls #79

Implemented marker-based pagination in box_ls #79

Uh oh!

Conversation

awong234 commented Jun 20, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem: [Issue #74]

Solution

Tests

Uh oh!

ijlyttle commented Jun 21, 2018

Uh oh!

awong234 commented Jun 21, 2018

Uh oh!

ijlyttle left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

awong234 commented Jun 24, 2018

Uh oh!

awong234 commented Jun 29, 2018

Uh oh!

ijlyttle commented Jul 2, 2018

Uh oh!

awong234 commented Jul 2, 2018 via email

Uh oh!

ijlyttle left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

awong234 commented Jul 12, 2018

Uh oh!

awong234 commented Jul 13, 2018

Uh oh!

ijlyttle left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Implemented marker-based pagination in `box_ls` #79

Implemented marker-based pagination in `box_ls` #79

awong234 commented Jun 20, 2018 •

edited

Loading

ijlyttle left a comment •

edited

Loading