gcs_list_buckets only returns up to 1000 buckets.

[hidekoji]

Currently, it seems gcs_list_buckets does not support pagination unlike gcs_list_objects but it's nice if gcs_list_buckets supports pagination so that it can list more than 1000 buckets.

[MarkEdmondson1234]

I never had enough buckets to test it! But yes it's a relatively quick update using the gar_page() utility from googleAuthR, thanks for flagging it.

[MarkEdmondson1234]

I dropped in the function from list objects, if you have more than 1000 buckets could you give it a try via the github version now?

[MarkEdmondson1234]

Is this sorted?

[hidekoji]

Apologies for the belated update.
I installed the GitHub version but it still gives me up to 1000.

if(!require("remotes")){
    install.packages("remotes")
}
remotes::install_github("cloudyr/googleCloudStorageR")

> df <- googleCloudStorageR::gcs_list_buckets(projectId = 'exploratory-io',  projection = c("noAcl"), maxResults = 1000)
> nrow(df)
[1] 1000

[hidekoji]

I was testing with 1003 buckets like the below screenshots.

[MarkEdmondson1234]

In your example you set max results to be 1000? What happens if unset or set to > 1003 ?

[hidekoji]

For both cases, it fetches up to 1000.

> df <- googleCloudStorageR::gcs_list_buckets(projectId = 'exploratory-io',  projection = c("noAcl"))
> nrow(df)
[1] 1000
> df <- googleCloudStorageR::gcs_list_buckets(projectId = 'exploratory-io',  projection = c("noAcl"), maxResults = 1010)
> nrow(df)
[1] 1000

[MarkEdmondson1234]

Hmm ok, would you mind showing sessionInfo() and also have options(googleAuthR.verbose = 2) on to give some more logging info.

[hidekoji]

> df <- googleCloudStorageR::gcs_list_buckets(projectId = 'exploratory-io', maxResults = 1010, projection = c("noAcl"))
ℹ 2022-10-27 09:40:32 > Token exists.
ℹ 2022-10-27 09:40:32 > Request:  https://storage.googleapis.com/storage/v1/b/?project=exploratory-io&prefix=&projection=noAcl
ℹ 2022-10-27 09:40:33 > No paging required

> sessionInfo()
R version 4.2.1 (2022-06-23)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Big Sur 11.6

Matrix products: default
LAPACK: /Library/Frameworks/R.framework/Versions/4.2/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.8.3                   lattice_0.20-45                tidyr_1.2.0                    googleCloudStorageR_0.7.0.9000
 [5] zoo_1.8-10                     assertthat_0.2.1               digest_0.6.29                  psych_2.2.5                   
 [9] foreach_1.5.2                  utf8_1.2.2                     R6_2.5.1                       plyr_1.8.7                    
[13] backports_1.4.1                httr_1.4.3                     ggplot2_3.3.6                  pillar_1.7.0                  
[17] rlang_1.0.3                    curl_4.3.2                     rstudioapi_0.13                data.table_1.14.2             
[21] googleAuthR_2.0.0.1            Matrix_1.4-1                   urltools_1.7.3                 stringr_1.4.0                 
[25] bit_4.0.4                      triebeard_0.3.0                munsell_0.5.0                  proxy_0.4-27                  
[29] broom_1.0.0                    compiler_4.2.1                 pkgconfig_2.0.3                askpass_1.1                   
[33] mnormt_2.1.0                   openssl_2.0.2                  tidyselect_1.1.2               tibble_3.1.7                  
[37] anonymizer_0.2.2               dtw_1.22-3                     codetools_0.2-18               fansi_1.0.3                   
[41] crayon_1.5.1                   dplyr_1.0.9                    dbplyr_2.2.1                   grid_4.2.1                    
[45] nlme_3.1-157                   jsonlite_1.8.0                 bigrquery_1.4.0                gtable_0.3.0                  
[49] exploratory_6.11.3.1           lifecycle_1.0.1                DBI_1.1.2                      magrittr_2.0.3                
[53] scales_1.2.0                   zip_2.2.0                      cli_3.3.0                      stringi_1.7.6                 
[57] cachem_1.0.6                   reshape2_1.4.4                 fs_1.5.2                       ellipsis_0.3.2                
[61] generics_0.1.2                 vctrs_0.4.1                    prediction_0.3.14              iterators_1.0.14              
[65] tools_4.2.1                    bit64_4.0.5                    glue_1.6.2                     purrr_0.3.4                   
[69] parallel_4.2.1                 fastmap_1.1.0                  yaml_2.3.5                     colorspace_2.0-3              
[73] gargle_1.2.0                   memoise_2.0.1    

[MarkEdmondson1234]

No paging required hmmm