problem with s3 delete_objects

[dleopold]

I am unable to use delete_objects to batch delete objects from an s3 bucket. Copying the documentation exactly

svc$delete_objects(
  Bucket = "...",
  Delete = list(
    Objects = list(
      list(
        Key = "..."
      ),
      list(
        Key = "..."
      )
    ),
    Quiet = FALSE
  )
)

returns an error:

Error: MalformedXML (HTTP 400). The XML you provided was not well-formed or did not validate against our published schema

I have the correct permissions and can delete objects individually using delete_object.

Also, FYI, I noticed that attempting to use either delete_object or delete_objects when the bucket is in a different region, the operation fails with an unhelpful error message:

Error in enc2utf8(data) : argument is not a character vector

It took me a while to figure that one out.

Here is my current session info (though I have also tried with the dev version of paws):

> sessionInfo()
R version 4.2.2 Patched (2022-11-10 r83330)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Pop!_OS 22.04 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.10.0
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.10.0

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8    LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] furrr_0.3.1     future_1.29.0   forcats_0.5.2   stringr_1.5.0   dplyr_1.0.10    purrr_1.0.1     readr_2.1.3    
 [8] tidyr_1.2.1     tibble_3.1.8    ggplot2_3.4.0   tidyverse_1.3.2 paws_0.1.12    

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.9          lubridate_1.9.0     listenv_0.8.0       assertthat_0.2.1    digest_0.6.31       utf8_1.2.2         
 [7] parallelly_1.33.0   R6_2.5.1            paws.common_0.5.3   cellranger_1.1.0    backports_1.4.1     reprex_2.0.2       
[13] httr_1.4.4          pillar_1.8.1        rlang_1.0.6         googlesheets4_1.0.1 curl_5.0.0          readxl_1.4.1       
[19] rstudioapi_0.14     googledrive_2.0.0   munsell_0.5.0       broom_1.0.1         compiler_4.2.2      modelr_0.1.10      
[25] base64enc_0.1-3     pkgconfig_2.0.3     globals_0.16.2      tidyselect_1.2.0    codetools_0.2-18    fansi_1.0.3        
[31] crayon_1.5.2        tzdb_0.3.0          dbplyr_2.2.1        withr_2.5.0         grid_4.2.2          jsonlite_1.8.4     
[37] gtable_0.3.1        lifecycle_1.0.3     DBI_1.1.3           magrittr_2.0.3      scales_1.2.1        cli_3.6.0          
[43] stringi_1.7.8       fs_1.5.2            xml2_1.3.3          ellipsis_0.3.2      generics_0.1.3      vctrs_0.5.1        
[49] tools_4.2.2         glue_1.6.2          paws.storage_0.1.12 hms_1.1.2           rsconnect_0.8.28    parallel_4.2.2     
[55] timechange_0.1.1    colorspace_2.0-3    gargle_1.2.1        rvest_1.0.3         haven_2.5.1

[DyfanJones]

Hi @dleopold sorry about that. Have you tried to delete objects without Quiet parameter?

svc$delete_objects(
  Bucket = "...",
  Delete = list(
    Objects = list(
      list(
        Key = "..."
      ),
      list(
        Key = "..."
      )
    )
  )
)

Here is an example of it working in the up and coming s3fs package https://github.com/DyfanJones/s3fs/blob/main/R/s3filesystem_class.R#L349-L362 .

Interesting I didn't realise this:

Also, FYI, I noticed that attempting to use either delete_object or delete_objects when the bucket is in a different region, the operation fails with an unhelpful error message:

Error in enc2utf8(data) : argument is not a character vector

I will have to have a look why this is the case 🤔

[dleopold]

It does seem to work with the Quiet parameter removed. Not sure how I failed to try that. Thank you. I will close the issue since it is resolved for me, though maybe a quick update to the docs would prevent others from running into the same issue. Also, s3fs looks promising!

[wlandau]

I am hitting this issue too (working on ropensci/targets#1171) and it would be great to get Quiet = TRUE to work. Quiet mode sends a smaller HTTP response, so it could perform faster.

[DyfanJones]

Happy to re-open this ticket.

[DyfanJones]

Ok I think I have fixed it:

remotes::install_github("dyfanjones/paws/paws.common", ref = "xml_build_flatten")

client <- paws.storage::s3()

bucket <- "mybucket"
key <- "removable.txt"

resp <- client$put_object(
  Bucket = bucket,
  Key = key,
  Body = charToRaw("dummy")
)

client$delete_objects(
  Bucket = bucket,
  Delete = list(
    Objects = list(
      list(Key = key)
    ),
    Quiet = T
  )
)
#> $Deleted
#> list()
#> 
#> $RequestCharged
#> character(0)
#> 
#> $Errors
#> list()

^{Created on 2023-11-08 with reprex v2.0.2}

Session info

sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.3.2 (2023-10-31)
#>  os       macOS Sonoma 14.0
#>  system   aarch64, darwin20
#>  ui       X11
#>  language (EN)
#>  collate  en_US.UTF-8
#>  ctype    en_US.UTF-8
#>  tz       Europe/London
#>  date     2023-11-08
#>  pandoc   3.1.9 @ /opt/homebrew/bin/ (via rmarkdown)
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package      * version    date (UTC) lib source
#>  base64enc      0.1-3      2015-07-28 [1] CRAN (R 4.3.0)
#>  cli            3.6.1      2023-03-23 [1] CRAN (R 4.3.0)
#>  crayon         1.5.2      2022-09-29 [1] CRAN (R 4.3.0)
#>  curl           5.1.0      2023-10-02 [1] CRAN (R 4.3.1)
#>  digest         0.6.33     2023-07-07 [1] CRAN (R 4.3.0)
#>  evaluate       0.23       2023-11-01 [1] CRAN (R 4.3.1)
#>  fastmap        1.1.1      2023-02-24 [1] CRAN (R 4.3.0)
#>  fs             1.6.3      2023-07-20 [1] CRAN (R 4.3.0)
#>  glue           1.6.2      2022-02-24 [1] CRAN (R 4.3.0)
#>  htmltools      0.5.7      2023-11-03 [1] CRAN (R 4.3.1)
#>  httr           1.4.7      2023-08-15 [1] CRAN (R 4.3.0)
#>  knitr          1.45       2023-10-30 [1] CRAN (R 4.3.1)
#>  lifecycle      1.0.3      2022-10-07 [1] CRAN (R 4.3.0)
#>  magrittr       2.0.3      2022-03-30 [1] CRAN (R 4.3.0)
#>  mime           0.12       2021-09-28 [1] CRAN (R 4.3.0)
#>  paws.common    0.6.3.9000 2023-11-08 [1] Github (dyfanjones/paws@a7aaf4d)
#>  paws.storage   0.5.0      2023-11-02 [1] local (/Users/dyfanjones/Documents/Packages/paws/cran/paws.storage)
#>  purrr          1.0.2      2023-08-10 [1] CRAN (R 4.3.0)
#>  R.cache        0.16.0     2022-07-21 [1] CRAN (R 4.3.0)
#>  R.methodsS3    1.8.2      2022-06-13 [1] CRAN (R 4.3.0)
#>  R.oo           1.25.0     2022-06-12 [1] CRAN (R 4.3.0)
#>  R.utils        2.12.2     2022-11-11 [1] CRAN (R 4.3.0)
#>  R6             2.5.1      2021-08-19 [1] CRAN (R 4.3.0)
#>  Rcpp           1.0.11     2023-07-06 [1] CRAN (R 4.3.0)
#>  reprex         2.0.2      2022-08-17 [1] CRAN (R 4.3.0)
#>  rlang          1.1.1      2023-04-28 [1] CRAN (R 4.3.0)
#>  rmarkdown      2.25       2023-09-18 [1] CRAN (R 4.3.1)
#>  rstudioapi     0.15.0     2023-07-07 [1] CRAN (R 4.3.0)
#>  sessioninfo    1.2.2      2021-12-06 [1] CRAN (R 4.3.0)
#>  styler         1.10.2     2023-08-29 [1] CRAN (R 4.3.0)
#>  vctrs          0.6.4      2023-10-12 [1] CRAN (R 4.3.1)
#>  withr          2.5.2      2023-10-30 [1] CRAN (R 4.3.1)
#>  xfun           0.41       2023-11-01 [1] CRAN (R 4.3.1)
#>  xml2           1.3.5      2023-07-06 [1] CRAN (R 4.3.0)
#>  yaml           2.3.7      2023-01-23 [1] CRAN (R 4.3.0)
#> 
#>  [1] /Users/dyfanjones/Library/R/arm64/4.3/library
#>  [2] /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library
#> 
#> ──────────────────────────────────────────────────────────────────────────────

@wlandau @dleopold Please try it out and let me know :)

[wlandau]

Thanks so much, @DyfanJones! Works for me now!

[dleopold]

Works here. Thanks.

[DyfanJones]

paws.common 0.6.4 has been released to cran.