Issue with predictions when using step_scale() in recipe

[dudel666]

Problem Description:
It appears that including step_scale() in the recipe disrupts prediction generation within modeltime. I encountered this issue when evaluating models using modeltime_accuracy() - where model fit was significantly worse with step_scale() - and also in modeltime_calibrate(), where forecasts displayed a clear systematic error. As a result, the discrepancy between forecasts from recipes with and without step_scale() is visually evident in the plotted forecasts.
Sample code:

library(ggplot2)
library(recipes)
#> Loading required package: dplyr
#> 
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union
#> 
#> Attaching package: 'recipes'
#> The following object is masked from 'package:stats':
#> 
#>     step
library(parsnip)
library(rsample)
library(workflows)
library(modeltime)
library(timetk)

data(economics)

eco_splits <- time_series_split(economics, date, asses = '10 years', cumulative = T)

spec_lm <- linear_reg(mode = "regression", engine = "lm")

recep <- recipe(pce ~ ., data = economics) %>% 
  update_role(date, new_role = 'time_index') %>% 
  step_arrange(date) %>% 
  step_mutate(date_num = as.numeric(date))

recep_with_scale <- recep %>% 
  step_scale(all_numeric_predictors())

wflw_fit <- workflow() %>% 
  add_recipe(recep) %>% 
  add_model(spec_lm) %>% 
  fit(training(eco_splits))

wflw_fit_with_scale <- workflow() %>% 
  add_recipe(recep_with_scale) %>% 
  add_model(spec_lm) %>% 
  fit(training(eco_splits))

calibration_tbl <- modeltime_table(wflw_fit, wflw_fit_with_scale) %>% 
  mutate(.model_id = c('without_scale', 'with_scale')) %>% 
  modeltime_calibrate(new_data = testing(eco_splits))

calibration_tbl %>% 
  modeltime_accuracy()
#> # A tibble: 2 × 9
#>   .model_id     .model_desc .type    mae  mape  mase smape   rmse   rsq
#>   <chr>         <chr>       <chr>  <dbl> <dbl> <dbl> <dbl>  <dbl> <dbl>
#> 1 without_scale LM          Test   1510.  14.3  37.4  15.4  1588. 0.981
#> 2 with_scale    LM          Test  11778. 114.  292.  200   11794. 0.975

forcast_tbl <- calibration_tbl %>% 
  modeltime_forecast(new_data = testing(eco_splits),
                     actual_data = economics
  ) 

forcast_tbl %>% plot_modeltime_forecast(.interactive = F)
#> Warning in max(ids, na.rm = TRUE): no non-missing arguments to max; returning
#> -Inf

sessionInfo()
#> R version 4.4.1 (2024-06-14)
#> Platform: x86_64-pc-linux-gnu
#> Running under: Ubuntu 22.04.5 LTS
#> 
#> Matrix products: default
#> BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.10.0 
#> LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.10.0
#> 
#> locale:
#>  [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8       
#>  [4] LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8   
#>  [7] LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C          
#> [10] LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   
#> 
#> time zone: Europe/Warsaw
#> tzcode source: system (glibc)
#> 
#> attached base packages:
#> [1] stats     graphics  grDevices utils     datasets  methods   base     
#> 
#> other attached packages:
#> [1] timetk_2.9.0    modeltime_1.3.1 workflows_1.1.4 rsample_1.2.1  
#> [5] parsnip_1.2.1   recipes_1.1.0   dplyr_1.1.4     ggplot2_3.5.1  
#> 
#> loaded via a namespace (and not attached):
#>  [1] tidyselect_1.2.1    timeDate_4041.110   farver_2.1.2       
#>  [4] fastmap_1.2.0       reprex_2.1.1        digest_0.6.37      
#>  [7] rpart_4.1.23        timechange_0.3.0    lifecycle_1.0.4    
#> [10] StanHeaders_2.32.10 yardstick_1.3.1     survival_3.7-0     
#> [13] magrittr_2.0.3      compiler_4.4.1      rlang_1.1.4        
#> [16] tools_4.4.1         utf8_1.2.4          yaml_2.3.10        
#> [19] data.table_1.16.2   knitr_1.49          labeling_0.4.3     
#> [22] DiceDesign_1.10     withr_3.0.2         purrr_1.0.2        
#> [25] nnet_7.3-19         grid_4.4.1          tune_1.2.1         
#> [28] fansi_1.0.6         xts_0.14.1          colorspace_2.1-1   
#> [31] future_1.34.0       globals_0.16.3      scales_1.3.0       
#> [34] iterators_1.0.14    MASS_7.3-61         cli_3.6.3          
#> [37] rmarkdown_2.29      generics_0.1.3      RcppParallel_5.1.9 
#> [40] rstudioapi_0.17.1   future.apply_1.11.3 stringr_1.5.1      
#> [43] tidymodels_1.2.0    splines_4.4.1       dials_1.3.0        
#> [46] parallel_4.4.1      infer_1.0.7         vctrs_0.6.5        
#> [49] hardhat_1.4.0       Matrix_1.7-1        listenv_0.9.1      
#> [52] foreach_1.5.2       gower_1.0.1         tidyr_1.3.1        
#> [55] glue_1.8.0          modeldata_1.4.0     parallelly_1.39.0  
#> [58] codetools_0.2-20    stringi_1.8.4       lubridate_1.9.3    
#> [61] gtable_0.3.6        munsell_0.5.1       GPfit_1.0-8        
#> [64] tibble_3.2.1        furrr_0.3.1         pillar_1.9.0       
#> [67] workflowsets_1.1.0  htmltools_0.5.8.1   ipred_0.9-15       
#> [70] lava_1.8.0          R6_2.5.1            lhs_1.2.0          
#> [73] evaluate_1.0.1      lattice_0.22-6      backports_1.5.0    
#> [76] broom_1.0.7         class_7.3-22        Rcpp_1.0.13-1      
#> [79] prodlim_2024.06.25  xfun_0.49           forcats_1.0.0      
#> [82] zoo_1.8-12          fs_1.6.5            pkgconfig_2.0.3

^{Created on 2024-11-14 with reprex v2.1.1}

Expected Behavior:
Forecasts with step_scale() should ideally align with those without scaling when properly calibrated.

Observed Behavior:
Forecasts generated with step_scale() in the recipe display systematic errors and notably different results compared to those without scaling, as evident from accuracy metrics and forecast plots.

Additional Context:
It would be helpful to understand whether step_scale() is introducing unexpected data transformations that interfere with forecast accuracy in modeltime, or if additional steps are needed to ensure compatibility.

[mdancho84]

Normally I scale the target outside of the recipe.

Scaling the target inside the recipe has unwanted consequences. So I recommend not doing so as a recipe step.

[joranE]

@mdancho84 I don't believe the target variable pce is being scaled. Also, the results are the same even if you omit the date variable from the scaling.

[dudel666]

In this case, I am only scaling the predictors. Additionally, when generating predictions using predict(), the results are identical for both recipes (with and without step_scale). Therefore, the root of the issue must lie within modeltime_calibrate function.

[mdancho84]

Can you verify that pce is not being scaled?

bake(testing(splits)) %>% glimpse()

[joranE]

@mdancho84 I had checked this before commenting:

> recep |> prep() |> bake(new_data = testing(eco_splits)) |> glimpse()
Rows: 120
Columns: 7
$ date     <date> 2005-05-01, 2005-06-01, 2005-07-01, 2005-08-01, 2005-09-01, 2005…
$ pop      <dbl> 295704, 295936, 296186, 296440, 296707, 296972, 297207, 297431, 2…
$ psavert  <dbl> 3.5, 2.9, 2.2, 2.7, 2.7, 3.1, 3.5, 3.7, 4.2, 4.2, 4.2, 4.0, 3.8, …
$ uempmed  <dbl> 9.1, 9.0, 8.8, 9.2, 8.4, 8.6, 8.5, 8.7, 8.6, 9.1, 8.7, 8.4, 8.5, …
$ unemploy <dbl> 7651, 7524, 7406, 7345, 7553, 7453, 7566, 7279, 7064, 7184, 7072,…
$ pce      <dbl> 8643.9, 8724.8, 8829.5, 8832.4, 8885.8, 8926.6, 8938.5, 8969.6, 9…
$ date_num <dbl> 12904, 12935, 12965, 12996, 13027, 13057, 13088, 13118, 13149, 13…
> recep_with_scale |> prep() |> bake(new_data = testing(eco_splits)) |> glimpse()
Rows: 120
Columns: 7
$ date     <date> 2005-05-01, 2005-06-01, 2005-07-01, 2005-08-01, 2005-09-01, 2005…
$ pop      <dbl> 8.061196, 8.067520, 8.074336, 8.081260, 8.088539, 8.095763, 8.102…
$ psavert  <dbl> 1.1807652, 0.9783483, 0.7421953, 0.9108760, 0.9108760, 1.0458206,…
$ uempmed  <dbl> 2.215921, 2.191570, 2.142869, 2.240272, 2.045465, 2.094167, 2.069…
$ unemploy <dbl> 2.895957, 2.847887, 2.803223, 2.780134, 2.858863, 2.821013, 2.863…
$ pce      <dbl> 8643.9, 8724.8, 8829.5, 8832.4, 8885.8, 8926.6, 8938.5, 8969.6, 9…
$ date_num <dbl> 2.556326, 2.562467, 2.568410, 2.574552, 2.580693, 2.586636, 2.592…