Skip to content

Commit

Permalink
Switch from multisession parallelization to multicore in `evaluat…
Browse files Browse the repository at this point in the history
…e` stage (#261)

* Switch from `multisession` parallelization to `multicore` in `evaluate` stage

* Fix comment in pipeline/03-evaluate.R

Co-authored-by: Dan Snow <31494343+dfsnow@users.noreply.github.com>

---------

Co-authored-by: Dan Snow <31494343+dfsnow@users.noreply.github.com>
  • Loading branch information
jeancochrane and dfsnow authored Nov 6, 2024
1 parent f52c640 commit 2f054c8
Showing 1 changed file with 8 additions and 2 deletions.
10 changes: 8 additions & 2 deletions pipeline/03-evaluate.R
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,14 @@ tictoc::tic("Evaluate")
# Load libraries, helpers, and recipes from files
purrr::walk(list.files("R/", "\\.R$", full.names = TRUE), source)

# Enable parallel backend for generating stats more quickly
plan(multisession, workers = num_threads)
# Enable parallel backend for generating stats faster
if (supportsMulticore()) {
# Limit to half the available cores to avoid hogging resources
plan(multicore, workers = ceiling(num_threads / 2))
} else {
# Multisession performance begins to degrade beyond 5 workers
plan(multisession, workers = 5)
}

# Renaming dictionary for input columns. We want the actual value of the column
# to become geography_id and the NAME of the column to become geography_name
Expand Down

0 comments on commit 2f054c8

Please sign in to comment.